[Gambas-user] Using test interface for scripter testing

Brian G brian at westwoodsvcs.com
Sat Jun 27 20:15:59 CEST 2020


I have been implementing a test framework for scripter, and some of my own apps from my experience so far these are some things I have found difficult and thus the suggestions. 
Please understand that I really do appreciate the work that has been done and and the functionality it provides. This is a bit long! 
I would like to suggest the following be added to test framework if possible. It would make unit testing of the application and its modules a little simpler. 

Assert.Startup("startupModExpected") ' used to verify you have not forgotten to set to real startup ... lol 
Assert.Exported("namelist",......... ) ' list of expected exported modules and classes fails if don't match 
Assert.Interface("modOrClass", "interfaceFuncOrVarsOrProperty".....) ' test exported mods or classes that the interface is correct ie public functions and variables exposed, returns fail if they don't match. 

AssertStdOut.Equal(Function(), "Expected",".....") 
AssertStdErr.Equal(Function(), "Expected,"....") 

Now I have to explain that most of the time, I replaced the 'got value' with the actual function call from the tested module 

Is it possible to have an 

AssertWithTimeout.equal(time, OpendbInput("parm"),"Timeout","The Called func Times out and returns an error from function") ' function times out and returns error before the assert timeout happens 
Does something along the lines of pressing stop in the ide.... if timeout expires. 

Or have an optional setting test.SetTimeout(time) such that if a call hangs it does not end the sequence of tests. 
For situations where a bad function is really bad or the test module itself fails. 

Having a macro/switch which prevents an app from exiting on quit/etc when testing error conditions generated for the interface of modules. And so ensure errors are being handled correctly. 
Perhaps test should do by default. 

e.g. 
Replaced the 'got value' with the actual function call from the test module 
test.exitDisable() 

AssertStdOut.Equal(main(), "Operation Competed OK", "test if module runs to completion") ' this because test normally uses stdout 
AssertStdErr.Equal(PrintMessage("Complete")," achevé","Test translation") 

try CriticalFile("junk") Assert.Error( ..... 'if the module quits when it gets an error and the quit is overridden can we still see the last error 
if the module function does this for bad example: 
sub CriticalFile(Filename as string) as file 
mfile as file 
try mfile = open FileName for read 
if error then quit 6 
return mfile 
end 
and do this following 

If it is possible to to catch exits from app i.e. quit 6, is it possible to catch and verify the exit code returned 
CriticalFile("junk") 
Assert.ExitCode(6,"Exit value test for 6") 

test.exitEnable() 

also perhaps as a dream 
try GenProg("xxx") assert.NotError("Build Prog no error") 
AssertFile.Equal("ModelFilePathName", "OutPutFilePathName","Function Output text Verify") 'binary compare 

Is it possible to have a way of pushing app command line args before calling a module, to test for correct parsing and processing of options. 
Yes right now we can call functions but can not test the interface to the app. 
e.g. 

test.exitDisable() ' maybe app exiting should be disabled by test framework by default 
test.args("-v", ....) 
AssertStdOut.Equal(main() ,"Version output", "version output error from main") ' test captures the stream and compares 

test.args("-z",....) 
AssertStdErr.Equal(optparser(),"optparser output parm error","test bad parameter passed") ' test captures the stream and compares 
etc. 

Maybe just dreaming 

Is it possible to make all variables/functions public and override the private setting during testing so that global tables that are constructed by private functions from input can be checked without making them 
public in the actual code. This would help a great deal during the testing phase. 

Right now I needed add a flag into the actual code that stops app exits/quit with return. This itself adds a possible point of error. 

It often takes far more code to correctly test an application and all of it's possible errors and outputs than the app code actually is in size. And it is quite possible if left in place that some of the tests may create a security risk as they seem to be public. 

Are you thinking of this as more of a modelling tool than a test framework. Where you build an app framework with test then build your app to match that framework rather than actually testing the complete functionality and robustness of an application for production use. 

Should not a good test frame work allow simple direct testing providing interfaces to the most common functions of an average app. 

I t would be really nice if you could input data to a form field/button or input command through a keyboard or mouse simulation rather than just using test to call bits of an app. 

Test.KeyStrokes(&x20,&x34,&x35,.....) 
assert.equal(SomeField.text," 45","test that input works correctly") 
or 
Test.KeyStrokes(ctrl-F, &x20) 
assert.equal(SomeControl.HasFocus,true,"test that input works correctly") 

These are some of my thoughts. 
I have to say that what you have built is a great thing, and has helped me find a number of possible bugs, but I have had to build much of what I am talking about to test more completely. 

I believe that the framework you have built is a very solid foundation. Thank you for the effort and thought you have put into it. 

And I understand that there are thing I still don't understand that are included into the test module. 

Thank You 
Brian G 


From: "Brian" <brian at westwoodsvcs.com> 
To: "Gambas mailing list" <user at lists.gambas-basic.org> 
Sent: Saturday, June 27, 2020 6:35:43 AM 
Subject: Re: [Gambas-user] Using test interface for scripter testing 



Does that not defeat the purpose of testing the the app for production use. Ie checking if correct errors are generated by error conditions etc. Have to alter code, then not test the actual method or module that outputs them.... Ummm 


-- 
Thanks 
Brian 
Monday, 22 June 2020, 11:29PM -07:00 from Christof Thalhofer [ mailto:chrisml at deganius.de | chrisml at deganius.de ] : 




Am 23.06.20 um 00:00 schrieb Brian G: 

> I am including a sample app with test module for things I will need 
> to do my testing. As you suggested I am capturing printed output, 
> which works, but also seems to print into the test result file. 
> 
> Is there some better way to use test to test functions which produce 
> printed messages. 

Use a function that creates the message as string and test the output of 
that function. 

And then use that function in a method that prints. 

Alles Gute 

Christof Thalhofer 

-- 
Dies ist keine Signatur 


----[ [ http://gambaswiki.org/wiki/doc/netiquette | http://gambaswiki.org/wiki/doc/netiquette ] ]---- 





----[ http://gambaswiki.org/wiki/doc/netiquette ]---- 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.gambas-basic.org/pipermail/user/attachments/20200627/541c9fc6/attachment-0001.html>


More information about the User mailing list