[Gambas-user] Using test interface for scripter testing

[Brian G]

> I have been implementing a test framework for scripter, and some of my own apps from my experience so far these are some things I have found difficult and thus the suggestions. 
> Please understand that I really do appreciate the work that has been done and and the functionality it provides. This is a bit long! 
> I would like to suggest the following be added to test framework if possible. It would make unit testing of the application and its modules a little simpler. 
> 
> Assert.Startup("startupModExpected") ' used to verify you have not forgotten to set to real startup ... lol 
> Assert.Exported("namelist",......... ) ' list of expected exported modules and classes fails if don't match 
> Assert.Interface("modOrClass", "interfaceFuncOrVarsOrProperty".....) ' test exported mods or classes that the interface is correct ie public functions and variables exposed, returns fail if they don't match. 
> 
> AssertStdOut.Equal(Function(), "Expected",".....") 
> AssertStdErr.Equal(Function(), "Expected,"....") 
> 
> Now I have to explain that most of the time, I replaced the 'got value' with the actual function call from the tested module 
> 

I don't see these basic functions as special, but normal checks for most libraries and components.

>>>These tests seem a little special to me. Did you know that you can override
>>>the Assert module and extend it? This is a standard Gambas feature, see [1].
>>>It allows you to add your own project-specific assertions to the Assert
>>>class, so it looks like they were built in. Use the primitive assertions
>>>as you need to build up more complex, specific ones.

I Don't believe that testing the interface etc. to a module or exported class is very special, it allows to verify the expected interface for a library or component
are as expected, they are not very complicated and always be a part of a library or component test. 

Should everyone using the test modules have to re implement for these simple things?

These may be a better example, and easy to use

AssertStdOut.equal(objectOrClass, "method",[parms...], "expected","Comment")  is how I have implemented it in actuality for my testing
AssertStdErr.equal(objectOrClass, "method",[parms...], "expected","Comment")  is how I have implemented it in actuality 


Although I used different function names and style ...lol

Private Sub Printoutput(mObject As Object, func As String, ...) As String ' to capture a printed string from function and list of parameters
  
  Dim buffer As Byte[]
  Dim len As Integer
  Dim mFile As File
  
  mFile = Open Null For Write
  Output To mFile
  Object.Call(mObject, func, Param.all)
  len = Seek(mFile)
  Close mFile
  buffer = New Byte[len + 100]
  
  mFile = Memory buffer.data For Write
  
  Output To mFile
   Object.Call(mObject, func, Param.all)
 
  len = Seek(mfile)
  Flush 
  Close mFile
  Output To Default
  
  Return buffer.ToString(0, len)
  
End

Private Sub PrintErrOut(mObject As Object, func As String, ...) As String ' to capture a printed string from function and list of parameters

  Dim buffer As Byte[]
  Dim len As Integer
  Dim mFile As File
  
  mFile = Open Null For Write ' can skip this if we just guess at a meg or so...
  Output To mFile
   Object.Call(mObject, func, Param.all)
  
  len = Seek(mFile)
  Close mFile
  buffer = New Byte[len + 100]
  
  mFile = Memory buffer.data For Write

  Error To mFile
   Object.Call(mObject, func, Param.all)
  len = Seek(mfile)
  Close mFile

  Error To Default
  Dim s As String = buffer.ToString(0, len)
  Return s
  
End

I was giving a simplified view. But I do believe that these two functions should be part of a general test framework. And not have to be re-implemented by everyone
that writes a cli application Or Outputs text.

So my calls look like this

assert.equal(PrintErrorOut(MMain, "Compile",[]),"Completed OK","Test Main Method"))

These are very crude, but if something built in was better implemented?
Testing the actual output from a module including text , is a far better test than simple checking if a method responds, as it tests through a complex system.

> Is it possible to have an 
> 
> AssertWithTimeout.equal(time, OpendbInput("parm"),"Timeout","The Called func Times out and returns an error from function") ' function times out and returns error before the assert timeout happens 
> Does something along the lines of pressing stop in the ide.... if timeout expires. 
> 
> Or have an optional setting test.SetTimeout(time) such that if a call hangs it does not end the sequence of tests. 
> For situations where a bad function is really bad or the test module itself fails. 
> 

The correct way to implement this would be (AssertWithTimeout.equal(timeout,objectOrClass,"method",[Params,...],"expected","Comment")

>>>No, because

>>>  Assert.Equal(Function(), "Expected")

>>>executes just like

>>>  Dim tmp As Variant = Function()
>>  Assert.Equal(tmp, "Expected")

>>>That is, the function call is evaluated by Gambas even before control
>>>is passed into gb.test. There is no way we can guard the execution of
>>>Function() by a Timer then.
My example was wrong of course!

>>>Test.SetTimeout could set a global process timeout at most. And even
>>>then, Gambas is non-preemptive: if Function() is never calling the
>>>event loop, gb.test would never get to see the timer event.

>>>It seems to me that you'd be much happier to let the *other* side
>>>of the testing, the test runner, do timeouts. Since it is a separate
>>>process connected to the TAP stream, it can easily set a timer to
>>>kill the test after X seconds total or Y seconds after the last TAP
>>>line was received from it (meaning a hanging test). A signaled process
>>>dies with non-zero exit code, which is already recognized as a test
>>>failure indicator. The test runner, in this case, would be the IDE.

> Having a macro/switch which prevents an app from exiting on quit/etc when testing error conditions generated for the interface of modules. And so ensure errors are being handled correctly. 
> Perhaps test should do by default. 
> 

>>>I don't think catching Quit is possible at all. As for errors from your
>>>project that you don't catch as a test writer, they are caught by gb.test
>>>and count as failed assertions IIRC. This does not cover voluntary Quits
>>>from inside of your tested code, nor really abnormal exits like signals.

Being able to test how modules handle errors is important even if it terminates.
Can a hook be added into the ide/compiler, that allows this behavior for the test module.


> Is it possible to have a way of pushing app command line args before calling a module, to test for correct parsing and processing of options. 
> Yes right now we can call functions but can not test the interface to the app. 
> e.g. 
> 
> test.exitDisable() ' maybe app exiting should be disabled by test framework by default 
> test.args("-v", ....) 
> AssertStdOut.Equal(mmain, "main",[] ,"Version output", "version output error from main") ' test captures the stream and compares 
> 
> test.args("-z",....) 
> AssertStdErr.Equal(mmain, "optparser",[],"optparser output parm error","test bad parameter passed") ' test captures the stream and compares 
> etc. 
> 

>>>This one is tricky because you cannot just rewrite your program in a way
>>>that makes testing easier, because of limitations in gb.args: it always
>>>wants to only work on the per-process Args object which is unfortunately
>>>read-only.

>>>The problem with setting args is that the program is already started
>>>when your code can call onto gb.test to set something up for you.
>>gb.test runs inside the process and just calls your tests.

>>>I have no idea how to test gb.args-style arguments handling.

> Maybe just dreaming 
> 
> Is it possible to make all variables/functions public and override the private setting during testing so that global tables that are constructed by private functions from input can be checked without making them 
> public in the actual code. This would help a great deal during the testing phase. 
> 
> Right now I needed add a flag into the actual code that stops app exits/quit with return. This itself adds a possible point of error. 
> 
> It often takes far more code to correctly test an application and all of it's possible errors and outputs than the app code actually is in size. And it is quite possible if left in place that some of the tests may create a security risk as they seem to be public. 
> 
> Are you thinking of this as more of a modelling tool than a test framework. Where you build an app framework with test then build your app to match that framework rather than actually testing the complete functionality and robustness of an application for production use. 
> 
> Should not a good test frame work allow simple direct testing providing interfaces to the most common functions of an average app. 
> 
> I t would be really nice if you could input data to a form field/button or input command through a keyboard or mouse simulation rather than just using test to call bits of an app. 
> 
> Test.KeyStrokes(&x20,&x34,&x35,.....) 
> assert.equal(SomeField.text," 45","test that input works correctly") 
> or 
> Test.KeyStrokes(ctrl-F, &x20) 
> assert.equal(SomeControl.HasFocus,true,"test that input works correctly") 
> 

My concern is focused on security, I want to test the functions etc inside my library or component which requires they are public, but I can't have them exposed
during the actual use of the lib or component as they may give access to things a production application should not see or be able to manipulate. And I don't want to flip them back and forth
as that can lead to more errors. If a major component builds an internal table from input and I want to verify it is correctly created and populated under error conditions/Use etc is that wrong.
If testing means its public, but its critical to the libraries function and It can not be public. Do I have to start writing test hooks etc..... It this what your saying. To me that sort of thing
is unneeded overhead. When the test suite could simply access it and verify it. Believe me if I add test hooks and they are public, some hacker will figure out how to use them.

>>>Well, it does take effort to write a program in such a way that
>>>its individual parts as well as their combinations are testable!

The things I have listed are part of any well implemented and modular program, They are not special to monolithic programs or functional code or modular code
The very best interfaces exposed externally have underlying functions, tables, classes etc. who's interface needs to be tested.
You are correct it does take careful and thoughtful development to achieve reliable and usable code.

It also takes thought and flexibility to develop a good suite of testing tools. 


>>>gb.test can be a modeling or specification tool (aka "test driven
>>>development"). It can also be used to test core library functions
>>>(mapping input to predictable output) or to add regression tests
>>>to a project (give me a piece of code that shows a bug, I fix it
>>>and keep your piece of code around to ensure that the same bug
>>>never manifests again).

>>>What you cannot expect from it is to help you test a program whose
>>>internals are tightly coupled and written with no regard for clear,
>>>separable APIs and testability.

I don't see how anything I have asked for precludes good api's or interfaces

>>>Testing the behavior of a GUI is a reasonable but not an easy task.
>>>Testing your internal program state via its effect on the GUI, on the
>>>other hand, is misguided. "Testability" means that your project is
>>>_not_ such an impenetrable monolith that the only way to verify it
>>>is simulating how a human would verify it without gb.test. That is,
>>>instead of faking keyboard input and examining GUI properties, better
>>>try and isolate the handling of keystrokes into a (pure) function
>>>with an input and an output and test that.

Umm well before I turn a app over to the user community I usually like to test if the user interface works!
Like if I push button a, list box b gets populated correctly etc.

Now if the test tool can allow me to simulate keystrokes and verify that the interface functions perform correctly
and do this in a repeatable and reliable manor. I don't see how this makes my program some sort of monolith or examining the resultant listbox
is some how an act of hacking. Look I want to know the if the gui functions, including all its bits and bobs. 
I cant just test a key stream in isolation to much of the real world would go untested.....

I get the feeling that you believe that testing a few module inside the app or library is enough. What makes every app or library useful is
for most apps, the user interface, for library/Component the published interface.

Part of my requests were to approach the verification of these for libraries, refer to my first request exported, and interface to allow certification of these.

And the keystrokes, not to just look at gui field, simple example I gave but to test the the user interface features and functions as a user would see or use it.
Again I don't see how this is out of line with proper testing. It does extend your framework a little past unit testing, but would be invaluable as a certification tool.

I spent many years as a principle systems architect, seen many test system, done a lot of development, having an out of the box powerful tool that provides simple to use features
that don't demand following a forced schema is always a joy. There are so many people in the world with so many different ideas and methods, a good tool is one that can accommodate the greatest subset
or users needs not try to define them.

The Test feature you have implemented is a good solid feature and I like it for the most part, I don't think the thing I have suggested would break anything and
would absolutely simplify many aspects of the test delivery cycle.

I can see how hard you have worked on test, you should feel great at your achievement I would never have used it at all if i did not see it's merits and strengths.
I am certainly grateful you undertook this project and  have brought it so far along, thank you.

>>>Yes, it takes care to write testable programs.

>>>Best,
>>>Tobias

Thanks
Brian G

[1] http://gambaswiki.org/wiki/doc/object-model#t21

-- 
"There's an old saying: Don't change anything... ever!" -- Mr. Monk

----[ http://gambaswiki.org/wiki/doc/netiquette ]----