[Gambas-user] Using test interface for scripter testing

[Tobias Boege]

On Sat, 27 Jun 2020, Brian G wrote:
> I have been implementing a test framework for scripter, and some of my own apps from my experience so far these are some things I have found difficult and thus the suggestions. 
> Please understand that I really do appreciate the work that has been done and and the functionality it provides. This is a bit long! 
> I would like to suggest the following be added to test framework if possible. It would make unit testing of the application and its modules a little simpler. 
> 
> Assert.Startup("startupModExpected") ' used to verify you have not forgotten to set to real startup ... lol 
> Assert.Exported("namelist",......... ) ' list of expected exported modules and classes fails if don't match 
> Assert.Interface("modOrClass", "interfaceFuncOrVarsOrProperty".....) ' test exported mods or classes that the interface is correct ie public functions and variables exposed, returns fail if they don't match. 
> 
> AssertStdOut.Equal(Function(), "Expected",".....") 
> AssertStdErr.Equal(Function(), "Expected,"....") 
> 
> Now I have to explain that most of the time, I replaced the 'got value' with the actual function call from the tested module 
> 

These tests seem a little special to me. Did you know that you can override
the Assert module and extend it? This is a standard Gambas feature, see [1].
It allows you to add your own project-specific assertions to the Assert
class, so it looks like they were built in. Use the primitive assertions
as you need to build up more complex, specific ones.

> Is it possible to have an 
> 
> AssertWithTimeout.equal(time, OpendbInput("parm"),"Timeout","The Called func Times out and returns an error from function") ' function times out and returns error before the assert timeout happens 
> Does something along the lines of pressing stop in the ide.... if timeout expires. 
> 
> Or have an optional setting test.SetTimeout(time) such that if a call hangs it does not end the sequence of tests. 
> For situations where a bad function is really bad or the test module itself fails. 
> 

No, because

  Assert.Equal(Function(), "Expected")

executes just like

  Dim tmp As Variant = Function()
  Assert.Equal(tmp, "Expected")

That is, the function call is evaluated by Gambas even before control
is passed into gb.test. There is no way we can guard the execution of
Function() by a Timer then.

Test.SetTimeout could set a global process timeout at most. And even
then, Gambas is non-preemptive: if Function() is never calling the
event loop, gb.test would never get to see the timer event.

It seems to me that you'd be much happier to let the *other* side
of the testing, the test runner, do timeouts. Since it is a separate
process connected to the TAP stream, it can easily set a timer to
kill the test after X seconds total or Y seconds after the last TAP
line was received from it (meaning a hanging test). A signaled process
dies with non-zero exit code, which is already recognized as a test
failure indicator. The test runner, in this case, would be the IDE.

> Having a macro/switch which prevents an app from exiting on quit/etc when testing error conditions generated for the interface of modules. And so ensure errors are being handled correctly. 
> Perhaps test should do by default. 
> 

I don't think catching Quit is possible at all. As for errors from your
project that you don't catch as a test writer, they are caught by gb.test
and count as failed assertions IIRC. This does not cover voluntary Quits
from inside of your tested code, nor really abnormal exits like signals.

> Is it possible to have a way of pushing app command line args before calling a module, to test for correct parsing and processing of options. 
> Yes right now we can call functions but can not test the interface to the app. 
> e.g. 
> 
> test.exitDisable() ' maybe app exiting should be disabled by test framework by default 
> test.args("-v", ....) 
> AssertStdOut.Equal(main() ,"Version output", "version output error from main") ' test captures the stream and compares 
> 
> test.args("-z",....) 
> AssertStdErr.Equal(optparser(),"optparser output parm error","test bad parameter passed") ' test captures the stream and compares 
> etc. 
> 

This one is tricky because you cannot just rewrite your program in a way
that makes testing easier, because of limitations in gb.args: it always
wants to only work on the per-process Args object which is unfortunately
read-only.

The problem with setting args is that the program is already started
when your code can call onto gb.test to set something up for you.
gb.test runs inside the process and just calls your tests.

I have no idea how to test gb.args-style arguments handling.

> Maybe just dreaming 
> 
> Is it possible to make all variables/functions public and override the private setting during testing so that global tables that are constructed by private functions from input can be checked without making them 
> public in the actual code. This would help a great deal during the testing phase. 
> 
> Right now I needed add a flag into the actual code that stops app exits/quit with return. This itself adds a possible point of error. 
> 
> It often takes far more code to correctly test an application and all of it's possible errors and outputs than the app code actually is in size. And it is quite possible if left in place that some of the tests may create a security risk as they seem to be public. 
> 
> Are you thinking of this as more of a modelling tool than a test framework. Where you build an app framework with test then build your app to match that framework rather than actually testing the complete functionality and robustness of an application for production use. 
> 
> Should not a good test frame work allow simple direct testing providing interfaces to the most common functions of an average app. 
> 
> I t would be really nice if you could input data to a form field/button or input command through a keyboard or mouse simulation rather than just using test to call bits of an app. 
> 
> Test.KeyStrokes(&x20,&x34,&x35,.....) 
> assert.equal(SomeField.text," 45","test that input works correctly") 
> or 
> Test.KeyStrokes(ctrl-F, &x20) 
> assert.equal(SomeControl.HasFocus,true,"test that input works correctly") 
> 

Well, it does take effort to write a program in such a way that
its individual parts as well as their combinations are testable!

gb.test can be a modeling or specification tool (aka "test driven
development"). It can also be used to test core library functions
(mapping input to predictable output) or to add regression tests
to a project (give me a piece of code that shows a bug, I fix it
and keep your piece of code around to ensure that the same bug
never manifests again).

What you cannot expect from it is to help you test a program whose
internals are tightly coupled and written with no regard for clear,
separable APIs and testability.

Testing the behavior of a GUI is a reasonable but not an easy task.
Testing your internal program state via its effect on the GUI, on the
other hand, is misguided. "Testability" means that your project is
_not_ such an impenetrable monolith that the only way to verify it
is simulating how a human would verify it without gb.test. That is,
instead of faking keyboard input and examining GUI properties, better
try and isolate the handling of keystrokes into a (pure) function
with an input and an output and test that.

Yes, it takes care to write testable programs.

Best,
Tobias

[1] http://gambaswiki.org/wiki/doc/object-model#t21

-- 
"There's an old saying: Don't change anything... ever!" -- Mr. Monk