[Gambas-user] Using test interface for scripter testing

[Tobias Boege]

On Sun, 28 Jun 2020, Brian G wrote:
> > I have been implementing a test framework for scripter, and some of my own apps from my experience so far these are some things I have found difficult and thus the suggestions. 
> > Please understand that I really do appreciate the work that has been done and and the functionality it provides. This is a bit long! 
> > I would like to suggest the following be added to test framework if possible. It would make unit testing of the application and its modules a little simpler. 
> > 
> > Assert.Startup("startupModExpected") ' used to verify you have not forgotten to set to real startup ... lol 
> > Assert.Exported("namelist",......... ) ' list of expected exported modules and classes fails if don't match 
> > Assert.Interface("modOrClass", "interfaceFuncOrVarsOrProperty".....) ' test exported mods or classes that the interface is correct ie public functions and variables exposed, returns fail if they don't match. 
> > 
> > AssertStdOut.Equal(Function(), "Expected",".....") 
> > AssertStdErr.Equal(Function(), "Expected,"....") 
> > 
> > Now I have to explain that most of the time, I replaced the 'got value' with the actual function call from the tested module 
> > 
> 
> I don't see these basic functions as special, but normal checks for most libraries and components.
> 
> >>>These tests seem a little special to me. Did you know that you can override
> >>>the Assert module and extend it? This is a standard Gambas feature, see [1].
> >>>It allows you to add your own project-specific assertions to the Assert
> >>>class, so it looks like they were built in. Use the primitive assertions
> >>>as you need to build up more complex, specific ones.
> 
> I Don't believe that testing the interface etc. to a module or exported class is very special, it allows to verify the expected interface for a library or component
> are as expected, they are not very complicated and always be a part of a library or component test. 
> 

The reason why I say these look special is that _normally_ the compiler
(or when late-bound, the interpreter) does these checks for you already.

So I wonder what you want to do with this:

  - Check that XmlDocument has a Root property of type XmlElement in
    all your projects that use gb.xml? That seems excessive and that
    test, if at all, belongs in gb.xml instead.

  - If you are the author of gb.xml and want to add these tests to your
    own test suite, simply _use_ the Root property of a valid XmlDocument
    in your tests. If the tests compile and execute, that already implies
    that the property exists and has the correct enough type. If the
    interpreter raises a runtime error that you do not catch, the test
    counts as failed.

Your mileage may vary, but I would consider it done at that point.
By the way, Assert.Startup(Name) can be written

  Assert.Equals(Application.Startup.Name, Name)

which I find short enough given that you should have to test this only
once per project.

> These may be a better example, and easy to use
> 
> AssertStdOut.equal(objectOrClass, "method",[parms...], "expected","Comment")  is how I have implemented it in actuality for my testing
> AssertStdErr.equal(objectOrClass, "method",[parms...], "expected","Comment")  is how I have implemented it in actuality 
> 
> [...]
> I was giving a simplified view. But I do believe that these two functions should be part of a general test framework. And not have to be re-implemented by everyone
> that writes a cli application Or Outputs text.
> 
> So my calls look like this
> 
> assert.equal(PrintErrorOut(MMain, "Compile",[]),"Completed OK","Test Main Method"))
> 
> [...]
> The correct way to implement this would be (AssertWithTimeout.equal(timeout,objectOrClass,"method",[Params,...],"expected","Comment")
> 

I generally dislike this style of stringy callback because it turns code
into a string blob and that excludes the IDE or the compiler from checking
it as code. You will notice that we have strictly avoided such an interface
in Assert so far.

Granted, Gambas doesn't leave you a choice as you cannot make an actual
"lambda function object" to pass around as argument on-the-fly. And I do
see the value in capturing stdout or stderr of a process under test or
wrapping any amount of changes to the runtime around the execution of a
piece of test code.

Right now, gb.test's Assert's stance on this is: "do it yourself, you are
the main program. Only bother me when you have an actual test result to
announce".

> >>>I don't think catching Quit is possible at all. As for errors from your
> >>>project that you don't catch as a test writer, they are caught by gb.test
> >>>and count as failed assertions IIRC. This does not cover voluntary Quits
> >>>from inside of your tested code, nor really abnormal exits like signals.
> 
> Being able to test how modules handle errors is important even if it terminates.
> Can a hook be added into the ide/compiler, that allows this behavior for the test module.
> 

Probably, the interpreter has a hook for quitting the application. Not sure
under which conditions it works for what you are proposing, though, that is,
if it can be used to inhibit quitting at all.

> My concern is focused on security, I want to test the functions etc inside my library or component which requires they are public, but I can't have them exposed
> during the actual use of the lib or component as they may give access to things a production application should not see or be able to manipulate. And I don't want to flip them back and forth
> as that can lead to more errors. If a major component builds an internal table from input and I want to verify it is correctly created and populated under error conditions/Use etc is that wrong.
> If testing means its public, but its critical to the libraries function and It can not be public. Do I have to start writing test hooks etc..... It this what your saying. To me that sort of thing
> is unneeded overhead. When the test suite could simply access it and verify it. Believe me if I add test hooks and they are public, some hacker will figure out how to use them.
> 

After compilation, your tests end up being normal classes inside the
project being tested and gb.test runs them like alternative startup
classes. Accordingly they have the best view possible on your project's
classes and modules. *But* we are also only cooking with water.

Tests can see and use your unexported classes, modules, forms, ...
but they cannot magically break encapsulation and visibility rules
in Gambas since they are just normal Gambas code.

So yes, if you want to access something, it must be accessible to
code from your project. That's as close as we can get at the moment.

> >>>Testing the behavior of a GUI is a reasonable but not an easy task.
> >>>Testing your internal program state via its effect on the GUI, on the
> >>>other hand, is misguided. "Testability" means that your project is
> >>>_not_ such an impenetrable monolith that the only way to verify it
> >>>is simulating how a human would verify it without gb.test. That is,
> >>>instead of faking keyboard input and examining GUI properties, better
> >>>try and isolate the handling of keystrokes into a (pure) function
> >>>with an input and an output and test that.
> 
> Umm well before I turn a app over to the user community I usually like to test if the user interface works!
> Like if I push button a, list box b gets populated correctly etc.
> 

Yes, please do that. I was saying that simulating a click on button a and
examining the contents of list box b is not a good way of inferring whether
the internal function f, that you know is run as part of the click event
handler, works correctly. Testing behavior of the GUI, on the first hand
again, is a reasonable but not an easy task.

I begin to suspect that we have a miscommunication because you talk with
a specific, apparently well-written code base in mind that I haven't seen
or expected (and I wouldn't have had the time to study it either, even if
you were to make it public).

> Now if the test tool can allow me to simulate keystrokes and verify that the interface functions perform correctly
> and do this in a repeatable and reliable manor. I don't see how this makes my program some sort of monolith or examining the resultant listbox
> is some how an act of hacking. Look I want to know the if the gui functions, including all its bits and bobs. 
> I cant just test a key stream in isolation to much of the real world would go untested.....
> 
> I get the feeling that you believe that testing a few module inside the app or library is enough. What makes every app or library useful is
> for most apps, the user interface, for library/Component the published interface.
> 
> Part of my requests were to approach the verification of these for libraries, refer to my first request exported, and interface to allow certification of these.
> 
> And the keystrokes, not to just look at gui field, simple example I gave but to test the the user interface features and functions as a user would see or use it.
> Again I don't see how this is out of line with proper testing. It does extend your framework a little past unit testing, but would be invaluable as a certification tool.
> 
> I spent many years as a principle systems architect, seen many test system, done a lot of development, having an out of the box powerful tool that provides simple to use features
> that don't demand following a forced schema is always a joy. There are so many people in the world with so many different ideas and methods, a good tool is one that can accommodate the greatest subset
> or users needs not try to define them.
> 
> The Test feature you have implemented is a good solid feature and I like it for the most part, I don't think the thing I have suggested would break anything and
> would absolutely simplify many aspects of the test delivery cycle.
> 
> I can see how hard you have worked on test, you should feel great at your achievement I would never have used it at all if i did not see it's merits and strengths.
> I am certainly grateful you undertook this project and  have brought it so far along, thank you.
> 
> [snip]
> Should everyone using the test modules have to re implement for these simple things?
> 

As a general rule of thumb: yes!

We've struggled against time (among others) to get gb.test to a stable state
for the 3.15 release. There are definitely things to improve which we decided
to do as part of the 3.15+ lifecycle(s) while users get accustomed to using
gb.test and point out shortcomings.

You are the first to do so, but I think the fact that you praise the efforts
so far and propose additions only is a very good sign for the 3.15 release.

Your ideas, except the absolutely easiest that we all agree are useful and
well-designed, are unlikely to go into master now and hence into 3.15.0.
GUI testing, for example, would likely require an addon component gb.test.gui
(perhaps already a misleading name!) so as to not put a GUI dependency on
gb.test.

So, after 3.15.0 is released with a stable base gb.test, I would ask everyone
to try out gb.test and the IDE (and gbt3 if it still exists), compile your
own additions to Assert and Test that you often need in user libraries and
share them.

Regards,
Tobias

-- 
"There's an old saying: Don't change anything... ever!" -- Mr. Monk