[Gambas-user] Integrate unittest component to Gambas

[Tobias Boege]

On Sun, 13 May 2018, Adrien Prokopowicz wrote:
> > Yes, you got it right, the tests are executed one after another and the
> > results are buffered in TestResult (but only the failed ones). Therefore
> > I had to restore the sequence by iterating through TestSuite (which
> > contains the Tests) in Unittest.PrintTapResult().
> > 
> > This is not very cool, indeed it would be better if the Unittest would
> > print out the TAP results on the fly without buffering.
> > 
> > I you have a look at the stable code there it is done with events, but
> > was delivered to the GUI to the tracer form. Maybe we can take these
> > events and deliver to the TAP printer.
> > 
> > One thing, the current code does right, is: It distinguishes between
> > failures (the result is not as expected) and errors and can buffer (not
> > so good) errors (this happens in TestCase.Run [59-62]).
> > 
> > But if the tested code leads to a segfault then all previous test
> > results are gone. That's pretty bad and should not happen.
> > 
> > This week I won't be able to do anything further so feel free to play
> > around.
> > 
> > 
> > Alles Gute
> > 
> > Christof Thalhofer
> 
> Hey folks, I know I may be a bit late to the party, but I some (long) time
> ago I also made some experiments to make automated tests for Gambas (see
> here[0]).
> 

The code has been mostly rotting in my repository for the last two(?) weeks,
but your comments touch exactly the area where I hit a road block and put it
on the backburner, in favour of uni stuff.

I need to convert to convert the UnitTest class and all its users to the
TAP-based classes and possibly redefine how tests are done and the testing
interface.

> It isn't as sophisticated as yours (no TAP serialization or setup/teardown),
> but the interesting bit was that is relied on Tasks to run the tests
> (instead of running them directly), which has several benefits :
> 
> - You are 100% sure you go back to the initial memory state between each
> test (since each test creates a new Task, hence re-forking the main
> process).
> - It allows to run tests in parallel on multiple cores.
> - (Most importantly IMHO) Since each test is ran in a separate process, it
> is resilient to crashes and segfaults, which makes it ideal for testing
> native components.
> 
> If you are interested in this, I could probably send a few PRs towards you
> to integrate this system. :-)
> 

I believe my fork [1] is currently the most advanced branch of gb.unittest.

I'm generally wary of using Tasks because "Many components will not like
being forked. Especially the GUI ones. So be careful." In addition, we
already have the TAP architecture in [1], of which you say, if I understand
it right, that it's more sophisticated. Using TAP, we spawn a new process
for each unit test, which can contain several assertions, and get output in
a standard text format, which can be stored on disk and also be read by
non-Gambas tools. Running multiple tests simultaneously is as easy as using
GNU parallel.

Let me tell you what I imagined in the meantime. This also an RFC for Benoit.
The project directory structure should get a new official directory .tests/
or .t/ where tests are located. Tests are classes which inherit UnitTest
from the gb.unittest component. They contain a bunch of related tests.
The IDE will handle this directory specially for in-IDE testing and there
will be a command-line utility "gbprove3" similar to "prove". More signifi-
cantly, the compiler and archiver should ignore tests by default, as you
don't want to ship those in production.

NB: Since the unit tests are not located inside the unit they test, we only
test public API. I think that's the more sensible approach by itself and
it allows for the separation I just mentioned.

[1] https://github.com/taboege/gb.deg.unittest/tree/TAP

> Also, a few thoughts I had about the Assertion API: in your component, to
> check an assertion you have to do this (my old one has something along those
> lines as well):
> 
> Me.Result.AssertEqualsString("Hello", myString)
> 
> However, after putting in some thought, not only I find this pretty lengthy
> for assertions (which your tests will be full of), but all in all I don't
> find the "use protected methods from inheritance" very Gambas-y (since
> Gambas does not have a "protected" visibility anyway).
> 
> I think a simpler way to do this would be to make a static Assert class (see
> example attached, actual implementation would be more complex obviously).
> This way, you can call it directly without needing to go through ME or other
> objects :
> 
> Assert(myBoolExpression)
> Assert.Equals("Hello", myString)
> Assert.Same(objRef1, objRef2)
> Assert.Null(nullRef)
> '… and whatever other helper method we could think of
> 
> This makes it much shorter and simpler, and it also makes assertions useable
> outside tests, which could be desirable I think.
> 

Certainly the current API is too lengthy. The TAP classes provide pretty
much what you suggest, but there are at least two variations of your
Assert class. (1) We could exploit the inheritance of the UnitTest class
in tests and write

  Me.Equals("Hello", myString)

This is short and makes sense from an implementation point of view, since
the TAP state (how many tests are planned, what is the current test number,
should the next test be tagged as TODO or SKIP, which Stream do we send the
TAP output to?) is hidden away inside the parent UnitTest class and all of
that must be accessed by whatever implements your Assertion interface.
[ NOTE that I'm talking about *my* UnitTest class here, which isn't
committed yet. ]

(2) We could also ditch the UnitTest inheritance, put the relevant state
into a static module and go as far as writing builtin-lookalike assertions:

  Equals("Hello", myString) ' Equals is a module with a _call method

but perhaps that's too much, as even if we only load gb.unittest when
running classes from .tests/, it might conflict with the project's
class names.

Those are my thoughts, anyway.

Regards,
Tobi

-- 
"There's an old saying: Don't change anything... ever!" -- Mr. Monk