[Gambas-user] New component gb.test

[Adrien Prokopowicz]

Le 19/05/2018 à 02:44, Tobias Boege a écrit :
> Hi all,
> 
> continuing from this thread: https://lists.gambas-basic.org/pipermail/user/2018-April/063732.html
> I have pushed the to-be unit testing component to (a separate branch of)
> the official repository [1]. I think this is the easiest way to enable
> collaboration on it, and it has to be merged at some point anyway, as we
> plan to have the IDE use it. I have substantially cleaned the repository
> up today and it is now in a state where others can dive into it and get
> the right idea of how it is supposed to work.
> 
> It is now named gb.test as Adrien suggested, was previously gb.unittest
> and gb.deg.unittest before that. It was originally written by Christof
> and inspired by COMUnit, which in turn was inspired by JUnit. But I have
> been rewriting it to the Test Anything Protocol.

Oh that's very good, thank you! I was currently looking into cleaning it 
up a bit myself actually (I already created the same branch locally with 
the same name), but nice to see it's done. :-)

Now I'll start working on what I suggested in the previous thread, which 
is to split the component's logic between :

- The test harness itself, which will be the "gbtest3" command-line 
tool, and will generate TAP data to be read by the IDE and CI
- The gb.test component itself, which will contain only the bare minimum 
to run a single test, print the result and exit the process (or list all 
the available tests), as well as the Assert utility module.

That way it will be much easier for the IDE, the CI system and the 
end-user to run the tests. :-)

> 
> Lots of history... and I pulled all that into the branch. I hope that's
> okay, even if the old commit messages aren't in the usual format.

Having commit messages not in the usual format only mean they won't show 
up in the changelog (which is good here), but in the worst case we can 
still squash the branch before merging it into master. :-)

> To kick it off, I created a TODO file in the component:
> 
>    * Integration into the IDE with bells and whistles and a command-line tool for automated testing
>    * Compiler and interpreter support for test classes (list them), fixtures and data files (do not include them in installable packages)
> 
>    * Internal and external (wiki) documentation, test how-to
>    * Test ourselves, including how well we interface with other TAP software
>    * More assertion convenience functions, use diagnostic messages to detail test failures for the user (Test::Differences in perl5)
>    * Subtest support (see http://search.cpan.org/~exodist/Test-Simple-1.302136/lib/Test/More.pm)
> 
> The former two points are bigger endeavours and probably need Benoit's
> guidance, but until these two points are implemented, the usability of
> the component is rather limited. The latter points could be taken by
> anyone right now

About those two points I would rather ask for Benoît's mentoring rather 
than having him implement it himself (especially the compiler/interpeter 
part). Despite my best efforts to understand them, I've always felt like 
these parts of Gambas were huge jungles of undocumented methods and ugly 
gotos. And it's not fair that he's keeping all the fun to himself. ;-)

(That's when he's back from vacation, of course).

> I am probably starting with the documentation, as I have noticed that
> the TAP architecture is quite different from the one inspired by JUnit.
> Besides the raw meaning of the interface, some explanations of how the
> classes are intended to work together are in order.

As I'll be splitting some code right now there will certainly be some 
user-facing API changes, so don't write too much docs for now. :-P

For instance I noticed the Assert module right now leaks a fair amount 
of implementation details, and I don't think the TestHarness class 
should be exposed (unless you meant it to be used as a base class for 
all test classes, in which case we should probably find a better name).

As for the external documentation, I don't think the end-user should 
bother too much with TAP : from their point of view, it should be just 
another test result format, only meant to be eaten by their favourite CI 
tool so they can have a nice generated report. And that's if they even 
bother with CI/external tooling at all, otherwise it's just an 
implementation detail between the test harness and the IDE. :-)

As for the API itself, I think it would be better to keep it simple : 
make a bunch of test modules, make them inherit a base TestModule class 
(at least until the compiler does it by itself, like with forms), write 
your tests in it using the various Assert.* helpers, and let Gambas do 
the rest!

Testing is often seen as a chore (because it is), so the last thing we 
want is to frighten users by making them deal with a bunch of new 
classes. ;-)

(By the way I didn't really understand what Subtests could look like in 
a Gambas, nor what they could be used for, could you explain it a bit ?)

This aside, I definitely agree with your other points. :-)

-- 
Adrien Prokopowicz