[Gambas-user] New component gb.test

[Tobias Boege]

On Sat, 19 May 2018, Adrien Prokopowicz wrote:
> Le 19/05/2018 à 02:44, Tobias Boege a écrit :
> > I am probably starting with the documentation, as I have noticed that
> > the TAP architecture is quite different from the one inspired by JUnit.
> > Besides the raw meaning of the interface, some explanations of how the
> > classes are intended to work together are in order.
> 
> As I'll be splitting some code right now there will certainly be some
> user-facing API changes, so don't write too much docs for now. :-P
> 
> For instance I noticed the Assert module right now leaks a fair amount of
> implementation details, and I don't think the TestHarness class should be
> exposed (unless you meant it to be used as a base class for all test
> classes, in which case we should probably find a better name).
> 

I believe TestHarness says "Base class for a test harness." in the class
documentation :-)

In general, the component is currently modeled after perl's test facilities.
Not because I think it's the best option for Gambas but because I didn't
have to think up my own thing before getting something remotely usable.
So feel free to change it, but only for the better :-)

But in particular, the TestHarness is inside gb.test and intended to be a
base class because TAP allows for extensions. Lines which are not recognised
by the specification are not errors but can be passed upwards to the user,
who might implement his specialised harness.

> As for the external documentation, I don't think the end-user should bother
> too much with TAP : from their point of view, it should be just another test
> result format, only meant to be eaten by their favourite CI tool so they can
> have a nice generated report. And that's if they even bother with
> CI/external tooling at all, otherwise it's just an implementation detail
> between the test harness and the IDE. :-)
> 

I guess you can say that I view gb.test mainly as an implementation of TAP
for the purpose of giving Gambas automated tests, as opposed to an automated
testing component for Gambas using "whatever" internally. That might also
tell you why I Export'd all these classes: to be used by someone writing a
TAP-using application. Granted, that's not a big area, but you can imagine
writing a graphical testing front-end in Gambas for any language which has
TAP producers. The IDE is a TAP consumer specialised for Gambas projects.

I doubt that the API can be made less leaky without mutilating the advantages
of TAP for the reporting of test failures (adhering to a test plan, item
descriptions, diagnostics, TODO and SKIP tags). In my opinion, following a
test format specification is something that can be advertised to the user.
They don't have to understand the whole spec, but they need to realise that
they have to follow a certain structure with a test plan and such and that
the Assert class may print special diagnostic lines, prefixed by "#" which
contain details for failed tests, like where exactly two long strings or
arrays differ or where a regex gave up matching.

> As for the API itself, I think it would be better to keep it simple : make a
> bunch of test modules, make them inherit a base TestModule class (at least
> until the compiler does it by itself, like with forms), write your tests in
> it using the various Assert.* helpers, and let Gambas do the rest!
> 
> Testing is often seen as a chore (because it is), so the last thing we want
> is to frighten users by making them deal with a bunch of new classes. ;-)
> 

Are the current classes in SelfTest too complicated?

> (By the way I didn't really understand what Subtests could look like in a
> Gambas, nor what they could be used for, could you explain it a bit ?)
> 

Me neither. But looking at it again, I'm now convinced that it is an extension
to the TAP specification. A subtest groups many tests into a single item.
Technically, the TAP printer will indent the subtest items (so that they
are not valid standard-TAP lines) but print a real summary TAP line for the
subtest at the end -- "ok" when all subitems passed, "not ok" if one
of them failed.

Think of testing a single method with many different parameter configurations.
Instead of writing

  Assert.Ok(..., ..., "mymethod 1/200")
  Assert.Ok(..., ..., "mymethod 2/200")
  Assert.Ok(..., ..., "mymethod 3/200")
  ...

you could write

  Inc Assert.Subtest
    Assert.Plan(200, "mymethod")
    Assert.Ok(..., ...)
    Assert.Ok(..., ...)
    Assert.Ok(..., ...)
    ...
  Dec Assert.Subtest

[ The implementation can be as simple as printing every line indented by
  String$(Assert.Subtest, "\t") and maintaining a stack of TAP states
  instead of the current private variables; when Assert.Subtest is
  decremented, you pop a state off the stack and emit a summary for it. ]

Of course, the TAP parser needs to be told how to read these indented
TAP lines as well. See https://www.effectiveperlprogramming.com/2011/08/group-tests-by-their-task-with-testmores-subtest/

Regards,
Tobi

-- 
"There's an old saying: Don't change anything... ever!" -- Mr. Monk