[Gambas-user] Questions about gb.test and gb.test.tap

[Benoît Minisini]

Le 24/02/2020 à 23:20, Tobias Boege a écrit :
> 
> My take is that gb.test is for running tests in Gambas projects. 
> gb.test.tap is a reusable component for reading and writing TAP.
> 
> Now let me say a few things in order to explain a few problems I have
> with the current gb.test and why I wrote gb.test.tap -- which is not
> intended as an alternative test framework but as a dependency for
> gb.test, to make a part of it reusable and enforce proper
> encapsulation.
> 
> TAP is a two-party protocol. A "test" is a process which loads and
> uses the "system under test" (think a test module inside the
> project), makes assertions about function return values, errors
> happening or not happening, etc. Whether the assertions at runtime
> turn out to be true or not is reflected by "ok" / "not ok" lines
> printed on stdout of the test process, together with other control or
> diagnostic messages that may be helpful to the other side.
> 
> The "test harness" is another process (think gbt3 or the IDE) which
> plans a series of tests, by picking test modules and methods, then
> runs each test, collects its TAP output and analyzes if the test was
> successful.
> 
> gb.test currently mixes these two concerns. It does output TAP, but
> instead of having the other party parse it, it prints a
> "self-summary" in the end, for inspection by I don't know who.
> *Typically*, in my experience, the test process is only concerned
> with outputting honest TAP and then terminates, meanwhile the harness
> reads that TAP and rehashes it for the user. gb.test does not follow
> this separation and at the moment I cannot do that because it does
> not include a general TAP parser.

I agree.

If there is a public protocol, it's because we want two different
processes collaborating through that protocol.

I think too that 'gb.test' must output TAP only, and the IDE (or another 
process) parses the output to display the result, using a 'gb.test.tap' 
component or not (the TAP output seems not very difficult to parse 
directly).

> [...]
> 
> If a *test method* raises an uncaught error, then a *test
> harness*-like piece of code catches it and spoofs a test failure in
> the TAP stream. Arguably that is not correct, semantically. If an
> error is raised and not caught, then the *test code* is buggy, not
> the *tested* code.

I agree.

-- 
Benoît Minisini