[Gambas-user] New component gb.test

[Tobias Boege]

On Mon, 21 May 2018, Tobias Boege wrote:
> I, on the other hand, am not familiar with those. For me, naively, testing
> is executing scripts that probe the API of your project and report their
> results. How you write those scripts is entirely up to you, as long as
> their output follows the protocol.
>
> [...]
>
> It is kind of hard at the moment to see where the design of gb.test is
> supposed to go and I definitely lack synapses in the usability department,
> which many of the recent remarks were about. If you have experience with
> mature testing methods, please take the lead.
> 
> However, despite your comment about state sharing above, or because
> I didn't understand the problem about it, I still think TAP is a good
> format, regardless of how tests are presented to the user in the end.
> I can't really imagine that the TAP architecture can get this much
> in the way of testing workflows basically everywhere non-Perl.
> 

After taking a shower, my mind has cleared up a bit and I think I could
have said this in a more productive way. The two things I want to preserve
are: (1) tests should be run in a separate process (but they don't have to,
thanks to the OPEN STRING instruction and the stream interface we have),
and (2) they are programs which produce a live TAP stream as they execute.

What I mentioned in the first paragraph quoted above, seems to be more
general than the test containers and methods. If you want to "constrain"
a test file to certain additional behaviours, that should be done by
introducing a base class that is inherited by the test. For example,
TestContainer can be created to manage the "I can run a single of my methods
in isolation, perhaps after setting up an environment" functionality.
Of course, behind the scenes, it has to wrestle with the TapPrinter to
set everything up for that single method to produce a valid TAP stream.
But the only obstacle to that is determining the test plan, i.e. how many
assertions the method is going to run. However, the plan is optional and
may be omitted in this difficult case (the TapPrinter will just count
and print how many tests *were* run in the end, which is also valid).
Also, nothing prevents us from sending arguments to a test, to indicate
which method should be executed. Then you have to teach the test runner
about the calling conventions, perhaps by subclassing or extending
what is currently called TestHarness.

The bottom line here is: not everything needs the (conceptual) overhead
of a TestContainer; some tests might just be scripts that perform a bunch
of trivial sanity checks, or something different entirely. Those tests
which need more framework have to set it up in their own process.
Inheritance seems to be a good tool to do that and spare the user
the writing of boilerplate code.

This is all very dependent on how we want to expose testing facilities,
e.g. in the IDE. At the moment we are far away from "I press F10 and
it runs the test method corresponding to the function under the cursor",
but my *hope* is that the current base classes allow for such extensions,
without it becoming clumsy.

Regards,
Tobi

-- 
"There's an old saying: Don't change anything... ever!" -- Mr. Monk