[Gambas-user] New component gb.test

[Adrien Prokopowicz]

Le 22/05/2018 à 10:57, Christof Thalhofer a écrit :
> We should talk about the way how the code should be executed. Maybe we
> need some magic here. Can (or should) we hand out the code to a separate
> interpreter process? How?

Here's a flowchart of how I (ideally) view the different internal parts 
interacting:

                                   +-----------+
+--------------+      Starts      |           |
|              | +--------------> |           |
|  CI System   |                  |  gbtest3  | <--> CLI User
|  Gambas IDE  |                  |           |
|              | <--------------+ |           |
+--------------+   Test Results   +-----------+
                    (TAP stream)
                                      +     ^
                                      |     |  Returns test result
                     Starts processes |     |  (Exit code)
                        (gbx3 -s …)   |     |  (STDERR for errors)
                                      v     +

                                +------------------+
                                |-------------------+
                                ||                  |
                                ||   Test Methods   |
                                ||    (gb.test)     |
                                ++                  |
                                 +------------------+

(The separation of gbtest3 aside, this flow is quite similar to Tobi's 
implementation, but with some notable differences I'll cover here)

The needed magic resides in the "-s" flag of the interpreter (gbx3), 
which sets a new startup class for running the project (different from 
the MMain/FMain class set by the user).

We can set this flag to an hidden class exported by gb.test (for example 
_TestRunner), which will run its Main() method directly, but in the 
scope of the user's project. :-)

 From there, this hidden class can do two different things (dictated by 
CLI arguments, sent by gbtest3).

First, walk through every class file in the hidden ".gambas" directory, 
keep only the classes that inherit from TestModule/TestContainer (name 
still up to bikeshedding, I prefer the first personally), then list all 
the callable test methods. Then this list of tests is sent back to 
gbtest3 to make the test plan before any test is run.
While this plan is not strictly necessary for the TAP stream (but still 
nice to have, according to the spec), it enables gbtest3 to efficiently 
split the testing work across multiple process running in parallel. It 
also enables the IDE (and external CI tools maybe) to show a nice 
progress bar while the user waits for their tests to run, since the TAP 
output is streamed and not buffered.

Then, when the plan is set up, the gbtest3 executable will start a new 
runner process, instructing it to run a single test and return the 
result (and repeat for evey test in the plan).
Here Tobi's implementation communicates with the test harness (here 
gbtest3) using TAP as well, but since in this model a process lives long 
enough to only run a single test, all TAP outputs would have only one 
results, essentially making TAP useless for this case, which is why I 
rather chose standard exit codes for status (ok/not ok/segfault/etc.), 
and looking through STDERR for the error message (if any).
It also makes the implementation of gb.test much lighter, which is 
always a plus. :-)

This whole model however, has an idea different from the one used by 
Perl (which inspired TAP) and implemented by Tobi here, and much closer 
the ones I see used day-to-day in production at work (whatever the 
language), which I believe is why there were some misunderstandings in 
the previous messages.

It is based on the fact that, semantically, an Assertion is not a just a 
test or check, but a strong and obvious affirmation about how the tested 
system works.
Assertions that succeed are simply ignored and do nothing (i.e. they do 
not show up in any log whatsoever), but if an assertion happens to fail, 
it aborts the current test (or throws an error, in Gambas terminology), 
therefore not running any subsequent assertions or code, and immediately 
marking the test as failed.

A few examples of this model can be found in JUnit's Assert class[0] or 
in Rust's assert! macro[1] (among many others, those are just the ones 
that come to my mind right now).

I believe this is better than Perl's approach of running and logging 
every assertion, because assertions are supposed to be very lightweight 
instructions (in the "good" case at least, where they succeed), which 
enables you to put a lot of them whithout slowing down your test suite 
nor spamming your test log (like in long loops, or testing big chunks of 
data).
And also because when testing, you don't care to the assertions that 
succeeded, you only care about the ones that failed. :-)

(Note this is only about assertions, all test methods that ran are of 
course logged in the TAP output).

> What we also should talk about is, how a couple of tests could be
> arranged (organized) in GUI and/or code.

The GUI should be completely decorrelated to how everything works 
internally and in code (which is what Benoît's initial remarks were 
about), so its presentation shouldn't be a concern right now.

[0] http://junit.sourceforge.net/javadoc/org/junit/Assert.html
[1] https://doc.rust-lang.org/std/macro.assert.html

-- 
Adrien Prokopowicz