[Gambas-user] Questions about gb.test and gb.test.tap

[Tobias Boege]

On Mon, 24 Feb 2020, Benoît Minisini wrote:
> So I bring the following suggestions:
> 
> 1) Adding a running test option to the interpreter. (Eventually use '-t' and
> find another name for the tracing option).
> 
> 2) The running test option will have an argument that specifies which test
> modules to run (everything by default).
> 
> 3) The interpreter in running test mode will not run the startup class, but
> will load every test module, and will run a global public method in an
> hardcoded global public class that must be implemented by the 'gb.test'
> component. For example "Test.Main". It will receive the class objects in
> argument. This is how it will be able to access non-exported test modules
> and run them.
> 
> 4) The '.test' file created by the compiler is still needed. It will be used
> by the interpreter to find the test modules.
> 

Talking to Christof today and thinking about this, the suggestions are
pretty much what we need. The interpreter flag and that it calls a pre-
defined startup class is quite a nice and uniform solution.

We were thinking about ways to pass a very fine-grained selection of
test modules and test methods to run to gb.test and one possibility is
passing a JSON object with a <test module> to <array of test methods>
mapping. The interpreter would NOT have to read this JSON string,
it would just normally pass this string on to the application as part
of Args, i.e. like this

  gbx3 -t $PROJECT -- $JSON_GIBBERISH  # or -T if -t is taken

The JSON format and its handling then is an implementation detail of
gb.test. Consequently the interpreter would have to pass ALL test
module Class objects to gb.test's Main and the test framework does
the actual selection. What do you think about that?

There should probably be simpler fallback formats for when you want
to run all methods in one named module... You should not be required
to write JSON unless you need maximal flexibility. But that can all
be delegated to command-line arguments handled by gb.test's Main.

The class objects can either be passed as a Class[] or as a Collection,
we need lookup by name and both structures allow this. If you want to
keep the signature of Test.Main() (or UnitTest.Main() as it's called
right now in gb.test) clean, I could also imagine offering this as an
Application.Tests property.

I didn't take notes during the phone call with Christof and then got
distracted for a while, but I think this is all. Or at the very least
it is the next step.

On Tue, 25 Feb 2020, Benoît Minisini wrote:
> Another point: test modules will be available for any project, and gb.test
> may be hidden. At the moment, you have to check the component to be able to
> create test modules.
> 
> When the interpreter will be run in test mode, it will load the 'gb.test'
> component automatically.
> 

There is certainly a precedent for making components hidden and auto-
matically loaded and I think this would fit here. Given the tighter
integration of testing into the interpreter, gb.test could be considered
a "system component". And it should usually not appear in the .project
file, but only be loaded automatically in test mode, I think.

> And I plan to add to the archiver an option that removes the test modules
> from the executable.
> 
> So there will be no overhead when running a project with test modules
> normally, and you will be able to create an executable with no test inside.
> 
> I'm thinking: is it useful to be able to create executables with test
> modules inside?
> 

I could imagine packing an installation package for a distribution,
which will just contain the .gambas file and an install script which
automatically runs tests on the host to verify it's working...
Note sure if that is common practice though. I guess normally the
packager does the testing and supposes the target system is similar
enough.

And finally, thanks for your suggestions. They're on point and I think
we're on a good track together.

Regards,
Tobias

-- 
"There's an old saying: Don't change anything... ever!" -- Mr. Monk