[Gambas-user] Crash with latest version

[Tobias Boege]

On Wed, 10 Feb 2021, Jussi Lahtinen wrote:
> You can even use the standard format used by 'gb.test' component.
> >
> > (I even guess Christof would tell you that it should be rewrite as a
> > 'gb.test' test suite!)
> >
> 
> If I remember correctly, the test suite has flaw that make me not want to
> use it.
> It does run the tests as separate processes..? That greatly increases
> chances that possible side-effects will *not* be exposed.
> 

I sense a misunderstanding between Christof and you, so let me ask clearly:
What kind of side effects do you need?

Because gb.test does *not* start a new process for every test unit, as I
think that you think. All tests run in the same process. What you maybe
remember is that there are two processes involved in testing: a program
that runs the tests and a program (usually the IDE) that captures and
parses the output of the test running process and judges whether the test
passed or failed.

Your GambasTester does both at the same time: (1) It is a Gambas program
which contains tests and its mTest arranges for the tests to be run, and
(2) it judges whether the entire test run was successful.

In the gb.test model, (1) and (2) are separated. But test code can still
create side effects for other test code, as it all runs in the same
interpreter process. For instance, if one test sets a global variable,
all tests running afterwards see the changed value.

gb.test can even allow you to reflect on the test progress, if you are
talking about that kind of side effect (I want this test assertion to fail
if the last 100 tests succeeded -- but I doubt you are talking about this
and I am not sure if this is implemented currently in gb.test, but it
could be done easily by exposing the internal bookkeeping).

gb.test is a component that you can load in your program which takes care
of *organizing*, *labeling*, *running* tests as well as provide an API to
test code for reporting success and failure of a test. It will also catch
any **uncaught** errors in your test code and report them as failures, so
that a test run continues as long as possible, but this all happens in the
same single process.

It basically just imposes some structure on test code which allows a user
to select only a subset of "interesting" tests to run in one go. It manages
the execution of your code. Its API also emits test results in a standard
format that can be stored, transmitted and parsed.

What gb.test cannot do: catch segfaults and such really bad interpreter
crashes and report them as failed tests. I take it that your tester cannot
do this either. However, because of the separation of test runner (1) and
test harness (2) in gb.test, the harness (i.e. the IDE) will detect even
segfaults and report them as unsuccessful test runs. I suppose in your
tester, the harness are the two eyes of the user in front of the screen.

With gb.test, your project is started with `gbx3 -T`, so the management
logic for test running of gb.test kicks in. With your GambasTester, you
start it with `gbx3` and it organizes itself through mTest.Main().

>From a glance at your code, I don't think that it is incompatible with
gb.test, but what I looked at is not enough for me to answer the question
I posed above: what side effects are you talking about that could prevent
it from being ported to gb.test?

Regards,
Tobias

-- 
"There's an old saying: Don't change anything... ever!" -- Mr. Monk