[Gambas-user] Coming back to gb.test

[Christof Thalhofer]

Am 03.04.20 um 13:11 schrieb Tobias Boege:

> I fully agree with the necessity of testing abnormal conditions.
> If you want to test whether your library throws errors appropriately,
> then write an actual test for it in your test code:
> 
>   Try MyLibrary.Method("invalid")
>   Assert.Ok(Error, "error happened as expected")
>   Assert.Match(Error.Text, "Method cannot process 'invalid' input", "error message is correct")

This is a case I don't talk about.

> That's all that's being debated here: move the blanket Try *around*
> a test method *into* the test method. Test specifically for errors
> that you know should be thrown under this and this condition.
> 
> If gb.test has to catch errors on the programmer's code paths, then
> that means that they are *unexpected* errors which slip through the
> programmer's test methods. Errors from the library which the test code
> does not explicitly anticipate. To me that means that the test code
> is buggy (expects things to work that don't) or that the library is
> buggy (doesn't work when it should).

Yes. I wrote about this already in February:

https://lists.gambas-basic.org/pipermail/user/2020-February/068729.html

> To a certain extent that's ok. Test code can have bugs too which need
> to be debugged when they occur. I feel like your catching of all errors
> in gb.test is a way to deal with unintentionally broken tests -- tests
> which don't catch all the errors they provoke -- and get some information
> about what happened in order to fix it. But as I said, letting errors
> escape and crash the interpreter will allow you to run the debugger on it
> and get more information directly from the source of the error than
> catching it in gb.test can give.

Look. With the original gb.test we were able to test it's own behaviour
in cases of errors and crashes. With a small possible change (reverse
the following assertion, false becomes true) we could do this
automatically but not only for gb.test itself but for every code tested.

Now this functionality is gone. If you want to test if in any project in
future a certain behavior leads to a crash you can't do this any more. I
know that this is a seldom case as a programmer always wants to create
code that does not crash, but who knows ...?

The only reason why it is gone is, that Asserts are now in gb.test.tap.

This in my opinion is – and it gets clearer the more I think about it –
the wrong decision.

And the decision before – which you didn't like as I understood mostly
for aesthetical reasons –  that gb.test tested and printed TAP and also
did accounting and bookkeeping, was right.

It was not only right because of my arguments above but also the
separation of functionality into two components leads to much more
complexity (which also can be seen in your flowchart), reduces
flexibility and hinders debugging.

> The norm should be that test methods don't raise errors of their own:
> 
>   - If a test decides that it needs to stop immediately because
>     something is really bad, then it has the option of BailOut
>     in the protocol.
> 
>   - Test code should also not provoke errors in the tested code,
>     except for the purpose of catching them and making Assert'ions
>     about them.

There is no norm. Test are there to find out the not normal.

Again, to solve this and make things easier, my proposal is: Let's move
your Assert class and also accounting to gb.test and restore it's old
behavior so that it prints out TAP, a plan and an optional summary
without any need for an additional component.

In short: Merge gb.test and Assert as well as the printing and
accounting part of gb.test.tap.

Then let's create TAP eaters that can do anything with it, display in
GUI, write to log or whatever we want. For that a lot of code already
exists in in gb.test.tap.

As since February we are going around in circles on this question I
think, Benoît should have a look at it and decide.

Alles Gute

Christof Thalhofer

-- 
[x] nail here for new monitor