[Gambas-user] Coming back to gb.test

[Tobias Boege]

On Fri, 03 Apr 2020, Christof Thalhofer wrote:
> Am 03.04.20 um 08:09 schrieb Christof Thalhofer:
> 
> > I did not write this test just for a joke.
> 
> And by the way. While developing this feature of gb.test I found a
> failure in Gambas' Error.Clear, which did not clear the error number
> correctly. This bug is meanwhile fixed.
> 
> To find errors, to detect crashes, to be able to control situations
> where a program crashes and to test it is an important part to ensure
> stability of code.
> 
> For instance your gb.openssl or any other security relevant program: I
> can imagine that there are situations, where you would prefer to stop
> the program immediately. Then you would be glad if you are able to test it.
> 

I fully agree with the necessity of testing abnormal conditions.
If you want to test whether your library throws errors appropriately,
then write an actual test for it in your test code:

  Try MyLibrary.Method("invalid")
  Assert.Ok(Error, "error happened as expected")
  Assert.Match(Error.Text, "Method cannot process 'invalid' input", "error message is correct")

That's all that's being debated here: move the blanket Try *around*
a test method *into* the test method. Test specifically for errors
that you know should be thrown under this and this condition.

If gb.test has to catch errors on the programmer's code paths, then
that means that they are *unexpected* errors which slip through the
programmer's test methods. Errors from the library which the test code
does not explicitly anticipate. To me that means that the test code
is buggy (expects things to work that don't) or that the library is
buggy (doesn't work when it should).

To a certain extent that's ok. Test code can have bugs too which need
to be debugged when they occur. I feel like your catching of all errors
in gb.test is a way to deal with unintentionally broken tests -- tests
which don't catch all the errors they provoke -- and get some information
about what happened in order to fix it. But as I said, letting errors
escape and crash the interpreter will allow you to run the debugger on it
and get more information directly from the source of the error than
catching it in gb.test can give.

The norm should be that test methods don't raise errors of their own:

  - If a test decides that it needs to stop immediately because
    something is really bad, then it has the option of BailOut
    in the protocol.

  - Test code should also not provoke errors in the tested code,
    except for the purpose of catching them and making Assert'ions
    about them.

Regards,
Tobi

-- 
"There's an old saying: Don't change anything... ever!" -- Mr. Monk
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.gambas-basic.org/pipermail/user/attachments/20200403/258b6b48/attachment.sig>