[Gambas-user] Questions about gb.test and gb.test.tap

[Benoît Minisini]

Le 25/02/2020 à 01:02, Christof Thalhofer a écrit :
> Am 24.02.20 um 23:20 schrieb Tobias Boege:
> 
>> gb.test tries to do all of that in one process at the moment and this is
>> a lack of separation that facilitates it doing "impure" things. Concretely
>> I mean this
>>
>>    https://gitlab.com/gambas/gambas/-/blob/3752088d37/comp/src/gb.test/.src/TestSuite/TestCase.class#L48
>>
>>    Try Object.Call($MyTestModule, Me.name)
>>    If Error Then
>>      Inc Track.Counter
>>      Track.NOKs.Add(Track.Counter)
>>      Print "not ok " & Track.Counter & " " &
>>      Track.TestModuleName & ":" & Track.TestName & " Raised error: " & Error.Text
>>      Error.Clear
>>    Endif
>>
>> If a *test method* raises an uncaught error, then a *test harness*-like
>> piece of code catches it and spoofs a test failure in the TAP stream.
>> Arguably that is not correct, semantically. If an error is raised and not
>> caught, then the *test code* is buggy, not the *tested* code. Furthermore,
>> this trick isn't always helpful: it only inserts one test failure when
>> the entire test method aborted due to an error at any point, what if the
>> test method had a couple more tests planned? Then the TAP stream will be
>> subsequently shifted and parts will be missing but it continues to run.
> 
> I like your criticism very much! It is wonderful if someone takes a
> close look at the code and the decisions in it.
> 
> Regarding this case:
> 
> I think that is not really of interest (that the testmodule failed in an
> early state, and subsequent testmethods in it may have reported Oks
> later ...).
> 
> At this point I also could have done a BailOut and stop the entire
> testing process instead of just report that an error happened in the
> testmodule. Earlier I think there was a BailOut and that was a bit
> better. But gb.test reports here, where the error happened, in
> "Track.TestModuleName & ":" & Track.TestName" and that's all the
> programmer has to know about. I don't give a damn what happens after that!
> 
> If there is one error/crash the program is buggy and this is, what I
> want to see as a programmer. For sure, the testcode could also be buggy,
> but who cares? The programmer has to fix!
> Either the testee or the tester. That's all. That's the only thing that
> interests me as a programmer.
> 
> The real reason I did that in that way you don't like, was:
> 
> With the current gb.test I can step through the test code and the tested
> code with the debugger *because the processes are not separated*! This
> is very convenient for daily work as a programmer! You step through and
> see: "Oh, my testcode was buggy" or "Oh, my tested code does a silly thing".
> 
> Its the matter of how to work fluently with testing and programming. The
> current gb.test supports my way to program best and it's not because I
> like spaghetti.
> 
> If the testing was done according to your ideas, I could only see that
> something is broken, but I couldn't step through. I would really miss that.
> 
> You complain about that other test systems you know also do not follow
> your idea of "pureness" and "semantically correct" separation.
> 
> I bet, their programmers faced the same problem and also came to the
> same decision as I did here.
> 
> Btw, this is something we discussed a year ago and our positions have
> not changed.
> 
> :-)
> 
> Alles Gute
> 
> Christof Thalhofer
> 

I don't see the point.

Imagine the IDE with a "test" toggle button.

Once the toggle is on, either you run the project with F5, and then you 
run the full project in test mode just to get the full result, either 
you run it in the debugger, and then you can debug your test code and 
find why it crashes.

So 'gb.test' does not have to catch test code errors. The TAP syntax is 
done so that a test can just crash, and it will be detected by the TAP 
analyzer.

Or did I miss something?

-- 
Benoît Minisini