[Gambas-user] Integrate unittest component to Gambas

[Adrien Prokopowicz]

Le 24/04/2018 à 07:52, Christof Thalhofer a écrit :
> Am 23.04.2018 um 11:08 schrieb Tobias Boege:
> 
>> The trap of distributed development. My fault I didn't look through your
>> other branches. But in the end, it wasn't that much double work. I have
>> a dedicated class to print TAP, which may be worth including, and also a
>> TAP parser. Your work touched my point (3) above, too. I can easily rebase
>> my changes onto yours and consolidate them -- but not before this evening.
> 
> Ok super, it is very good when you have a look at the code. The TAP
> printing function cost me about one hour, it was not that expensive ...
> 
> This week I have not very much time, I couldn't even look at your code
> ... :-(
> 
>> If I understood the code at a glance, there is also a bit of a difference
>> in architecture. You have a TestSuite which contains tests, executes them,
>> buffers the result and prints it TAP-formatted afterwards, whereas I
>> imagined (and am used to from perl) that there are many functions which
>> reduce input data to Booleans (are two strings equal? does this string
>> match a regex? is this object defined and of a certain type? does this
>> code not raise an error? are these arrays recursively equal? and so on)
>> and that Boolean is printed as a TAP "ok/not ok" line, *as you go* through
>> the tests, in case there is an unhandled error and the test crashes.
>> The TAP parser will notice such a crash because either not all tests
>> in the plan "1..N" were executed or the test plan wasn't even printed.
>> It can still tell you about the tests that *were* run, which tell you
>> which one crashed.
> 
> Yes, you got it right, the tests are executed one after another and the
> results are buffered in TestResult (but only the failed ones). Therefore
> I had to restore the sequence by iterating through TestSuite (which
> contains the Tests) in Unittest.PrintTapResult().
> 
> This is not very cool, indeed it would be better if the Unittest would
> print out the TAP results on the fly without buffering.
> 
> I you have a look at the stable code there it is done with events, but
> was delivered to the GUI to the tracer form. Maybe we can take these
> events and deliver to the TAP printer.
> 
> One thing, the current code does right, is: It distinguishes between
> failures (the result is not as expected) and errors and can buffer (not
> so good) errors (this happens in TestCase.Run [59-62]).
> 
> But if the tested code leads to a segfault then all previous test
> results are gone. That's pretty bad and should not happen.
> 
> This week I won't be able to do anything further so feel free to play
> around.
> 
> 
> Alles Gute
> 
> Christof Thalhofer

Hey folks, I know I may be a bit late to the party, but I some (long) 
time ago I also made some experiments to make automated tests for Gambas 
(see here[0]).

It isn't as sophisticated as yours (no TAP serialization or 
setup/teardown), but the interesting bit was that is relied on Tasks to 
run the tests (instead of running them directly), which has several 
benefits :

- You are 100% sure you go back to the initial memory state between each 
test (since each test creates a new Task, hence re-forking the main 
process).
- It allows to run tests in parallel on multiple cores.
- (Most importantly IMHO) Since each test is ran in a separate process, 
it is resilient to crashes and segfaults, which makes it ideal for 
testing native components.

If you are interested in this, I could probably send a few PRs towards 
you to integrate this system. :-)

Also, a few thoughts I had about the Assertion API: in your component, 
to check an assertion you have to do this (my old one has something 
along those lines as well):

Me.Result.AssertEqualsString("Hello", myString)

However, after putting in some thought, not only I find this pretty 
lengthy for assertions (which your tests will be full of), but all in 
all I don't find the "use protected methods from inheritance" very 
Gambas-y (since Gambas does not have a "protected" visibility anyway).

I think a simpler way to do this would be to make a static Assert class 
(see example attached, actual implementation would be more complex 
obviously).
This way, you can call it directly without needing to go through ME or 
other objects :

Assert(myBoolExpression)
Assert.Equals("Hello", myString)
Assert.Same(objRef1, objRef2)
Assert.Null(nullRef)
'… and whatever other helper method we could think of

This makes it much shorter and simpler, and it also makes assertions 
useable outside tests, which could be desirable I think.

Tell me what you think. :-)
Regards,

[0] https://sourceforge.net/p/gambas/mailman/message/33128886/

-- 
Adrien Prokopowicz