[Gambas-user] Integrate unittest component to Gambas

[Adrien Prokopowicz]

Le 13/05/2018 à 19:21, Tobias Boege a écrit :
> 
> The code has been mostly rotting in my repository for the last two(?) weeks,
> but your comments touch exactly the area where I hit a road block and put it
> on the backburner, in favour of uni stuff.
> 
> I need to convert to convert the UnitTest class and all its users to the
> TAP-based classes and possibly redefine how tests are done and the testing
> interface.
> 
>> It isn't as sophisticated as yours (no TAP serialization or setup/teardown),
>> but the interesting bit was that is relied on Tasks to run the tests
>> (instead of running them directly), which has several benefits :
>>
>> - You are 100% sure you go back to the initial memory state between each
>> test (since each test creates a new Task, hence re-forking the main
>> process).
>> - It allows to run tests in parallel on multiple cores.
>> - (Most importantly IMHO) Since each test is ran in a separate process, it
>> is resilient to crashes and segfaults, which makes it ideal for testing
>> native components.
>>
>> If you are interested in this, I could probably send a few PRs towards you
>> to integrate this system. :-)
>>
> 
> I believe my fork [1] is currently the most advanced branch of gb.unittest.

Oh, I completely missed your fork, it seems much more advanced indeed!
I'll give it a more thorough look. :-)

> 
> I'm generally wary of using Tasks because "Many components will not like
> being forked. Especially the GUI ones. So be careful." In addition, we
> already have the TAP architecture in [1], of which you say, if I understand
> it right, that it's more sophisticated. Using TAP, we spawn a new process
> for each unit test, which can contain several assertions, and get output in
> a standard text format, which can be stored on disk and also be read by
> non-Gambas tools. Running multiple tests simultaneously is as easy as using
> GNU parallel.

To be completely honest I haven't read the source code very carefully 
(and as I said I missed your fork), so I missed the part about spawning 
processes. It is indeed much safer (and therefore better) that forking, 
considering forking breaks pretty much everything that is socket-related 
(which includes GUIs, but also DBs, etc.).

> Let me tell you what I imagined in the meantime. This also an RFC for Benoit.
> The project directory structure should get a new official directory .tests/
> or .t/ where tests are located. Tests are classes which inherit UnitTest
> from the gb.unittest component. They contain a bunch of related tests.
> The IDE will handle this directory specially for in-IDE testing and there
> will be a command-line utility "gbprove3" similar to "prove". More signifi-
> cantly, the compiler and archiver should ignore tests by default, as you
> don't want to ship those in production.

I'm the kind of dev who thinks that any production-grade software should 
have some level of automated testing, preferably with a decent code 
coverage. :-)
Therefore I completely agree about having the IDE, compiler and archiver 
deeply integrate testing in them, since basically almost every piece of 
software will need it at some point.

(But please no more cryptic one-letter directory names! I think my 
filesystem will handle the 4 extra chars just fine ;-) )

Another thing to think about is test fixtures and/or test data. For 
instance for testing gb.xml I have (since recently) took the habit of 
keeping track of the various test projects people sent me, so I have a 
project with a bunch of wierd XML files to see if parsing works 
correctly. Having a way to access them from the test classes whithout 
them being shipped with the production code would be nice. :-)

> NB: Since the unit tests are not located inside the unit they test, we only
> test public API. I think that's the more sensible approach by itself and
> it allows for the separation I just mentioned.

Well, if your unit tests don't test internal units, then it's not unit 
testing anymore. :-P

Considering the fact that making assertions and rendering their results 
on a standard interface is not specific to Unit testing but is used 
across all testing levels (like Integration testing, System testing, and 
all the other ones I'm reading on Wikipedia that I've never heard 
about), maybe it would be simpler for everyone to drop the "unit" in the 
name and just call the component gb.test ? I sure wouldn't mind not 
having to deal with "testing level" theory. :-)

> Certainly the current API is too lengthy. The TAP classes provide pretty
> much what you suggest, but there are at least two variations of your
> Assert class. (1) We could exploit the inheritance of the UnitTest class
> in tests and write
> 
>    Me.Equals("Hello", myString)
> 
> This is short and makes sense from an implementation point of view, since
> the TAP state (how many tests are planned, what is the current test number,
> should the next test be tagged as TODO or SKIP, which Stream do we send the
> TAP output to?) is hidden away inside the parent UnitTest class and all of
> that must be accessed by whatever implements your Assertion interface.
> [ NOTE that I'm talking about *my* UnitTest class here, which isn't
> committed yet. ]
> 
> (2) We could also ditch the UnitTest inheritance, put the relevant state
> into a static module and go as far as writing builtin-lookalike assertions:
> 
>    Equals("Hello", myString) ' Equals is a module with a _call method
> 
> but perhaps that's too much, as even if we only load gb.unittest when
> running classes from .tests/, it might conflict with the project's
> class names.
> 
> Those are my thoughts, anyway.

Considering the three proposed variations:

1/ Assert.Equals("Hello", myString)
2/ Me.Equals("Hello", myString)
3/ Equals("Hello", myString)

I still prefer the first one, mostly because what it does looks obvious 
: even when not knowing exactly what's going on, I can tell I'm making 
an assertion of equality. Having a common "Assert" base also makes all 
assertions obvious by scanning through the code, which helps readability.

The second variant makes it look like I'm making an operation on ME, 
which is not the case, and also prevents the use of assertions outside 
of test case classes (which may be desirable if you want to make custom 
test helpers and such).

The third variant is the shortest, but it looks a bit wierd to me, and 
as you said it is best to not have names like "Equals/NotEquals" or 
"Null/NotNull" thrown into the global context (I think having a class or 
module named Null is not possible actually).

-- 
Adrien Prokopowicz