[Gambas-user] Integrate unittest component to Gambas

[Jussi Lahtinen]

How to test GUI widgets automatically?


Jussi

On Sun, May 13, 2018 at 8:21 PM, Tobias Boege <taboege at gmail.com> wrote:

> On Sun, 13 May 2018, Adrien Prokopowicz wrote:
> > > Yes, you got it right, the tests are executed one after another and the
> > > results are buffered in TestResult (but only the failed ones).
> Therefore
> > > I had to restore the sequence by iterating through TestSuite (which
> > > contains the Tests) in Unittest.PrintTapResult().
> > >
> > > This is not very cool, indeed it would be better if the Unittest would
> > > print out the TAP results on the fly without buffering.
> > >
> > > I you have a look at the stable code there it is done with events, but
> > > was delivered to the GUI to the tracer form. Maybe we can take these
> > > events and deliver to the TAP printer.
> > >
> > > One thing, the current code does right, is: It distinguishes between
> > > failures (the result is not as expected) and errors and can buffer (not
> > > so good) errors (this happens in TestCase.Run [59-62]).
> > >
> > > But if the tested code leads to a segfault then all previous test
> > > results are gone. That's pretty bad and should not happen.
> > >
> > > This week I won't be able to do anything further so feel free to play
> > > around.
> > >
> > >
> > > Alles Gute
> > >
> > > Christof Thalhofer
> >
> > Hey folks, I know I may be a bit late to the party, but I some (long)
> time
> > ago I also made some experiments to make automated tests for Gambas (see
> > here[0]).
> >
>
> The code has been mostly rotting in my repository for the last two(?)
> weeks,
> but your comments touch exactly the area where I hit a road block and put
> it
> on the backburner, in favour of uni stuff.
>
> I need to convert to convert the UnitTest class and all its users to the
> TAP-based classes and possibly redefine how tests are done and the testing
> interface.
>
> > It isn't as sophisticated as yours (no TAP serialization or
> setup/teardown),
> > but the interesting bit was that is relied on Tasks to run the tests
> > (instead of running them directly), which has several benefits :
> >
> > - You are 100% sure you go back to the initial memory state between each
> > test (since each test creates a new Task, hence re-forking the main
> > process).
> > - It allows to run tests in parallel on multiple cores.
> > - (Most importantly IMHO) Since each test is ran in a separate process,
> it
> > is resilient to crashes and segfaults, which makes it ideal for testing
> > native components.
> >
> > If you are interested in this, I could probably send a few PRs towards
> you
> > to integrate this system. :-)
> >
>
> I believe my fork [1] is currently the most advanced branch of gb.unittest.
>
> I'm generally wary of using Tasks because "Many components will not like
> being forked. Especially the GUI ones. So be careful." In addition, we
> already have the TAP architecture in [1], of which you say, if I understand
> it right, that it's more sophisticated. Using TAP, we spawn a new process
> for each unit test, which can contain several assertions, and get output in
> a standard text format, which can be stored on disk and also be read by
> non-Gambas tools. Running multiple tests simultaneously is as easy as using
> GNU parallel.
>
> Let me tell you what I imagined in the meantime. This also an RFC for
> Benoit.
> The project directory structure should get a new official directory .tests/
> or .t/ where tests are located. Tests are classes which inherit UnitTest
> from the gb.unittest component. They contain a bunch of related tests.
> The IDE will handle this directory specially for in-IDE testing and there
> will be a command-line utility "gbprove3" similar to "prove". More signifi-
> cantly, the compiler and archiver should ignore tests by default, as you
> don't want to ship those in production.
>
> NB: Since the unit tests are not located inside the unit they test, we only
> test public API. I think that's the more sensible approach by itself and
> it allows for the separation I just mentioned.
>
> [1] https://github.com/taboege/gb.deg.unittest/tree/TAP
>
> > Also, a few thoughts I had about the Assertion API: in your component, to
> > check an assertion you have to do this (my old one has something along
> those
> > lines as well):
> >
> > Me.Result.AssertEqualsString("Hello", myString)
> >
> > However, after putting in some thought, not only I find this pretty
> lengthy
> > for assertions (which your tests will be full of), but all in all I don't
> > find the "use protected methods from inheritance" very Gambas-y (since
> > Gambas does not have a "protected" visibility anyway).
> >
> > I think a simpler way to do this would be to make a static Assert class
> (see
> > example attached, actual implementation would be more complex obviously).
> > This way, you can call it directly without needing to go through ME or
> other
> > objects :
> >
> > Assert(myBoolExpression)
> > Assert.Equals("Hello", myString)
> > Assert.Same(objRef1, objRef2)
> > Assert.Null(nullRef)
> > '… and whatever other helper method we could think of
> >
> > This makes it much shorter and simpler, and it also makes assertions
> useable
> > outside tests, which could be desirable I think.
> >
>
> Certainly the current API is too lengthy. The TAP classes provide pretty
> much what you suggest, but there are at least two variations of your
> Assert class. (1) We could exploit the inheritance of the UnitTest class
> in tests and write
>
>   Me.Equals("Hello", myString)
>
> This is short and makes sense from an implementation point of view, since
> the TAP state (how many tests are planned, what is the current test number,
> should the next test be tagged as TODO or SKIP, which Stream do we send the
> TAP output to?) is hidden away inside the parent UnitTest class and all of
> that must be accessed by whatever implements your Assertion interface.
> [ NOTE that I'm talking about *my* UnitTest class here, which isn't
> committed yet. ]
>
> (2) We could also ditch the UnitTest inheritance, put the relevant state
> into a static module and go as far as writing builtin-lookalike assertions:
>
>   Equals("Hello", myString) ' Equals is a module with a _call method
>
> but perhaps that's too much, as even if we only load gb.unittest when
> running classes from .tests/, it might conflict with the project's
> class names.
>
> Those are my thoughts, anyway.
>
> Regards,
> Tobi
>
> --
> "There's an old saying: Don't change anything... ever!" -- Mr. Monk
>
> --------------------------------------------------
>
> This is the Gambas Mailing List:
> https://lists.gambas-basic.org/listinfo/user
>
> Search the list:
> https://lists.gambas-basic.org/cgi-bin/search.cgi
>
> Hosted by https://www.hostsharing.net
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gambas-basic.org/pipermail/user/attachments/20180513/6b2bc3ce/attachment-0001.html>