[Gambas-user] New component gb.test

[Tobias Boege]

On Sun, 20 May 2018, Christof Thalhofer wrote:
> Hello Tobi,
> 
> Am 19.05.2018 um 02:44 schrieb Tobias Boege:
> 
> > I am probably starting with the documentation, as I have noticed that
> > the TAP architecture is quite different from the one inspired by JUnit.
> 
> As I now analyze the functionality in TestHarness.Run I see, that it
> executes one test-module at all by doing:
> 
> Exec ["gbx3", "-s", Test, Project] To sOutput
> 
> Then in sOutput there is:
> 
> "1..10
> ok 1 - True is ok
> ok 2 - False is not ok
> ok 3 - Multiplication
> ok 4 - Gambas is sufficiently fast
> ok 5 - Float absolute error
> not ok 6 - Float relative error # SKIP Intentionally wrong
> ok 7 - Float relative error
> ok 8 - CStr returns a String
> ok 9 - LIKE email address
> ok 10 - MATCH email address
> ok 11 - not MATCH email address"
> 
> So far so good. Everything seems to be fine.
> 
> But I have a couple of problems with this approach compared to the one
> inspired from JUnit.
> 
> 1st:
> --------------------------------------------------------------------
> 
> In the original code there was the concept of "fixtures" implemented
> which is now gone.
> 
> Please look at https://github.com/Deganius/gb.deg.unittest/ and scroll
> down to "Test fixture"
> 
> A fixture is the possibility of creation and (after the test was run)
> deletion of a special environment. A fixture could be created for each
> single test or for a bunch of tests enclosed in a single module.
> 
> For example like so:
> ----------------------------------
> create fixture (create environment)
>     run testfunction 1
>     run testfunction 2
>     ...
> end fixture (destroy environment)
> ----------------------------------
> 
> Or so:
> ----------------------------------
> create fixture (create environment)
>     run testfunction1
> end fixture (destroy environment)
> create fixture (create environment)
>     run testfunction1
> end fixture (destroy environment)
> ...
> ----------------------------------
> 
> Or both combined.
> 
> As your code now executes a module at all without looking at the content
> these possibilities are gone.
> 
> This functionality (analyzing the testmodule and creating so called
> testcases out of the methods contained) was done in Unittest.class,
> executing them was done in TestSuite.Run() together with TestCase.Run()
> 

The ["gbx3", "-s", Test, Project] is a stopgap until it's clear how much
convenience we want to code into the interpreter for running tests.
Currently a test must be a startup-able class with a Static Public Main,
and I couldn't say it doesn't make sense to me this way.

The perspectives on testing in this thread alone are varying wildly and
I enjoy that very much. As for fixtures, you are right that they aren't
managed by the component anymore. Again, my design is inspired by perl:
a test is a script which you hand over control to to execute some tests
and print the results. It's procedural and autonomous, as opposed to your
idea of making it an object-oriented building block which fits into the
component's framework and is managed by it.

To set up a fixture, put the code to set it up inside your test module,
afterwards do the tests. Remember that you can even outsource and share
code between your test modules, since they are just classes inside the
same project.

But fixture and data handling are also a TODO item with respect to
compiler/interpreter support, so point out any flaws and awkwardness.
Looking at your second example, I agree that it's clumsier to lose
the separation of test code and environment, but it's only a few lines
more to set up and tear down environments on your own, if you already
have the environment definitions.

To fix this, we could re-instantiate the rule that test modules have
to inherit some base class for testing, which takes care of setting up
and varying fixtures for the current test module. As you briefly
mentioned, the technical problem here is that the tester can't inject
variables or function calls into the testee's execution, so the testee
has to create the fixtures themselves. (Unless we abuse debugging mode
but does anyone besides Benoit know how that even works?)

To be a bit more specific, I imagine the following base class:

  ' Test.class
  Static Public Sub Main()
    For Each Me.Fixtures
      Me.Test()
    Next
  End

Now, a test module must provide a Test method, instead of Main.
It can also provide a Fixtures property which returns a special
object, which is enumerable by For Each. The implementation of
_next() takes care of setting up and tearing down fixtures.
Test.class provides a single, empty fixture by default, so that
the Test() method is run once. Variations of this are possible,
like not being too clever with _next() and using an array of
TestFixture objects from Me.Fixtures instead, which have a
Setup() and Teardown() method.

> 2nd:
> --------------------------------------------------------------------
> 
> If I want a test function do intentionally throw a Gambas error, it
> seems to me that this is now impossible, because the Gambas error would
> stop further processing. Am I right?
> 
> In the original code this functionality together with the one above was
> done in TestResult.AssertError
> 

You're right, but in this case my recipe would be:

  Try ShouldFail()
  Assert.Ok(Error, "failed like it should")

You could even make Assert.Ok(Error, ...) into a new method Assert.Error(...)

> --------------------------------------------------------------------
> 
> Im am quite unsure how this functionality could be restored with this
> current approach:
> 
> Exec ["gbx3", "-s", Test, Project] To sOutput
> 
> which I find very interesting as a segfault could be detected and would
> not destroy the process running the tests.
> 
> Any ideas?
> 

I can't think of any amount of cheating that would make

  Assert.Error(ShouldFail(), "failed like it should")

work since Gambas doesn't have closures, for instance, but to me
Try seems good enough.

Regards,
Tobi

-- 
"There's an old saying: Don't change anything... ever!" -- Mr. Monk