On Sun, 05 Apr 2020, Christof Thalhofer wrote:
> Am 03.04.20 um 01:30 schrieb Tobias Boege:
>
> > Christof has reminded me what the critical points are:
> >
> > On Thu, 02 Apr 2020, Christof Thalhofer wrote:
> >> ----------------------------------------------------------------
> >> 1) Assert is now in gb.test.tap and not in gb.test. But this makes no
> >> sense because asserts must not be associated to the output format. In
> >> addition the flow of testing and assertions cannot be debugged inside
> >> gb.test any more.
> >>
> >
> > I agree with this. gb.test must contain the Assert module which provides
> > "high-level" assertions like comparing strings and printing diagnostics
> > when the test fails ("expected this string but got that string").
> >
> > But gb.test.tap should still export a *minimal* Assert module that just
> > contains Ok, Diag, BailOut, Todo, Skip and Subtest functionality.
> > gb.test builds its assertions on top of these primitives by extending
> > the Assert module. This allows TAP to be switched out for another format
> > as long as the other format's Assert module provides the above primitives.
> >
> > I think Benoît also suggested something along those lines, so we agree.
> > D'accord?
>
> The more i think of it my doubts grow.
>
> The test system core must have to be able to test itself to ensure it's
> reliability. Are "Ok, Diag, BailOut, Todo, Skip and Subtest" not
> essential parts of this core?
>
> If one of them swallows things this is as bad as if an assertion does
> something wrong.
>
> I would like it better if the test system is reliable in itself and
> spits out its results in the form of TAP (as transport format).
>
> If we do the separation for fictional Gui, CVS, Log ... why can't they
> take TAP and convert it into their special output format?
>
I have absolutely no problem with committing to the TAP format, quite the
opposite. IIRC allowing CSV, JUnit XML or writing directly to syslog were
just things we noticed fell out of the separation. Nobody, including me,
had plans to use that flexibility, and compatibility of the interface has
never been thought about. The GUI part in the IDE has always been based
on reading TAP in my mind because that is what'll be available in v3.15.
That gb.test.tap is a separate component implies that it can't test
itself, but it can be tested using gb.test like every other component.
The self-tests of gb.test exercise gb.test.tap in conjunction with
gb.test, too, and they show failures, don't they?
Why I object to including gb.test.tap into gb.test is:
- I want to make gb.test into a "system-level" component which is
loaded automatically by the interpreter when the project is run
in "test mode" (and otherwise not normally used). gb.test's Main()
takes over the command-line arguments (which describe which tests
to run) and calls into specific parts of the host project. Sort of
like gb.httpd. The reason for this is that some form of tighter
coupling with the runtime is *needed*: tests have already become
special kinds of classes which, in the future, need to be available
in one context (test mode), but more often not. And since gb.test
is part of the official Gambas source tree now and has the right
name, we might as well elevate its status to *the* thing that the
interpreter gives the scepter to when "test mode" is requested.
The classes in gb.test are never directly used by any project. They
have to be loaded only when someone from the outside says "I want to
execute the tests in this project" and gb.test controls how that is
done from inside the process that is running the project.
- gb.test.tap is an application-level component. It is a general
implementation of TAP. It is used by test code to print TAP and
it is used by the IDE and gbt3 to read TAP coming out of the
test process. It can be used in all applications written in Gambas
that need to read and/or write TAP. None of these applications
have to do with administering and running tests in the current
Gambas project, which is what gb.test specializes in.
You see, from the aesthetical viewpoint that you have critized before,
I am well aware, the merging of gb.test.tap into gb.test is like merging
the CGI class into gb.httpd.
Now, you can attack this viewpoint from multiple angles: it starts with
the premise that gb.test should become a system component at all --
it can be done differently for sure. An early version of test mode was
`gbx3 -s UnitTest` which works even if tests are not becoming an explicit
concern of the runtime. Benoît would have an opinion on this. Maybe he
even thinks that system components are just a necessary evil sometimes
and we shouldn't do that here. You can also say that nobody would care
if they import a framework that can run tests in the current Gambas
project when they really just want to create a script that parses some
TAP from a C++ program. Importing stuff you don't need is quite common
with components after all, and the test running parts don't even have
to be exported. A balance has to be found.
By merging gb.test.tap into gb.test you are implying the last argument
above, that it is fine that gbt3 and the IDE will depend on gb.test
for parsing TAP and pull in classes that are never used during normal
operation. But I haven't seen you explicitly argue this way because
you seem to be focused on the integrity of gb.test's source code and
not the whole picture of testing all around Gambas. That the decision
to merge TAP into gb.test is a trade-off doesn't become apparent until
you need TAP processors in multiple parties involved in the testing.
What I have sketched in my flowchart came from adapting the TAP-based
testing architecture in Perl to the constraints and abilities of Gambas,
what I perceived to be solutions familiar in Gambas, like the system
component concept which seems natural when .test becomes a recognized
file type all over compiler, interpreter, archiver and IDE, and gb.test
becomes the interpreter's extended arm for testing concerns.
You said you dislike the basic idea of my flowchart, which is the basis
of all the work I have done and intended to do. (Contrary to what it may
seem like, I'm not just going around changing working components with no
bigger plan!) Some code I wrote can be incorporated into the final testing
system but there is no bad blood if it's all going to be thrown away. So
I ask which parts of the flowchart would you change and into what provided
that you can only zoom further out (i.e. consider the interplay of *more*
things, not less)? Change all mentions of "gb.test.tap" into "gb.test",
meaning that they are merged? I'm not categorically opposed to this, see
the counterarguments after my list of objections above! It would be a fairly
simple thing to do. But I think it is important that we're arguing all pros
and cons from the same zoom level.
Pros:
- Everything in one place. Easier to debug.
Cons:
- Adds dead code to everything that wants to use TAP.
Neutral:
- Ties gb.test to the TAP output format.
Arguments I don't understand:
- Allows gb.test to do accounting in the test process while running.
--> What is this good for? And gb.test.tap could expose this, too.
- Makes TAP printing testable in the first place.
--> gb.test.tap can be tested like everything else. gb.test's self-
tests also test compatibility of gb.test.tap with gb.test.
Anything I missed?
Regards,
Tobias
--
"There's an old saying: Don't change anything... ever!" -- Mr. Monk
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.gambas-basic.org/pipermail/user/attachments/20200405/bb6b5355/attachment-0001.sig>