[Gambas-user] New component gb.test

[Adrien Prokopowicz]

Le 21/05/2018 à 20:39, Christof Thalhofer a écrit :
 > This was not my implementation

Whoops, I meant to reply to Tobias, to whom the "your" was addressed, sorry!

Le 21/05/2018 à 21:28, Tobias Boege a écrit :
> On Mon, 21 May 2018, Adrien Prokopowicz wrote:
>> There's something I just noticed in your implementation: why is a "test" (in
>> TAP terminology) mapped to a single assertion, and not to a whole test
>> method ?
>>
> 
> There are two things that might be called a "test". One is a single
> assertion, which produces an "ok" or "not ok" line, and the other is
> multiple assertions bundled into a file -- that file would be called
> a test, too, if you ask me.
> 
>> The problem with this approach is that all assertions share state, which
>> should not be the case with tests: ideally a test should have no influence
>> on subsequent tests, otherwise a test following a failure could be either a
>> false positive or a false negative, and you can't really test things in
>> isolation.
>>
> 
> I think that depends. If you create an object and want to test two methods,
> these two assertions should share state. Or what do you mean?

I believe we have a terminology issue here. For me, a bunch of 
assertions make a Test (represented as a method), and a bunch of tests 
are grouped in a Test Module for convenience.


> 
>> After reading the TAP/Perl spec a bit more thoroughly, it seems they also
>> use this approch, which I find really wierd. Admittedly i've never ever
>> encountred Perl, but of all the testing tools I've seen and used (I've
>> looked into C, Rust, Ruby, Java, C# and Javascript for my testing) I've
>> never seen this approach before.
>>
> 
> I, on the other hand, am not familiar with those. For me, naively, testing
> is executing scripts that probe the API of your project and report their
> results. How you write those scripts is entirely up to you, as long as
> their output follows the protocol.

I also view testing as "executing scripts that probe the API of your 
project and report their results".

But then, to me it's more like "Here is a general structure to follow, 
use these building blocks (Assert etc.) to write your tests, and we'll 
do the rest". :-)

> 
> And that is about the level at which I worked. I just implemented TAP
> because Christof mentioned it, I recognised it and thought it was a
> good idea. I don't know if that was clear to you, but the "higher-level"
> stuff was mostly duct-taped on top to be able to run something that
> produces and consumes TAP and I took inspiration for that from perl.
> That part is very much intended to evolve. [ Maybe my deleting all
> those COMUnit classes gave away the impression that I replaced them,
> but I really just didn't want to spend time porting them to the new
> TAP base, and instead implemented a fresh and small TestHarness class
> to have something presentable. Sorry, Christof :-) ]
> 
> It is kind of hard at the moment to see where the design of gb.test is
> supposed to go and I definitely lack synapses in the usability department,
> which many of the recent remarks were about. If you have experience with
> mature testing methods, please take the lead.

Since I am better at writing code than words, I made an example snippet 
to present what I think of a gb.test API:

https://gambas-playground.proko.eu/?gist=77867b16bb9449b5375d62eabe71594c

Comments are welcome, as always. :-)

> However, despite your comment about state sharing above, or because
> I didn't understand the problem about it, I still think TAP is a good
> format, regardless of how tests are presented to the user in the end.
> I can't really imagine that the TAP architecture can get this much
> in the way of testing workflows basically everywhere non-Perl.

Don't get me wrong, I like TAP a lot! Mainly because it's simple to read 
as an human, but also to produce and parse (yay non-XML!), and also it's 
widely supported by various tools, including popular CI systems like 
Jenkins.

I just don't think TAP should leak into the user API for testing, mainly 
because of the idea of dependency to a specific format (most Test 
Harnesses out there support multiple formats as outputs), but also 
because API-wise, I don't want to feel like I'm calling a writer for a 
format while I'm writing hundreds tests, because those two things are at 
completely different level of abstaction.

Or, put it in another way, I shouldn't write my tests in TAP, I should 
write them in Gambas. ;-)

> 
> PS: I should also note that I have to gradute in a few months and can't
> spend much time researching testing and evaluating proposals. The frame
> is really tight this time.
> 

Don't worry too much about it, it's not like we're going to merge and 
release this next week anyways!

Considering this is ultimately quite a big feature in terms of user 
expericence, but quite small in terms of API and implementation work, I 
think it's preferable to spend weeks discussing it so we can then use it 
in most of Gambas code (and in all of Gambas itself) for the next 10+ 
years without regrets. :-)

And good luck for your exams!

-- 
Adrien Prokopowicz