<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div><style></style></div><div>I have been implementing a test framework for scripter, and some of my own apps from my experience so far these are some things I have found difficult and thus the suggestions.</div><div>Please understand that I really do appreciate the work that has been done and and the functionality it provides. This is a bit long!</div><div> </div><div>I would like to suggest the following be added to test framework if possible. It would make unit testing of the application and its modules a little simpler.</div><div><br data-mce-bogus="1"></div><div>Assert.Startup("startupModExpected") ' used to verify you have not forgotten to set to real startup ... lol</div><div>Assert.Exported("namelist",......... ) ' list of expected exported modules and classes fails if don't match</div><div>Assert.Interface("modOrClass", "interfaceFuncOrVarsOrProperty".....) ' test exported mods or classes that the interface is correct ie public functions and variables exposed, returns fail if they don't match.</div><div><br data-mce-bogus="1"></div><div>AssertStdOut.Equal(Function(), "Expected",".....")</div><div>AssertStdErr.Equal(Function(), "Expected,"....")</div><div><br data-mce-bogus="1"></div><div><span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">Now I have to explain that most of the time, I replaced the 'got value' with the actual function call from the tested module</span></div><div><span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><br data-mce-bogus="1"></span></div><div><div style="clear: both;" data-mce-style="clear: both;"><span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">Is it possible to have an</span></div><div style="clear: both;" data-mce-style="clear: both;"><span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><br data-mce-bogus="1"></span></div></div><div>AssertWithTimeout.equal(time, OpendbInput("parm"),"Timeout","The Called func Times out and returns an error from function") ' function times out and returns error before the assert timeout happens</div><div>Does something along the lines of pressing stop in the ide.... if timeout expires.</div><div><br data-mce-bogus="1"></div><div>Or have an optional setting test.SetTimeout(time) such that if a call hangs it does not end the sequence of tests.</div><div>For situations where a bad function is really bad or the test module itself fails.</div><div><br data-mce-bogus="1"></div><div>Having a macro/switch which prevents an app from exiting on quit/etc when testing error conditions generated for the interface of modules. And so ensure errors are being handled correctly.</div><div>Perhaps test should do by default.</div><div><br data-mce-bogus="1"></div><div>e.g.</div><div>Replaced the 'got value' with the actual function call from the test module</div><div>test.exitDisable()</div><div><br data-mce-bogus="1"></div><div> AssertStdOut.Equal(main(), "Operation Competed OK", "test if module runs to completion") ' this because test normally uses stdout</div><div> AssertStdErr.Equal(PrintMessage("Complete"),"<span style="background-color: rgb(248, 249, 250); color: rgb(34, 34, 34); font-family: inherit; font-size: 12pt; white-space: pre-wrap;" data-mce-style="background-color: #f8f9fa; color: #222222; font-family: inherit; font-size: 12pt; white-space: pre-wrap;">achevé","Test translation")</span><span style="background-color: rgb(248, 249, 250); color: rgb(34, 34, 34); font-family: inherit; font-size: 28px; white-space: pre-wrap;"><span style="font-size: 12pt;" data-mce-style="font-size: 12pt;"></span></span><span style="background-color: rgb(248, 249, 250); color: rgb(34, 34, 34); font-family: inherit; font-size: 28px; white-space: pre-wrap;"><span style="font-size: 12pt;" data-mce-style="font-size: 12pt;"></span></span></div><div><br data-mce-bogus="1"></div><div> try CriticalFile("junk") Assert.Error( ..... 'if the module quits when it gets an error and the quit is overridden can we still see the last error</div><div> if the module function does this for bad example:</div><div> sub CriticalFile(Filename as string) as file</div><div> mfile as file </div><div> try mfile = open FileName for read</div><div> if error then quit 6</div><div> return mfile</div><div> end</div><div> and do this following</div><div><br data-mce-bogus="1"></div><div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">If it is possible to to catch exits from app i.e. quit 6, is it possible to catch and verify the exit code returned</div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"><span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">CriticalFile("junk")</span></div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">Assert.ExitCode(6,"Exit value test for 6") </div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"><br></div></div><div>test.exitEnable()</div><div><br data-mce-bogus="1"></div><div>also perhaps as a dream</div><div> try GenProg("xxx") assert.NotError("Build Prog no error")</div><div> AssertFile.Equal("ModelFilePathName", "OutPutFilePathName","Function Output text Verify") 'binary compare</div><div><br data-mce-bogus="1"></div><div>Is it possible to have a way of pushing app command line args before calling a module, to test for correct parsing and processing of options. </div><div>Yes right now we can call functions but can not test the interface to the app.</div><div>e.g.</div><div><br data-mce-bogus="1"></div><div><span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">test.exitDisable() ' maybe app exiting should be disabled by test framework by default</span></div><div>test.args("-v", ....)</div><div>AssertStdOut.Equal(main() ,"Version output", "version output error from main") ' test captures the stream and compares</div><div><br data-mce-bogus="1"></div><div>test.args("-z",....)</div><div><span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">AssertStdErr.Equal(optparser(),"optparser output parm error","test bad parameter passed") ' test captures the stream and compares</span></span><div style="clear: both;" data-mce-style="clear: both;">etc.</div><div style="clear: both;" data-mce-style="clear: both;"><br></div></div><div>Maybe just dreaming</div><div><br data-mce-bogus="1"></div><div>Is it possible to make all variables/functions public and override the private setting during testing so that global tables that are constructed by private functions from input can be checked without making them</div><div>public in the actual code. This would help a great deal during the testing phase.</div><div><br data-mce-bogus="1"></div><div>Right now I needed add a flag into the actual code that stops app exits/quit with return. This itself adds a possible point of error.</div><div><br data-mce-bogus="1"></div><div>It often takes far more code to correctly test an application and all of it's possible errors and outputs than the app code actually is in size. And it is quite possible if left in place that some of the tests may create a security risk as they seem to be public.</div><div><br data-mce-bogus="1"></div><div>Are you thinking of this as more of a modelling tool than a test framework. Where you build an app framework with test then build your app to match that framework rather than actually testing the complete functionality and robustness of an application for production use.</div><div><br data-mce-bogus="1"></div><div>Should not a good test frame work allow simple direct testing providing interfaces to the most common functions of an average app.</div><div><br data-mce-bogus="1"></div><div>I<span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">t would be really nice if you could input data to a form field/button or input command through a keyboard or mouse simulation rather than just using test to call bits of an app.</span></div><div><span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><br data-mce-bogus="1"></span></div><div><div style="clear: both;" data-mce-style="clear: both;">Test.KeyStrokes(&x20,&x34,&x35,.....)</div><div style="clear: both;" data-mce-style="clear: both;"><div data-mce-style="clear: both;" style="clear: both;">assert.equal(SomeField.text," 45","test that input works correctly")</div></div><div style="clear: both;" data-mce-style="clear: both;">or</div><div style="clear: both;" data-mce-style="clear: both;">Test.KeyStrokes(ctrl-F, &x20)</div><div style="clear: both;" data-mce-style="clear: both;">assert.equal(SomeControl.HasFocus,true,"test that input works correctly")</div><div style="clear: both;" data-mce-style="clear: both;"><br></div><div style="clear: both;" data-mce-style="clear: both;">These are some of my thoughts.</div><div style="clear: both;" data-mce-style="clear: both;">I have to say that what you have built is a great thing, and has helped me find a number of possible bugs, but I have had to build much of what I am talking about to test more completely.</div><div style="clear: both;" data-mce-style="clear: both;"><br></div><div style="clear: both;" data-mce-style="clear: both;">I believe that the framework you have built is a very solid foundation. Thank you for the effort and thought you have put into it.</div><div style="clear: both;" data-mce-style="clear: both;"><br></div><div style="clear: both;" data-mce-style="clear: both;">And I understand that there are thing I still don't understand that are included into the test module.</div><div style="clear: both;" data-mce-style="clear: both;"><br></div></div><div data-marker="__SIG_PRE__">Thank You<br>Brian G</div><div><br></div><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><b>From: </b>"Brian" <brian@westwoodsvcs.com><br><b>To: </b>"Gambas mailing list" <user@lists.gambas-basic.org><br><b>Sent: </b>Saturday, June 27, 2020 6:35:43 AM<br><b>Subject: </b>Re: [Gambas-user] Using test interface for scripter testing<br></div><div><br></div><div data-marker="__QUOTED_TEXT__"><p style="margin-top: 0px;" dir="ltr">Does that not defeat the purpose of testing the the app for production use. Ie checking if correct errors are generated by error conditions etc. Have to alter code, then not test the actual method or module that outputs them.... Ummm</p>
<div id="mail-app-auto-default-signature">
<p dir="ltr">--<br> Thanks<br> Brian</p>
</div>Monday, 22 June 2020, 11:29PM -07:00 from Christof Thalhofer <a href="mailto:chrisml@deganius.de" target="_blank">chrisml@deganius.de</a>:<br><br><blockquote id="mail-app-auto-quote" cite="15928937590000016362" style="border-left:1px solid #0878BD; margin:0px 0px 0px 10px; padding:0px 0px 0px 10px;">
<div class="js-helper js-readmsg-msg">
<div>
<div id="style_15928937590000016362_BODY">Am 23.06.20 um 00:00 schrieb Brian G:<br>
<br>
<div class="mail-quote-collapse">> I am including a sample app with test module for things I will need<br>
> to do my testing. As you suggested I am capturing printed output,<br>
> which works, but also seems to print into the test result file.<br>
> <br>
> Is there some better way to use test to test functions which produce<br>
> printed messages.<br>
</div><br>
Use a function that creates the message as string and test the output of<br>
that function.<br>
<br>
And then use that function in a method that prints.<br>
<br>
Alles Gute<br>
<br>
Christof Thalhofer<br>
<br>
-- <br>
Dies ist keine Signatur<br>
<br>
</div>
<div><br>
----[ <a href="http://gambaswiki.org/wiki/doc/netiquette" target="_blank">http://gambaswiki.org/wiki/doc/netiquette</a> ]----<br>
<br>
</div>
</div>
</div>
</blockquote>
<br><br>----[ http://gambaswiki.org/wiki/doc/netiquette ]----<br></div></div></body></html>