New routine in the development version : Tokenize()

[Benoît Minisini <benoit.minisini@xxxxxxxxxxxxxxxx>]

Hi,

I have just added a general-purpose 'Tokenize()' routine to Gambas, that 
splits a string into tokens (identifiers, numbers, strings...), 
according to a few arguments.

The documentation is at: https://gambaswiki.org/wiki/lang/tokenize

You must login to get the last version that fixes many typos.

The aim of that routine is being useful without taking too many 
arguments - as a real dynamic parser is like 'gb.highlight', it needs a 
compiler!

Please tell me what you think about 'Tokenize()', if you find it useful, 
if you think some options can be added or removed, and so on.

Regards,

--
Benoît Minisini.