Re: New routine in the development version : Tokenize()

[Benoît Minisini <benoit.minisini@xxxxxxxxxxxxxxxx>]

Le 04/10/2025 à 04:21, BB a écrit :
> > It's done: 'Tokenize()' has been renamed as 'Parse()'!
> I disagree with this a bit, "parse" has much stronger connotations than 
> just splitting a string into component parts. Consider "parse this 
> sentence into its subject and predicate".

I just thought about the term "lexical parser", which is what mainly 
does the function. Moreover "Parse" was shorter than "Tokenize".

> Secondly "tokenize" is unique 
> in terms of code completion, tok<tab> is not currently in use, par<tab> 
> needs manual choice between Param and Parse.

Indeed, I didn't take that into account.

I'd like to have the opinion from other people to take a decision.

In Java, they use the term "Tokenizer". But it's Java, they like verbose 
names. :-)

> (By the way, as I haven't tried it yet, whats the difference between 
> Tokenize and Split?)

'Split()' just cuts a string into pieces, using one caracter separators, 
with a support for escaping the pieces.

For Parse / Tokenize, it really parses the string and detect lexical 
tokens (idenitifiers, numbers, strings...). See the documentation.

> Finally, the wiki is not up to your normal standard, for example it was 
> not clear until several readings that it returns a String[].

There is the sentence "The tokens are returned as a string array." in 
the documentation. Wasn't that clear?

Maybe I should add a "Return" section in all documentation pages, so 
that the information is always found in one sight?

Regards,

--
Benoît Minisini.