Re: New routine in the development version : Tokenize()

[Lee <t.lee.davidson@xxxxxxxxx>]

On 10/4/25 7:00 AM, Benoît Minisini wrote:
> Le 04/10/2025 à 04:21, BB a écrit :
> > > It's done: 'Tokenize()' has been renamed as 'Parse()'!
> > I disagree with this a bit, "parse" has much stronger connotations than just splitting a string into component parts. Consider 
> > "parse this sentence into its subject and predicate".
>
> I just thought about the term "lexical parser", which is what mainly 
> does the function. Moreover "Parse" was shorter than "Tokenize".
>
> > Secondly "tokenize" is unique in terms of code completion, tok<tab> is not currently in use, par<tab> 
> > needs manual choice between Param and Parse.
>
> Indeed, I didn't take that into account.
>
> I'd like to have the opinion from other people to take a decision.
>
> In Java, they use the term "Tokenizer". But it's Java, they like verbose names. 🙂

I have to agree with Bruce that parsing is a more comprehensive process than tokenizing. As I understand it, tokenizing is a 
sub-process of parsing, and parsing goes further by trying to analyze the relationships between the tokens.

And, "Tokenize" is only three letters longer than "Parse". :-D


--
Lee

--- Gambas User List Netiquette [https://gambaswiki.org/wiki/doc/netiquette] ----
--- Gambas User List Archive [https://lists.gambas-basic.org/archive/user] ----