Re: New routine in the development version : Tokenize()

[Benoît Minisini <benoit.minisini@xxxxxxxxxxxxxxxx>]

Le 04/10/2025 à 14:35, Lee a écrit :
> On 10/4/25 7:00 AM, Benoît Minisini wrote:
> > Le 04/10/2025 à 04:21, BB a écrit :
> > > > It's done: 'Tokenize()' has been renamed as 'Parse()'!
> > > I disagree with this a bit, "parse" has much stronger connotations 
> > > than just splitting a string into component parts. Consider "parse 
> > > this sentence into its subject and predicate".
> >
> > I just thought about the term "lexical parser", which is what mainly 
> > does the function. Moreover "Parse" was shorter than "Tokenize".
> >
> > > Secondly "tokenize" is unique in terms of code completion, tok<tab> 
> > > is not currently in use, par<tab> 
> > > needs manual choice between Param and Parse.
> >
> > Indeed, I didn't take that into account.
> >
> > I'd like to have the opinion from other people to take a decision.
> >
> > In Java, they use the term "Tokenizer". But it's Java, they like 
> > verbose names. 🙂
>
> I have to agree with Bruce that parsing is a more comprehensive process 
> than tokenizing. As I understand it, tokenizing is a sub-process of 
> parsing, and parsing goes further by trying to analyze the relationships 
> between the tokens.
>
> And, "Tokenize" is only three letters longer than "Parse". :-D

OK, back to "Tokenize" then.

--
Benoît Minisini.