Meaning tokenizing
What does tokenizing mean? Here you find 2 meanings of the word tokenizing. You can also add a definition of tokenizing yourself

1
0 Thumbs up   0 Thumbs down

tokenizing


Splitting up a program text into tokens. Also known as "lexing", in which case you get "lexemes" instead of tokens.
Source: archive.oreilly.com

2
0 Thumbs up   0 Thumbs down

tokenizing


Splitting up a program text into tokens
Source: perldoc.perl.org




<< tokener toolbox approach >>