tokenise造句

In POSIX mode, Bison is compatible with yacc, but also has several extensions over this earlier program . lexical analyser, is often used with Bison, to tokenise input data and provide Bison with tokens.
It also meant that the BASIC interpreter ( and later the compiler ) didn't need to tokenise a program when it was loaded, which would have been a significant load-time overhead for some of the larger GFA BASIC programs that were written.
It's difficult to find tokenise in a sentence. 用tokenise造句挺难的