tokenizer中文是什么意思

千万人都在用的超大词汇词典翻译APP

With the tokenizer, it is easy to decode the processing instructions
有了tokenizer，就很容易译码处理指令。
A good lexer example can help a lot with learning how to write a tokenizer
一个好的lexer例子会非常有助于学习如何编写断词器（tokenizer）。
We use flex to get a tokenizer . and then, the realization of error tolerance function and later modification problem is simplified
词法分析采用flex工具实现，简单化词法分析的容错功能的实现以及之后的程序修改问题。
Quantifiers can be used within the regular expressions of the spark tokenizer, and can be simulated by recursion in parse expression grammars
计量符可以在spark记号赋予器（tokenizer）的正则表达式中使用，并可以用解析表达式语法中的递归来进行模拟。
Not i-wish-it-were-a-second-faster slow, but take-a-long-lunch-and-hope-it-finishes slow . in my experiments, the tokenizer is plenty fast, but the parsing bogs down even with quite small test cases
在我的实验中，记号赋予器还比较快，但解析过程就很慢了，即便用很小的测试案例也很慢。