tokenizer中文什么意思

例句与用法

1. with the tokenizer, it is easy to decode the processing instructions
   有了tokenizer，就很容易译码处理指令。
2. a good lexer example can help a lot with learning how to write a tokenizer
   一个好的lexer例子会非常有助于学习如何编写断词器（tokenizer）。
3. we use flex to get a tokenizer . and then, the realization of error tolerance function and later modification problem is simplified
   词法分析采用flex工具实现，简单化词法分析的容错功能的实现以及之后的程序修改问题。
4. quantifiers can be used within the regular expressions of the spark tokenizer, and can be simulated by recursion in parse expression grammars
   计量符可以在spark记号赋予器（tokenizer）的正则表达式中使用，并可以用解析表达式语法中的递归来进行模拟。
5. not i-wish-it-were-a-second-faster slow, but take-a-long-lunch-and-hope-it-finishes slow . in my experiments, the tokenizer is plenty fast, but the parsing bogs down even with quite small test cases
   在我的实验中，记号赋予器还比较快，但解析过程就很慢了，即便用很小的测试案例也很慢。