#operator #tokenize #vector #transform #string

code-tokenizer

A string/code tokenizer. Transforms a string with a vector of operators into a vector of keywords.

1 unstable release

0.1.0 Feb 6, 2020

#119 in #transform

MIT license

5KB
75 lines

Code Tokenizer

A string/code tokenizer. Transforms a string with a vector of operators into a vector of keywords.

No runtime deps