Interesting study of vectorizing the tokenization of a complex language.

I’ve thought about vectorized sorting for a while, but I didn’t know that bitonic sorts can be implemented efficiently on SIMD registers. This is an interesting approach.