Don’t stop early: Case-folding source code at memory speed
How a branch-free loop and byte-space arithmetic let GitHub case-fold every byte of code search at >45 GiB/s on a single core.
Posts by this author
So many tokens, so little time: Introducing a faster, more flexible byte-pair tokenizer
We released a new open source byte-pair tokenizer that is faster and more flexible than popular alternatives.