📝 Easily create a beautiful website using Academic, Hugo, and Netlify

Code for the paper "BPE stays on SCRIPT", "Which Pieces Does Unigram Tokenization Really Need?" and MinGram

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Unsupervised text tokenizer for Neural Network-based text generation.

The best ChatGPT that $100 can buy.

UniScale: Scale-Invariant Tokenizer Research Project

Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization [arXiv 2025]

This is the repository for MorphScore, a tokenizer evaluation framework for morphological alignment.

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

Data and software for building the ACL Anthology.

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

PMI decoding using UCAM's sgnmt

An open-source NLP research library, built on PyTorch.

Topic Modelling for Humans