Ongoing research training transformer language models at scale, including: BERT & GPT-2

Memray is a memory profiler for Python

The nnsight package enables interpreting and manipulating the internals of deep learned models.

Tooling for exact and MinHash deduplication of large-scale text datasets

[ICML'24] Data and code for our paper "Training-Free Long-Context Scaling of Large Language Models"

Flexible and powerful tensor operations for readable and reliable code (for pytorch, jax, TF and others)

uops.info Code Analyzer

High-performance MinHash implementation in Rust with Python bindings for efficient similarity estimation and deduplication of large datasets

LSH index for approximate set containment search

All-pair set similarity search on millions of sets in Python and on a laptop

Python extension for MurmurHash (MurmurHash3), a set of fast and robust hash functions.

MinHash, LSH, LSH Forest, Weighted MinHash, HyperLogLog, HyperLogLog++, LSH Ensemble and HNSW

Fast Python Bloom Filter using Mmap

A specification that python filesystems should adhere to.

Text utilities, including beam search decoding, tokenizing, and more, built for use in Flashlight.

A simple n-gram language model.

KenLM: Faster and Smaller Language Model Queries

The RedPajama-Data repository contains code for preparing large datasets for training large language models.

Computing with Python functions.

Ship correct and fast LLM kernels to PyTorch

Python logging made (stupidly) simple

Parallel S3 and local filesystem execution tool.

MobileLLM Optimizing Sub-billion Parameter Language Models for On-Device Use Cases. In ICML 2024.

Streaming WARC/ARC library for fast web archive IO

[ICLR 2025] Official PyTorch Implementation of Gated Delta Networks: Improving Mamba2 with Delta Rule

cuDF - GPU DataFrame Library

Github Pages template based upon HTML and Markdown for personal, portfolio-based websites.