[ICML 2025] |TokenSwift: Lossless Acceleration of Ultra Long Sequence Generation

📰 Must-read papers and blogs on LLM based Long Context Modeling 🔥

LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation

Multilingual Corpus of Web Fiction

Efficient LLM Inference over Long Sequences

The HELMET Benchmark

LOFT: A 1 Million+ Token Long-Context Benchmark

Awesome List of Attention Modules and Plug&Play Modules in Computer Vision

[ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference

This repo contains the source code for RULER: What’s the Real Context Size of Your Long-Context Language Models?

Unofficial PyTorch/🤗Transformers(Gemma/Llama3) implementation of Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

Acceptance rates for the major AI conferences

Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.

Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch

Extend existing LLMs way beyond the original training length with constant memory usage, without retraining

Ring attention implementation with flash attention

Large Context Attention

Foundational Models for State-of-the-Art Speech and Text Translation

Code for the paper "Evaluating Large Language Models Trained on Code"

From Chain-of-Thought prompting to OpenAI o1 and DeepSeek-R1 🍓

Beyond the Imitation Game collaborative benchmark for measuring and extrapolating the capabilities of language models

Self-Supervised Speech Pre-training and Representation Learning Toolkit

This is the official implementation of "Progressive-Hint Prompting Improves Reasoning in Large Language Models"

Adaptive Experimentation Platform

The agent engineering platform.

Reproducible code for paper "qEUBO A Decision-Theoretic Acquisition Function for Preferential Bayesian Optimization" from AISTATS 2023