Implementation of Perceiver AR, Deepmind's new long-context attention network based on Perceiver architecture, in Pytorch

My own attempt at a long context genomics model, leveraging recent advances in long context attention modeling (Flash Attention + other hierarchical methods)

RAN: Recurrent Attention Networks for Long-text Modeling | Findings of ACL23

LongQLoRA: Extent Context Length of LLMs Efficiently

[DEPRECIATED] Very fast, large music transformer with 8k sequence length, efficient heptabit MIDI notes encoding, true full MIDI instruments range, chords counters and outro tokens

Streamlined variant of Long-Range Arena with pinned dependencies, automated data downloads, and deterministic shuffling.

Finetuning and evaluating LLMs to extract GHG emissions from PDF reports using RAG and grammar-based decoding.

Papers of Long Context Language Model

needle in a haystack for LLMs

The code of our paper "InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory"

PyTorch implementation of Infini-Transformer from "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" (https://arxiv.org/abs/2404.07143)

"Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding" Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, Zhangyang Wang.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

The official repo for "LLoCo: Learning Long Contexts Offline"

This is the official repo of "QuickLLaMA: Query-aware Inference Acceleration for Large Language Models"

open-source code for paper: Retrieval Head Mechanistically Explains Long-Context Factuality

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)

[COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding

Codes for the paper "∞Bench: Extending Long Context Evaluation Beyond 100K Tokens": https://arxiv.org/abs/2402.13718

ACL 2024 | LooGLE: Long Context Evaluation for Long-Context Language Models