Large Language Model Based Long Context Modeling Papers and Blogs

This repository curates papers and blogs on long-context language modeling, covering surveys; efficient attention; KV-cache optimization; recurrent transformers and state-space models; position encoding & length extrapolation; long-context training; long-term memory; retrieval-augmented generation; in-context learning; context and model compression; long reasoning (long CoT); long video & image; long-horizon agents; long-text generation; inference acceleration; benchmarks & evaluation; and technical reports.

🔥 Must-read papers for LLM-based Long Context Modeling.

🔥⚡🔥 Thanks for all the great contributors on GitHub!

🚀🤝🚀 I have the privilege of joining [LCLM-Horizon] and collaborating with them on providing a very complete and comprehensive scholarly survey (A Comprehensive Survey on Long Context Language Modeling) and repository (A-Comprehensive-Survey-For-Long-Context-Language-Modeling) dedicated to Long Context Language Modeling. I look forward to collaborating with them to advance research and deepen understanding in this area!

flowchart LR
  LCLM["Long-Context Modeling"]
  LCLM --> A["Attention & KV Cache"]
  LCLM --> T["Training & Alignment"]
  LCLM --> M["Memory & RAG"]
  LCLM --> C["Compression"]
  LCLM --> R["Reasoning & Generation"]
  LCLM --> V["Multimodal / Video"]
  LCLM --> E["Evaluation & Acceleration"]

  A --> A1["Sparse / Linear / IO-aware Attention"]
  A --> A2["Eviction / Quantization / Offloading"]
  T --> T1["Continual Pretraining / Long-SFT"]
  T --> T2["Adaptation & RL for Long Context"]
  M --> M1["Long-Term Memory"]
  M --> M2["RAG / Hybrid Long-Context"]
  C --> C1["Context Compression"]
  C --> C2["Model Compression"]
  R --> R1["Long CoT"]
  R --> R2["Long-Form Text Generation"]

If you find our repository and survey useful for your research, please consider citing the following paper:

@article{liu2025comprehensive,
  title={A Comprehensive Survey on Long Context Language Modeling},
  author={Liu, Jiaheng and Zhu, Dawei and Bai, Zhiqi and He, Yancheng and Liao, Huanxuan and Que, Haoran and Wang, Zekun and Zhang, Chenchen and Zhang, Ge and Zhang, Jiebin and others},
  journal={arXiv preprint arXiv:2503.17407},
  year={2025}
}

• 📢 News
  • Week Papers
  • Month Papers
• 📜 Papers
  • 1. Survey Papers
  • 2. Efficient Attention
  • 3. KV-Cache Optimization
  • 4. Recurrent Transformers
  • 5. State Space Models & Hybrids
  • 6. Position Encoding & Length Extrapolation
  • 7. Long-Context Training
  • 8. Long-Term Memory
  • 9. Retrieval-Augmented Generation
  • 10. In-Context Learning (Many-shot / Long-ICL)
  • 11. Context Compression
  • 12. Model Compression for Long Context
  • 13. Long Reasoning (Long CoT)
  • 14. Long Video & Image
  • 15. Long-Horizon Agents
  • 16. Long-form Text Generation
  • 17. Inference Acceleration & Serving
  • 18. Benchmarks & Evaluation
  • 19. Technical Reports (Long-Context Models)
  • 20. Blogs & Tutorials
• Acknowledgements
  • Contributors
  • Star History

• [2026.10.21]

  • Paper: Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law
  • Paper: Uni-Mem: Learning a Long-Horizon Agentic Memory Manager with Unified Online Policy Optimization
  • Paper: LongZero: Enhancing Long-Context Reasoning in LLMs from Zero Data
  • Paper: Cache Locally, Recompute Selectively for Streaming Video Understanding

• [2026.10.20]

  • Paper: Scan or Index? Large Evidence Coverage, Answer Quality, and Cost in Long-Document RAG
  • Paper: Cache Engineering for Agents

• [2026.10.19]

  • Paper: KEEP-Bench: Evaluating Evidence Preservation in Context Compression
  • Paper: LEAPS: Long-context Evaluation of Adaptive Pattern Learning and State Tracking
  • Paper: TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning
  • Paper: Kronecker Delta Rule for Long Context Linear Attention

• [2026.10.18]

  • Paper: DiaLong: Benchmarking Reminiscence in Long-Context Dialogues
  • Paper: The Bottleneck Is Retrieval, Not Reading: Diagnosing Evidence-Based Generation with RAG for Long Context Question Answering

• [2026.10.17]

  • Paper: TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference

• [2026.10.16]

  • Paper: Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design
  • Paper: DLite: History-free Speculative Decoding for Agentic AI

• [2026.10.15]

  • Paper: GACC-Bench: Benchmarking Goal-Adaptive Context Compression Ability across Large Language Models
  • Paper: Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning
  • Paper: DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction
  • Paper: Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries

• [2026.10.14]

  • Paper: DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding
  • Paper: Beyond State Capacity: Why Retrieval Breaks in Mamba
  • Paper: Cortex: Consolidating Context into Parametric Memory
  • Paper: Prune the Summary, Not the History: Source-Judged Compaction for Long-Horizon Agents
  • Paper: BizAgentBench: Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business

• [2026.10.13]

  • Paper: VideoJev: System One Model for Video Understanding and Interaction
  • Paper: DoubtLess: Training-Free Contrastive Decoding for Efficient Reasoning
  • Paper: Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agents
  • Paper: FrameScout: Scouting Query-Relevant Frames for Long Video Understanding
  • Paper: Joint Subspace Sparse Attention for Efficient Long-Context Decoding

• [2026.10.12]

  • Paper: FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation
  • Paper: Elephant in the Fridge: Constant-Memory Frame Packing for Long Video Understanding

• [2026.10.11]

  • Paper: History Replays Itself: Unleashing Long-Context Capability in Linear Attention
  • Paper: Apertus 1.5: From Open Text Models to Open Multimodal Reasoners

• [2026.10.10]

  • Paper: SPARE: Shared Perception and Adaptive Retrieval of Evidence for Streaming Video Understanding
  • Paper: SAFE-KV: Causal Suffix Repair for Behavior-Preserving Non-Prefix KV Cache Reuse in RAG
  • Paper: KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization

• [2026.10.09]

  • Paper: Constructing Pareto Frontiers of Optimal Reasoning Termination Methods for Overthinking in Reasoning Models
  • Paper: REMORY: Learning Residual Memory for Context Compaction

• [2026.10.08]

  • Paper: FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams?
  • Paper: From Retrieval to Reconstruction: Constructing Evolvable Cognitive Memory for Long-Term Dialogue
  • Paper: Thinking Inertia: LLMs Keep Thinking When Told Not To
  • Paper: V-CoLA: Vision Token Compression with Linear Attention

Paper entries live under papers/ so this README stays under GitHub's homepage size limit. For an interactive chapter reader (search + in-page paper cards), open the project homepage.

Please contact me if I miss your names in the list, I will add you back ASAP!