Skip to content

Add DSA (DeepSeek Sparse Attention) principle documentation - #1

Draft
hongbolv with Copilot wants to merge 2 commits into
mainfrom
copilot/glms-dsa-principle-introduction
Draft

Add DSA (DeepSeek Sparse Attention) principle documentation#1
hongbolv with Copilot wants to merge 2 commits into
mainfrom
copilot/glms-dsa-principle-introduction

Conversation

Copilot AI commented Jul 21, 2026

Copy link
Copy Markdown

GLM-5 integrates DeepSeek Sparse Attention (DSA) to cut long-context deployment cost, but the repo had no explanation of how it works. This adds a self-contained reference document.

Changes

  • New doc docs/DSA原理详解.md — Chinese-language explanation of DSA covering:
    • Motivation and the O(L²) bottleneck of dense attention at 1M-token context
    • Two-stage design: Lightning Indexer (cheap, FP8-friendly relevance scoring) → top-k token selection → precise attention only on selected tokens
    • Interplay with MLA (KV-cache compression vs. per-step compute reduction)
    • Complexity/memory comparison table (O(L²) → ≈ O(L·k))
    • Dense→sparse continued-training recipe
    • GLM-5 usage and GLM-5.2 IndexShare improvement (shared indexer per 4 layers, ~2.9× per-token FLOPs reduction at 1M)
    • Trade-offs, use cases, and references (technical report, IndexShare, Transformers glm_moe_dsa)

Notes

  • Documentation-only change; no code paths affected.
  • Some quantitative/implementation specifics are attributed to official reports, with a closing note directing readers to authoritative sources.

Comment thread docs/DSA原理详解.md
- **只产出分数,不产出输出**:Indexer 不参与 value 的加权求和,它唯一的职责是给出一个分数向量,供后续的 top-k 选择使用。

正因为它“轻如闪电”,即便需要对历史所有 token 打分,其代价也远低于稠密注意力本身。

@hongbolv hongbolv Jul 21, 2026

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot 详细介绍一下Lightning Indexer的原理。

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已扩展第 4 节,对 Lightning Indexer 的原理做了详细补充,新增以下子小节:

  • 4.1 它在做什么:为每个 query 对所有历史 token 打索引分数,只需保持正确的相对排序。
  • 4.2 打分公式(概念形式):给出 I(t, s) = Σ_j w_{t,j}·ReLU(q^index_{t,j}·k^index_s) 的概念公式,并解释各项(小维度索引投影、少量索引头、可学习门控权重、ReLU 非线性)——本质是“迷你版、多头、带门控、只算分数不算输出”的注意力打分器。
  • 4.3 为什么能“闪电般”便宜:极小索引维度、索引头少、FP8 低精度、不做 value 聚合、可跨层复用(IndexShare)。
  • 4.4 它是怎么学会打分的:稠密预热阶段用真实注意力分布做蒸馏/对齐(如 KL 目标)。
  • 4.5 在推理中的位置:解码每步先打分→取 top-k→仅在 top-k 上做高精度注意力。

已提交并推送。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants