Draft
Add DSA (DeepSeek Sparse Attention) principle documentation#1
Conversation
Copilot created this pull request from a session on behalf of
hongbolv
July 21, 2026 02:47
View session
hongbolv
reviewed
Jul 21, 2026
| - **只产出分数,不产出输出**:Indexer 不参与 value 的加权求和,它唯一的职责是给出一个分数向量,供后续的 top-k 选择使用。 | ||
|
|
||
| 正因为它“轻如闪电”,即便需要对历史所有 token 打分,其代价也远低于稠密注意力本身。 | ||
|
|
Author
There was a problem hiding this comment.
已扩展第 4 节,对 Lightning Indexer 的原理做了详细补充,新增以下子小节:
- 4.1 它在做什么:为每个 query 对所有历史 token 打索引分数,只需保持正确的相对排序。
- 4.2 打分公式(概念形式):给出
I(t, s) = Σ_j w_{t,j}·ReLU(q^index_{t,j}·k^index_s)的概念公式,并解释各项(小维度索引投影、少量索引头、可学习门控权重、ReLU 非线性)——本质是“迷你版、多头、带门控、只算分数不算输出”的注意力打分器。 - 4.3 为什么能“闪电般”便宜:极小索引维度、索引头少、FP8 低精度、不做 value 聚合、可跨层复用(IndexShare)。
- 4.4 它是怎么学会打分的:稠密预热阶段用真实注意力分布做蒸馏/对齐(如 KL 目标)。
- 4.5 在推理中的位置:解码每步先打分→取 top-k→仅在 top-k 上做高精度注意力。
已提交并推送。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GLM-5 integrates DeepSeek Sparse Attention (DSA) to cut long-context deployment cost, but the repo had no explanation of how it works. This adds a self-contained reference document.
Changes
docs/DSA原理详解.md— Chinese-language explanation of DSA covering:O(L²)bottleneck of dense attention at 1M-token contextO(L²)→ ≈O(L·k))glm_moe_dsa)Notes