Skip to content

One Buffer per Attention unit - #40

Open
almogtavor wants to merge 1 commit into
mainfrom
buffer-per-attention-unit
Open

almogtavor wants to merge 1 commit into
mainfrom
buffer-per-attention-unit

Conversation

@almogtavor

Copy link
Copy Markdown
Member

Step 1 of #38.

Goal: each Attention unit writes to its own Buffer, so a unit can be added or removed without rebuilding the Buffers of the others.

Units(group, config, units) builds one Buffer per Attention unit over the same FFN ranks. Nothing else changes: the FFN ranks still wait for every unit.

Results (8x H100, one NVLink domain, exact match with the single-Buffer reference, eager and CUDA graph):

FFN ranks Units Attention ranks per unit Payload Hybrid Result
2 2 3 BF16 on pass
2 3 2 FP8 off pass
2 6 1 BF16 on pass
1 7 1 FP8 on pass
2 1 6 BF16 on pass
2 2 1 BF16 off pass

In every run the last unit closes its Buffer and the other units keep exchanging on theirs.
The existing suite (tests.gpu.deepep --lanes 2, BF16 and FP8) also passes, and so do the 59 unit tests and Ruff.

Not included: DBO inside a unit, and adding a unit while the others run.

Each Attention unit gets its own Gloo subgroup and Buffer, so an exchange or a close on one unit involves only the FFN ranks and that unit. The FFN ranks still wait for every unit they receive from. Lanes and Units share the helpers that give each Buffer its own communication stream.
@almogtavor
almogtavor requested a review from Etelis as a code owner November 17, 2026 12:55
@github-actions

Copy link
Copy Markdown
PR Preview Action v1.8.2

QR code for preview link

🚀 View preview at
https://afdEP.github.io/afdEP-comm/pr-preview/pr-40/

Built to branch gh-pages at 2026-11-17 12:56 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant