Skip to content

Take First Ready policy for the FFN ranks - #33

Draft
almogtavor wants to merge 11 commits into
buffer-per-attention-unitfrom
ffn-service
Draft

almogtavor wants to merge 11 commits into
buffer-per-attention-unitfrom
ffn-service

Conversation

@almogtavor

@almogtavor almogtavor commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

This PR adds the Take First Ready policy for the FFN ranks, on top of #40 (one Buffer per Attention unit). The existing behavior is named Take Everyone.

  • Take Everyone: the FFN ranks serve the units in a fixed order and wait for each one.
  • Take First Ready: the FFN ranks serve whichever unit is ready first; units that are ready together rotate (round-robin).

What FfnService does:

  1. Buffers of different sizes are registered once at start (tested with capacities 32 to 1024 together).
  2. The FFN runs constantly: one CUDA graph runs 32 rounds per launch, and the host only relaunches it.
  3. Each DP group is its own unit with its own Buffer and is scheduled by itself. A DBO micro-batch is a regular unit in the same rotation.
  4. The pick is made on the GPU inside the graph, or on the host (--mode device / --mode host in the test).
  5. Optional leader order (order=order_ring(buffer), test flag --leader): FFN rank 0 writes each pick into free arrival words of the first unit's Buffer, and the other FFN ranks read its copy through the NCCL window (ncclGetLsaPointer in the readiness kernel) and serve the same unit. Off by default.

Results on #40

Sum over units of rounds per second, median of 5 repeats.

Expert alignment 1:

rows per rank, split one Buffer, Take Everyone per-unit Buffers, Take Everyone per-unit Buffers, Take First Ready (GPU picks) per-unit Buffers, Take First Ready (host picks)
128, random 211 213 301 293
128, length 256 244 573 450
1024, random 259 241 327 258
1024, length 316 271 353 256

Expert alignment 128 (the default):

rows per rank, split one Buffer, Take Everyone per-unit Buffers, Take Everyone per-unit Buffers, Take First Ready (GPU picks) per-unit Buffers, Take First Ready (host picks)
128, random 212 212 251 202
128, length 253 224 282 214
1024, random 256 205 206 155
1024, length 300 207 207 154
  • With alignment 1, Take First Ready with GPU picks is the fastest in all four rows (1.12x to 2.24x of one Buffer with Take Everyone).
  • With alignment 128 it is faster at 128 rows (1.12x to 1.18x) and slower at 1024 rows (0.69x to 0.80x): there one Buffer with Take Everyone is the fastest, and per-unit Buffers cost throughput under either policy.
  • Host picks are slower than GPU picks everywhere, and slower than Take Everyone with alignment 128.
  • Take First Ready is less even between units when their load differs: Jain index of the per-unit rates 0.92 to 1.00 with alignment 1 and 0.98 to 1.00 with alignment 128 (1.00 for Take Everyone).
  • Without the leader order the two FFN ranks pick a different unit in up to 25% (alignment 1) and 40% (alignment 128) of the GPU picks.

Method

  • 8x H100 in one node: 6 Attention ranks, one unit each, and 2 FFN ranks.
  • Real replay: layer 31 experts of Qwen/Qwen3.5-397B-A17B-FP8 (512 routed experts, top-k 10, hidden size 4096) on ShareGPT V3 tokens (600 conversations). Random split: conversations are assigned to units at random. Length split: by conversation length, so units differ in context length.
  • The same sweep ran once per alignment: 2 capacities x 2 splits x 6 configurations x 5 repeats, 300 rounds per unit, the order of configurations rotated per repeat, leader order off, unprofiled, outputs not checked during the timed runs. All 240 timed runs completed; the five repeats stay within 2.5% of their median.
  • Before each sweep, 12 short runs (every configuration at both capacities) compared the outputs with the checkpoint: relative error at most 0.0116.
  • The fourth and fifth configurations build the per-unit Buffers by hand instead of through Units; they match the Units columns within 1% and are left out of the tables.
  • Native source digest: b1324d3262a904e7d7c6d77a0b324bb269eeca4872a0133fc3db180f152c179e.

These are communication plus expert compute numbers, not serving numbers.

Earlier results, before the move onto #40 (commit ab769d3, expert alignment 1)

Exchanges/s while all units are active, 300 rounds per unit.

units x tokens rounds rounds + leader host picks fixed order
6 x 128 492 494 395 244
6 x 512 394 394 290 276
6 x 2048 262 262 202 247
6, mixed 32 to 1024 412, 449 452 368 283
6 x 256, DBO 2 micro-batches 556 556 422 336
2 x 128 (mean context 325 and 876 tokens) 123 123 122 85
2, 64 and 1024 tokens (mean context 243 and 1516) 113 113 113 82

With the leader order no pick differs between the FFN ranks, and the test asserts that. Output check on the same replay: relative error 0.009 to 0.010 against the checkpoint with 2 and 4 FFN ranks, with and without the leader order. The real replay with 3 FFN ranks is not validated (the synthetic 3-rank check passes).

Recorded experiment: persistent kernel (code not in this PR)

One kernel launch for the whole run that picks, receives and replies itself reached 0.73-0.79x of rounds (395 against 539 at 6 x 128), so I removed it. The last commit that contains it is f8b992d.

@almogtavor
almogtavor requested a review from Etelis as a code owner November 15, 2026 14:37
@almogtavor almogtavor changed the title Add FFN service with ready-order scheduling FFN as a service Nov 15, 2026
@almogtavor almogtavor changed the title FFN as a service FFN as a Service (FaaS) Nov 15, 2026
One resident launch per run picks clients like the rounds method and does
the receive and the reply itself. The experts run as CUDA graphs on a second
stream (persistent kernel) or inside the same launch (megakernel, FP32
without tensor cores).
The megakernel reads each 16x8 tile as 256 contiguous bytes, claims column
blocks per warp without staging rows in shared memory, spreads the reply copy
over all warps, and is compiled with 64 registers so four CTAs fit per SM.
build_resident rejects a grid that cannot be resident at once.
On the Qwen3.5-397B-A17B-FP8 layer 31 replay (ShareGPT V3) it reaches 558
exchanges/s at expert alignment 1 and 559 at 128; rounds reach 538 and 267.
Comment thread afdep/service.cu
const int* state, const int* quota,
const int rotate) {
bool pending = false;
for (int i = 1; i <= num_clients; ++ i) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would one lane per client with a warp ballot make sense here, still using one CTA? I’m curious whether it would cut ready-to-receive latency with staggered arrivals or wider DP groups.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have one thread scanning all the rest idle iiuc.
Why not scan in parallel?

Comment thread tests/gpu/ffn_service.py
"all_active_per_s": active / min(r["wall_s"] for r in clients),
"all_active_served": served,
"client_rates": rates,
"order_divergence": diverged / total,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think the FFNs need a shared client order here? I’m curious what happens if one serves A then B and the other serves B then A, in both graph and resident modes.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two FFN workers should they do same requests or not?

Comment thread afdep/service.cu Outdated
}
if (received == replied and choice != -4)
end = choice;
else if (timer() - start > timeout_ns)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if a client arrives just as one FFN times out and another picks it up? Would the client still get all its replies?

Comment thread benchmarks/pull_push.py Outdated
if mode == "push" and rank == PRODUCER:
src = [buffer.data_ptr()] * (world - 1)
dst = [peers[r].data_ptr() for r in range(world) if r != PRODUCER]
elif mode == "pull" and rank != PRODUCER:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about Attention pulling and reducing the same reply rows? Would you expect the FFN to move on sooner, and how would a slow reader affect Attention tail latency and Buffer reuse?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's experimental.

@Etelis

Etelis commented Nov 16, 2026

Copy link
Copy Markdown
Member

Remove the megakernel and fusion pls.

@Etelis

Etelis commented Nov 16, 2026

Copy link
Copy Markdown
Member

DBO, how will we schedule different lanes?

The resident kernel that also computes the experts covers only the FFN side,
so the mode is now --mode fused.
The resident kernel no longer computes the experts itself. It lost to
rounds for every batch above 128 tokens per client (0.82x at 512 tokens,
0.38x at 2048), so the persistent kernel keeps only the pick, the receive
and the reply, and the expert graphs run on the second stream.
@almogtavor
almogtavor marked this pull request as draft November 16, 2026 16:00
Rounds (one CUDA graph, 32 rounds per launch) was faster than the resident
kernel at both expert alignments, so the service keeps the scan kernel only.
get_afd_service_words had no other caller.
@almogtavor

Copy link
Copy Markdown
Member Author

Hi, this is the measurement for the review question about a shared client order between the FFN ranks.

Today: every FFN rank picks the next client by itself from its own arrival counters, so two FFN ranks can serve the clients in different orders.

Experiment: FFN rank 0 (the leader) picks as before and publishes each pick; the other FFN ranks do not check readiness and serve the published order. The kernel change is small:

// leader, when it records pick i:
order[i] = choice + 1;
// follower, for its pick i (0 means not published yet):
const int published = order[state[2]];
return published ? published - 1 : -4;

Setup: one pod, 8x H100, 2 FFN ranks and 6 clients, Qwen/Qwen3.5-397B-A17B-FP8 layer 31 experts replayed on ShareGPT V3 tokens, expert alignment 1, rounds mode (32 rounds per launch). Both modes run from the same binary, 6 runs each, alternating which mode runs first. Values are mean +- standard deviation over the 6 runs. Latency is the client side time of one exchange, averaged over the clients.

layout exchanges/s independent exchanges/s leader p50 ms independent p50 ms leader p99 ms independent p99 ms leader picks that differ between FFN ranks (independent)
6 x 128 tokens 537.7 +- 1.0 538.1 +- 0.5 7.54 +- 0.03 7.56 +- 0.01 10.84 +- 0.03 10.84 +- 0.02 13.5%
6 x 512 tokens 393.6 +- 0.2 393.8 +- 0.1 10.81 +- 0.04 10.61 +- 0.10 15.61 +- 0.26 15.03 +- 0.30 22.2%
32, 64, 128, 256, 512, 1024 tokens 450.8 +- 3.6 451.9 +- 1.4 6.45 +- 0.10 6.43 +- 0.14 12.55 +- 0.10 11.93 +- 0.09 25.8%
6 x 256 tokens, 2 DBO micro-batches each 555.3 +- 2.0 555.8 +- 0.8 12.00 +- 0.12 11.63 +- 0.18 20.44 +- 0.09 20.43 +- 0.08 42.9%
6 x 128, short random client delay on 3 clients 537.8 +- 0.2 538.1 +- 0.4 7.50 +- 0.03 7.49 +- 0.03 10.84 +- 0.04 10.82 +- 0.01 13.6%
6 x 128, long random delay on all clients (FFN ranks wait for work) 372.6 +- 0.8 373.6 +- 0.7 3.00 +- 0.05 2.99 +- 0.02 7.77 +- 0.22 7.28 +- 0.21 1.6%
6 x 128, long random delay on 3 clients 434.5 +- 1.0 433.7 +- 2.3 3.04 +- 0.01 3.04 +- 0.01 8.51 +- 0.34 8.12 +- 0.36 2.7%

With the leader the two FFN ranks served the same order in every run (0% differing picks).

What the numbers say

  • Throughput does not change: every difference is within one standard deviation.
  • p99 latency is 4-6% lower with the leader in three layouts (512 tokens, mixed sizes, long delay on all clients), each by more than three standard errors. The 3 delayed clients row shows 5% with about two standard errors, so I do not count it. The other three rows show no difference.
  • p50 latency is 2-3% lower in two layouts (512 tokens, DBO) and unchanged elsewhere.
  • Correctness: with the leader, the synthetic suite passed with 2 and 3 FFN ranks in host and rounds modes (empty and skewed routes, DBO, delayed clients), and the real replay stayed within 0.009-0.010 relative error of the checkpoint in all five layouts I checked.

Limits

  • The order words in this experiment are host memory shared between the FFN processes, so it works on one node only. A version for the library needs these words in memory that the transport already shares between ranks (the NCCL window).
  • Everything ran on one pod and one node, with 2 FFN ranks in the timed runs.
  • The 512, mixed, DBO and long delay rows use 300 rounds per client, so a per-client p99 is the third largest sample of a run. The 128 token rows use 1000 rounds.
  • These are communication plus expert compute numbers, not serving numbers.

My reading: the followers do not need their own readiness check, and one order gives the same throughput with a somewhat lower tail latency in part of the layouts. I left the experiment out of this PR (it lives on a local branch on top of f0e44b0) because the one-node channel is a shortcut. I can do the NCCL window version as a follow-up PR if you want it in.

Native source digest of the timed binary: 8befb7fd041c6fd2a61c938f480fb7ba6b07223e8e33c1997ce01049769434c0.

@almogtavor
almogtavor requested a review from ronenkat November 16, 2026 21:43
FFN rank 0 writes each pick into a ring of free arrival words of the
first client Buffer. The other FFN ranks read its copy of those words
through the NCCL window (ncclGetLsaPointer in the readiness kernel) and
serve the same client, so every FFN rank serves in one order. The option
is off by default; tests/gpu/ffn_service.py --leader turns it on and
asserts that no pick differs between FFN ranks.
@almogtavor

Copy link
Copy Markdown
Member Author

Parallel readiness check: no gain.

readiness check

@Etelis

Etelis commented Nov 17, 2026

Copy link
Copy Markdown
Member

Hi, this is the measurement for the review question about a shared client order between the FFN ranks.

Today: every FFN rank picks the next client by itself from its own arrival counters, so two FFN ranks can serve the clients in different orders.

Experiment: FFN rank 0 (the leader) picks as before and publishes each pick; the other FFN ranks do not check readiness and serve the published order. The kernel change is small:

// leader, when it records pick i:
order[i] = choice + 1;
// follower, for its pick i (0 means not published yet):
const int published = order[state[2]];
return published ? published - 1 : -4;

Setup: one pod, 8x H100, 2 FFN ranks and 6 clients, Qwen/Qwen3.5-397B-A17B-FP8 layer 31 experts replayed on ShareGPT V3 tokens, expert alignment 1, rounds mode (32 rounds per launch). Both modes run from the same binary, 6 runs each, alternating which mode runs first. Values are mean +- standard deviation over the 6 runs. Latency is the client side time of one exchange, averaged over the clients.

layout exchanges/s independent exchanges/s leader p50 ms independent p50 ms leader p99 ms independent p99 ms leader picks that differ between FFN ranks (independent)
6 x 128 tokens 537.7 +- 1.0 538.1 +- 0.5 7.54 +- 0.03 7.56 +- 0.01 10.84 +- 0.03 10.84 +- 0.02 13.5%
6 x 512 tokens 393.6 +- 0.2 393.8 +- 0.1 10.81 +- 0.04 10.61 +- 0.10 15.61 +- 0.26 15.03 +- 0.30 22.2%
32, 64, 128, 256, 512, 1024 tokens 450.8 +- 3.6 451.9 +- 1.4 6.45 +- 0.10 6.43 +- 0.14 12.55 +- 0.10 11.93 +- 0.09 25.8%
6 x 256 tokens, 2 DBO micro-batches each 555.3 +- 2.0 555.8 +- 0.8 12.00 +- 0.12 11.63 +- 0.18 20.44 +- 0.09 20.43 +- 0.08 42.9%
6 x 128, short random client delay on 3 clients 537.8 +- 0.2 538.1 +- 0.4 7.50 +- 0.03 7.49 +- 0.03 10.84 +- 0.04 10.82 +- 0.01 13.6%
6 x 128, long random delay on all clients (FFN ranks wait for work) 372.6 +- 0.8 373.6 +- 0.7 3.00 +- 0.05 2.99 +- 0.02 7.77 +- 0.22 7.28 +- 0.21 1.6%
6 x 128, long random delay on 3 clients 434.5 +- 1.0 433.7 +- 2.3 3.04 +- 0.01 3.04 +- 0.01 8.51 +- 0.34 8.12 +- 0.36 2.7%

With the leader the two FFN ranks served the same order in every run (0% differing picks).

What the numbers say

  • Throughput does not change: every difference is within one standard deviation.
  • p99 latency is 4-6% lower with the leader in three layouts (512 tokens, mixed sizes, long delay on all clients), each by more than three standard errors. The 3 delayed clients row shows 5% with about two standard errors, so I do not count it. The other three rows show no difference.
  • p50 latency is 2-3% lower in two layouts (512 tokens, DBO) and unchanged elsewhere.
  • Correctness: with the leader, the synthetic suite passed with 2 and 3 FFN ranks in host and rounds modes (empty and skewed routes, DBO, delayed clients), and the real replay stayed within 0.009-0.010 relative error of the checkpoint in all five layouts I checked.

Limits

  • The order words in this experiment are host memory shared between the FFN processes, so it works on one node only. A version for the library needs these words in memory that the transport already shares between ranks (the NCCL window).
  • Everything ran on one pod and one node, with 2 FFN ranks in the timed runs.
  • The 512, mixed, DBO and long delay rows use 300 rounds per client, so a per-client p99 is the third largest sample of a run. The 128 token rows use 1000 rounds.
  • These are communication plus expert compute numbers, not serving numbers.

My reading: the followers do not need their own readiness check, and one order gives the same throughput with a somewhat lower tail latency in part of the layouts. I left the experiment out of this PR (it lives on a local branch on top of f0e44b0) because the one-node channel is a shortcut. I can do the NCCL window version as a follow-up PR if you want it in.

Native source digest of the timed binary: 8befb7fd041c6fd2a61c938f480fb7ba6b07223e8e33c1997ce01049769434c0.

Hey Almog — I’m not convinced the current numbers isolate the impact of mismatched FFN ordering. Since each Attention group needs all its FFN results before progressing, can we start with a minimal test — two FFN ranks and two Attention DP groups — comparing independent scheduling vs. leader ordering? We could introduce controlled delays so the FFNs choose opposite orders, then timestamp when each FFN finishes and when the client advances. Reporting per-client p99 and max latency would show the effect more directly. Then we can scale up to the larger workloads.

@ronenkat

Copy link
Copy Markdown
Collaborator

Lets start with the API and expected behavior. where is that?

@Etelis

Etelis commented Nov 17, 2026

Copy link
Copy Markdown
Member

Parallel readiness check: no gain.

readiness check

Weird.. Why is that?

@almogtavor

Copy link
Copy Markdown
Member Author

You are right, the throughput table does not isolate it. I ran the minimal case you proposed: 2 FFN ranks and 2 Attention units on 4 H100, real replay (Qwen/Qwen3.5-397B-A17B-FP8 layer 31 hidden states and router, ShareGPT V3 capture), host picks, native source digest b1324d32.

Setup, so that the order is the only thing that changes:

  • Both clients send one exchange at the same tick of a 100 ms wall-clock grid, and wait for it to complete before the next tick.
  • Each FFN rank holds until 5 ms after the tick, so both dispatches have arrived and the rank is free to choose either client. The latencies below include these 5 ms.
  • "same preference": both ranks begin with client 0. "opposite preference": FFN rank 1 begins with client 1, so the two ranks serve the two clients in opposite orders in every round. "leader": rank 1 follows the order of rank 0.
  • Timestamps are host wall-clock after a stream synchronize: client send, each FFN rank's finish for that client, client done. 300 rounds per run (first 2 dropped), 3 runs per row; p99 and max are per run, averaged over the 3 runs.

Latency in ms, client 0 / client 1:

tokens per client variant differing picks first FFN finish last FFN finish mean p99 max
128 same preference 0% 6.82 / 8.64 6.90 / 8.81 6.91 / 8.81 7.05 / 9.47 8.60 / 10.14
128 opposite preference 100% 6.89 / 6.83 8.65 / 8.80 8.66 / 8.81 8.98 / 9.05 9.65 / 10.42
128 leader 0% 6.83 / 8.72 6.92 / 8.83 6.92 / 8.83 7.23 / 9.82 8.61 / 10.63
512 same preference 0% 7.61 / 10.20 7.65 / 10.26 7.66 / 10.27 7.91 / 10.54 8.00 / 10.62
512 opposite preference 100% 7.63 / 7.60 10.21 / 10.25 10.22 / 10.26 10.39 / 10.50 11.14 / 11.58
512 leader 0% 7.61 / 10.23 7.65 / 10.28 7.66 / 10.28 7.85 / 10.54 8.99 / 12.14

What it shows:

  • A client advances right after its last FFN finish (the client done time is within 0.02 ms of it), so the last finish sets the latency.
  • With matching orders the first client is done after one FFN service time and the second after two. With opposite orders each client has one FFN result after one service time, but waits for the other rank, so both are done only after two service times. Client 0 goes from 6.91 to 8.66 ms at 128 tokens and from 7.66 to 10.22 ms at 512 tokens, which is one full FFN service time (about 1.8 and 2.6 ms). Client 1 does not change.
  • The slowest client and the max do not get worse. The cost is that the first client loses its head start: the mean over the two clients rises by about 0.9 ms (128 tokens) and 1.3 ms (512 tokens) per round.
  • With the leader order the result equals the matching case, and the cost of following is not visible here (within 0.02 ms).

Limits: this is the constructed worst case, where the orders differ in every round. Without the hold, in free-running runs with the same 2 clients, the two ranks picked the same order by themselves (0% differing picks) and the three variants were within run-to-run noise. This is communication-level evidence for one layer and not a serving result. The measurement script is a host-synchronous variant of tests/gpu/ffn_service.py and is not in the PR; I can add it if you want it there.

Next I can scale the same per-client measurement to 6 units and 4 FFN ranks, where the differing picks appear without any forcing (11% to 45% in the table of the description).

@Etelis Etelis changed the title FFN as a Service (FaaS) FFN Scheduling Nov 17, 2026
@almogtavor

Copy link
Copy Markdown
Member Author

Lets start with the API and expected behavior. where is that?

Today it is in two places:

  • API: the docstrings of FfnService, client_row and order_ring in afdep/service.py.
  • Expected behavior: the numbered points at the top of this PR description, and the module docstring of afdep/service.py.

In short: an FFN rank serves the first client whose dispatch has arrived (FCFS), and takes turns (round robin) when several are waiting. With order=order_ring(buffer), every FFN rank serves the order chosen by FFN rank 0.

docs/api.md has no section for the service yet. I can add one to this PR if you prefer the contract there.

@almogtavor

Copy link
Copy Markdown
Member Author

Weird.. Why is that?

Because the check was never the slow part. One pass reads a few arrival counters per client, and the kernel repeats it until a client has sent. The time goes to waiting for the next dispatch to arrive and to the exchange itself, so checking all clients in parallel finds the same client at the same moment.

The numbers agree: ms per exchange, serial / parallel, is 8.13 / 8.11 with 2 units, 2.02 / 2.03 with 6, 1.73 / 1.73 with 12 and 1.65 / 1.65 with 24.

@almogtavor almogtavor changed the title FFN Scheduling Async FaaS Scheduling Nov 17, 2026
The service test takes its Buffers from afdep.Units when the units are equal
and have no micro-batches; --manual-buffers keeps the Buffers built in the test.
The help names the two policies: take everyone (static) and take first ready
(host, device).
@almogtavor almogtavor changed the title Async FaaS Scheduling Take First Ready policy for the FFN ranks Nov 17, 2026
@almogtavor
almogtavor changed the base branch from main to buffer-per-attention-unit November 17, 2026 16:07
@almogtavor
almogtavor marked this pull request as ready for review November 17, 2026 18:10
@almogtavor
almogtavor marked this pull request as draft November 17, 2026 18:32

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants