Repository navigation
Take First Ready policy for the FFN ranks - #33
almogtavor wants to merge 11 commits into
Conversation
One resident launch per run picks clients like the rounds method and does the receive and the reply itself. The experts run as CUDA graphs on a second stream (persistent kernel) or inside the same launch (megakernel, FP32 without tensor cores).
The megakernel reads each 16x8 tile as 256 contiguous bytes, claims column blocks per warp without staging rows in shared memory, spreads the reply copy over all warps, and is compiled with 64 registers so four CTAs fit per SM. build_resident rejects a grid that cannot be resident at once. On the Qwen3.5-397B-A17B-FP8 layer 31 replay (ShareGPT V3) it reaches 558 exchanges/s at expert alignment 1 and 559 at 128; rounds reach 538 and 267.
| const int* state, const int* quota, | ||
| const int rotate) { | ||
| bool pending = false; | ||
| for (int i = 1; i <= num_clients; ++ i) { |
There was a problem hiding this comment.
Would one lane per client with a warp ballot make sense here, still using one CTA? I’m curious whether it would cut ready-to-receive latency with staggered arrivals or wider DP groups.
There was a problem hiding this comment.
You have one thread scanning all the rest idle iiuc.
Why not scan in parallel?
| "all_active_per_s": active / min(r["wall_s"] for r in clients), | ||
| "all_active_served": served, | ||
| "client_rates": rates, | ||
| "order_divergence": diverged / total, |
There was a problem hiding this comment.
Do you think the FFNs need a shared client order here? I’m curious what happens if one serves A then B and the other serves B then A, in both graph and resident modes.
There was a problem hiding this comment.
Two FFN workers should they do same requests or not?
| } | ||
| if (received == replied and choice != -4) | ||
| end = choice; | ||
| else if (timer() - start > timeout_ns) |
There was a problem hiding this comment.
What happens if a client arrives just as one FFN times out and another picks it up? Would the client still get all its replies?
| if mode == "push" and rank == PRODUCER: | ||
| src = [buffer.data_ptr()] * (world - 1) | ||
| dst = [peers[r].data_ptr() for r in range(world) if r != PRODUCER] | ||
| elif mode == "pull" and rank != PRODUCER: |
There was a problem hiding this comment.
What do you think about Attention pulling and reducing the same reply rows? Would you expect the FFN to move on sooner, and how would a slow reader affect Attention tail latency and Buffer reuse?
|
Remove the megakernel and fusion pls. |
|
DBO, how will we schedule different lanes? |
The resident kernel that also computes the experts covers only the FFN side, so the mode is now --mode fused.
The resident kernel no longer computes the experts itself. It lost to rounds for every batch above 128 tokens per client (0.82x at 512 tokens, 0.38x at 2048), so the persistent kernel keeps only the pick, the receive and the reply, and the expert graphs run on the second stream.
Rounds (one CUDA graph, 32 rounds per launch) was faster than the resident kernel at both expert alignments, so the service keeps the scan kernel only. get_afd_service_words had no other caller.
|
Hi, this is the measurement for the review question about a shared client order between the FFN ranks. Today: every FFN rank picks the next client by itself from its own arrival counters, so two FFN ranks can serve the clients in different orders. Experiment: FFN rank 0 (the leader) picks as before and publishes each pick; the other FFN ranks do not check readiness and serve the published order. The kernel change is small: // leader, when it records pick i:
order[i] = choice + 1;
// follower, for its pick i (0 means not published yet):
const int published = order[state[2]];
return published ? published - 1 : -4;Setup: one pod, 8x H100, 2 FFN ranks and 6 clients, Qwen/Qwen3.5-397B-A17B-FP8 layer 31 experts replayed on ShareGPT V3 tokens, expert alignment 1, rounds mode (32 rounds per launch). Both modes run from the same binary, 6 runs each, alternating which mode runs first. Values are mean +- standard deviation over the 6 runs. Latency is the client side time of one exchange, averaged over the clients.
With the leader the two FFN ranks served the same order in every run (0% differing picks). What the numbers say
Limits
My reading: the followers do not need their own readiness check, and one order gives the same throughput with a somewhat lower tail latency in part of the layouts. I left the experiment out of this PR (it lives on a local branch on top of f0e44b0) because the one-node channel is a shortcut. I can do the NCCL window version as a follow-up PR if you want it in. Native source digest of the timed binary: |
FFN rank 0 writes each pick into a ring of free arrival words of the first client Buffer. The other FFN ranks read its copy of those words through the NCCL window (ncclGetLsaPointer in the readiness kernel) and serve the same client, so every FFN rank serves in one order. The option is off by default; tests/gpu/ffn_service.py --leader turns it on and asserts that no pick differs between FFN ranks.
Hey Almog — I’m not convinced the current numbers isolate the impact of mismatched FFN ordering. Since each Attention group needs all its FFN results before progressing, can we start with a minimal test — two FFN ranks and two Attention DP groups — comparing independent scheduling vs. leader ordering? We could introduce controlled delays so the FFNs choose opposite orders, then timestamp when each FFN finishes and when the client advances. Reporting per-client p99 and max latency would show the effect more directly. Then we can scale up to the larger workloads. |
|
Lets start with the API and expected behavior. where is that? |
|
You are right, the throughput table does not isolate it. I ran the minimal case you proposed: 2 FFN ranks and 2 Attention units on 4 H100, real replay (Qwen/Qwen3.5-397B-A17B-FP8 layer 31 hidden states and router, ShareGPT V3 capture), host picks, native source digest Setup, so that the order is the only thing that changes:
Latency in ms, client 0 / client 1:
What it shows:
Limits: this is the constructed worst case, where the orders differ in every round. Without the hold, in free-running runs with the same 2 clients, the two ranks picked the same order by themselves (0% differing picks) and the three variants were within run-to-run noise. This is communication-level evidence for one layer and not a serving result. The measurement script is a host-synchronous variant of Next I can scale the same per-client measurement to 6 units and 4 FFN ranks, where the differing picks appear without any forcing (11% to 45% in the table of the description). |
Today it is in two places:
In short: an FFN rank serves the first client whose dispatch has arrived (FCFS), and takes turns (round robin) when several are waiting. With
|
Because the check was never the slow part. One pass reads a few arrival counters per client, and the kernel repeats it until a client has sent. The time goes to waiting for the next dispatch to arrive and to the exchange itself, so checking all clients in parallel finds the same client at the same moment. The numbers agree: ms per exchange, serial / parallel, is 8.13 / 8.11 with 2 units, 2.02 / 2.03 with 6, 1.73 / 1.73 with 12 and 1.65 / 1.65 with 24. |
The service test takes its Buffers from afdep.Units when the units are equal and have no micro-batches; --manual-buffers keeps the Buffers built in the test. The help names the two policies: take everyone (static) and take first ready (host, device).
This PR adds the Take First Ready policy for the FFN ranks, on top of #40 (one Buffer per Attention unit). The existing behavior is named Take Everyone.
What
FfnServicedoes:--mode device/--mode hostin the test).order=order_ring(buffer), test flag--leader): FFN rank 0 writes each pick into free arrival words of the first unit's Buffer, and the other FFN ranks read its copy through the NCCL window (ncclGetLsaPointerin the readiness kernel) and serve the same unit. Off by default.Results on #40
Sum over units of rounds per second, median of 5 repeats.
Expert alignment 1:
Expert alignment 128 (the default):
Method
Units; they match theUnitscolumns within 1% and are left out of the tables.b1324d3262a904e7d7c6d77a0b324bb269eeca4872a0133fc3db180f152c179e.These are communication plus expert compute numbers, not serving numbers.
Earlier results, before the move onto #40 (commit ab769d3, expert alignment 1)
Exchanges/s while all units are active, 300 rounds per unit.
With the leader order no pick differs between the FFN ranks, and the test asserts that. Output check on the same replay: relative error 0.009 to 0.010 against the checkpoint with 2 and 4 FFN ranks, with and without the leader order. The real replay with 3 FFN ranks is not validated (the synthetic 3-rank check passes).
Recorded experiment: persistent kernel (code not in this PR)
One kernel launch for the whole run that picks, receives and replies itself reached 0.73-0.79x of rounds (395 against 539 at 6 x 128), so I removed it. The last commit that contains it is f8b992d.