Add AsyncDistillationTrainer - #7

)

)

Co-authored-by: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

…gface#6497)

…contained (huggingface#6508)

…ngface#6509)

Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

…gingface#6510)

…s (add-only) (huggingface#6511)

…face#6518)

…te (add-only) (huggingface#6512)

)

Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

…nenv_harness.py (huggingface#6531)

…6493)

…uggingface#6513)

…_completions` (unwired) (huggingface#6521)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: quentin.gallouedec@huggingface.co <quentin_gallouedec@ip-10-53-86-250.ec2.internal>

…ingface#6534)

…_generation != gradient_accumulation_steps (huggingface#6024)

Co-authored-by: Behrooz Azarkhalili <ermiaazarkhalili@gmail.com>
Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>
Co-authored-by: Quentin Gallouédec <quentin.gallouedec@huggingface.co>

…ers dev build (huggingface#6542)

Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

…ce#6547)

…e#6121)

Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>

…ngface#6507)

…6726)

…#6727)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…utput (huggingface#6686)

Co-authored-by: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com>

`trl vllm-serve`'s custom app, and with it `/get_sequence_logprobs/`, goes
away in huggingface#6765, so the rollout worker teacher-forces the
student's completion through vLLM's own endpoint instead: `max_tokens=1`
with `prompt_logprobs=teacher_top_k`, sliced from `len(prompt_ids)`. Same
request `VLLMClient.get_sequence_logprobs` issues for the synchronous
server-teacher trainers.

The response is parsed here rather than through that client: it truncates
each position to exactly `top_logprobs` after rank-sorting and reports the
realized token in separate fields, which would drop that token wherever it
ranks below k, zeroing the teacher signal there at `beta=1.0`. Parsing the
raw `prompt_logprobs` mapping keeps it at any rank, so the candidate rows
stay `teacher_top_k + 1` wide and `_narrow_top1_actual_support` keeps
finding what it indexes.

`/v1/completions` names the model it addresses and, under MOPD, every
teacher serves a different one, so each teacher's served id is resolved
from its `/v1/models` at worker startup. That doubles as the teachers'
readiness wait.

Teacher servers are now plain `vllm serve` and need `--logprobs-mode
processed_logprobs` (or `teacher_temperature` never reaches their
logprobs) and `--max-logprobs -1` (or `teacher_top_k > 20` is rejected).
Documented, along with the pre-existing requirement that every teacher
share the student's tokenizer, since teacher candidate ids index the
student's vocabulary directly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…face#6748)

…6753)

…uggingface#6749)

…ctions (huggingface#6672)

…ce#6754)

)

…file (huggingface#6750)

)

…ainer

Upstream huggingface#6715 reworked what AsyncGRPO measures; huggingface#6674 reworked how the sync
distillation trainer reads a model's logit post-processing. Both apply here.

Metrics, following huggingface#6715's namespaces (`rollout/`, `completions/`, `sample/`,
`batch/`, `perf/`) and its reduction rule — the value's shape and the key's
suffix pick the reduction, so a rate is stored as a `(num, den)` pair and
reduced as Σnum/Σden instead of as a mean of ratios:

- the worker gets a `metrics_queue` the trainer drains in `log()`, so it can
  report what only it can see: `rollout/duration_s`, `rollout/score_s`,
  `rollout/generated_tok_s` (windowed), `rollout/inflight`,
  `rollout/backpressure_s`, `rollout/vllm_retry_total` and the
  `completions/*` family.
- per-sample gauges no longer ride on `RolloutSample.metrics` and are no
  longer NaN-padded into the batch, broadcast to every rank and reduced back
  to compute a number rank 0 already had. The collator logs the sample and
  packing metrics directly; `sample.enqueued_at` gives the trainer
  `sample/time_in_queue_s`.
- new per-step accounting (`perf/step_s`, `fwd_bwd_s`, `fwd_s`,
  `optimizer_s`, `rollout_wait_s`, the three weight-sync phases) plus
  throughput and MFU on both a compute and a wall-clock basis, which is what
  separates a slow trainer from a starved one.
- trainer-side queue metrics (`sample/rollout_queue_size`,
  `staleness_*`, `dropped_stale_total`, `batch/dropped_oversize_total`).
- traces are sampled per policy version rather than per N samples, which at
  a few hundred samples/step buried the real metrics under empty rows.

Two metrics are distillation's own, since the teacher call sits on the
critical path of every rollout rather than in a separate scoring loop:
`rollout/score_s`, and `teacher_score_s/{teacher_id}` under MOPD — one slow
expert throttles only the rollouts routed to it, which the blended mean
hides. `docs/source/async_distillation_trainer.md` documents every metric.

Also in this commit:

- read `logit_scale` / `final_logit_softcapping` through `get_text_config()`
  and fall back to Muse Glimmer's `output_multiplier`, matching
  `DistillationTrainer`.
- drop `# docstyle-ignore` from `AsyncDistillationConfig` (huggingface#6785) and let the
  styler reflow it. Two pre-existing line-break artifacts it exposed — a
  broken markdown link and a broken class reference — are fixed rather than
  reflowed into literal spaces.
- raise `AsyncRolloutWorker`'s vLLM gate from 0.17.1 to 0.22.0, which is what
  `weight_transfer.py` and the docs already require and which is above TRL's
  own supported floor.
- drop a stale `trl vllm-serve` mention left over from the huggingface#6765 migration.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>