Skip to content

docs: add staged adoption guide - #415

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
Bruce-Yii:docs/staged-adoption-guide
Sep 25, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
Bruce-Yii:docs/staged-adoption-guide

Conversation

@Bruce-Yii

@Bruce-Yii Bruce-Yii commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

What

  • Add a short docs/staged-adoption.md guide for shadow observation, incumbent comparison, held-out threshold policy, and bounded promotion.
  • Link the guide from docs/index.md.
  • Keep the change documentation-only; no runtime or public API changes.

Why

Laya already exposes the decision evidence and hooks needed to evaluate adoption, but the path from shadow observation to a bounded real-world slice is not documented. This guide makes the application-owned rollout boundary explicit: high confidence is not execution permission, and irreversible/high-cost actions retain a review boundary.

How it was verified

  • python tests/test_packaging.py — 42 passed, 0 failed
  • python tests/test_doc_tables.py — 54 passed, 0 failed
  • python -m compileall -q laya tests — passed
  • python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 — passed
  • zensical build --strict --clean — no issues found
  • git diff --check — passed

Base: 970dc8c5f63d7b886a68409493f37d569424f933
Branch: docs/staged-adoption-guide
Commit: f8b3d8071f9c5f27260c2c35ab618ff8927a4053

Closes #367

Checklist

  • Focused on one change
  • Rebased on the latest main
  • Tests pass locally
  • Documentation-only change; no public API change

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for writing this up, it is exactly the guide #367 asked for. I ran the shadow hook against the Router and every context field it logs is there, and the note about capturing failures outside the hook lifecycle is correct. It keeps the threshold decision with the application and never suggests a fixed number, which is the message we want. Merging.

@NandhaKishorM
NandhaKishorM merged commit 13840cb into NandhaKishorM:main Sep 25, 2026
21 checks passed
NandhaKishorM pushed a commit that referenced this pull request Sep 27, 2026
Implements guide 5 from [#390](#390), which asks
for "a general walkthrough of the notebook, fitting calibration temperatures, and pushing a
checkpoint to the Hub, alongside the existing browser-agent example". Nobody has claimed it in
the thread: @emiliano-go is not working on any of the six, and @Bruce-Yii took the command
line + MCP guide (with #415 for staged rollout).

Every number on the page is from committed material -- the README's 0.36/0.35/0.766 and 4-5 h
runtime, the notebook's own cells (slice sizes, hyper-parameters, the 4-6 minute demo run, the
fixed comparison rows) -- or a public issue number. Nothing is estimated, and no training was
run to write it.

## What the page covers

- **The notebook end to end**, as a table: what each step does, the Kaggle settings, and where
  the outputs land.
- **The training recipe**: gold distributions as targets, the GRPO-style policy gradient with
  proper-scoring rewards, the soft cross-entropy term, and the knob table for a 16 GB card.
- **Calibration is part of the run**, the page's centre: the slice is held out *before* the
  shard (fitting on trained items returns a degenerate scale), the fit is per question type and
  clamped, and the inherited `temperature_by_options` is removed in the same write because it
  would take precedence at inference. `tests/test_calibration_persistence.py` is the regression.
- **Evaluating**: the test-split pass, the metric list, the fixed reference rows, and the two
  habits -- hold out your own slices, report calibration next to accuracy.
- **Pushing to the Hub**, and **what to watch** (targets as the ceiling, the small calibration
  slice, primitive limits, config persistence).

Deliberate choices tested:

- **One nav edit.** The Fine-tuning section exists and already holds the browser-agent page, so
  `finetune.md` is listed beside it rather than left to the Guides glob -- #390's rule is that
  an unlisted page shows up under Guides and can be "placed in another section when needed".
- **No link out of the docs tree, and none into pages still in review.** `BENCHMARKS.md` is
  plain text for the same reason as the Routing guide (zensical `--strict` resolves relative
  links), and links to the Questions and answers guide (#418, in review) failed both gates --
  zensical aborted, and `test_packaging.py` named the missing target -- so the primitives'
  limits are linked as issues (#394, #377) instead.
- **It expands the README's Fine-Tuning section rather than copying it**, per #390 ("the
  README summaries kept in place"): the walkthrough, the calibration order and the publish
  cell are notebook-level detail the README does not carry.

```
$ zensical build --strict --clean        exit 0, no griffe lines
$ python tests/test_packaging.py         all packaging tests passed
$ python tests/test_doc_tables.py        54 passed, 0 failed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```

Known limitation: this is the fine-tuning walkthrough only. The other guides #390 lists --
HTTP API (which should wait for #304/#281 to settle), command line and MCP (@Bruce-Yii; #415
landed for staged rollout), benchmarks -- are not covered here.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Clarify a staged rollout pattern for adopting Laya decisions

2 participants