Repository navigation
docs: add staged adoption guide - #415
Merged
NandhaKishorM merged 2 commits intoSep 25, 2026
Merged
Conversation
Owner
|
Thank you for writing this up, it is exactly the guide #367 asked for. I ran the shadow hook against the Router and every context field it logs is there, and the note about capturing failures outside the hook lifecycle is correct. It keeps the threshold decision with the application and never suggests a fixed number, which is the message we want. Merging. |
NandhaKishorM
pushed a commit
that referenced
this pull request
Sep 27, 2026
Implements guide 5 from [#390](#390), which asks for "a general walkthrough of the notebook, fitting calibration temperatures, and pushing a checkpoint to the Hub, alongside the existing browser-agent example". Nobody has claimed it in the thread: @emiliano-go is not working on any of the six, and @Bruce-Yii took the command line + MCP guide (with #415 for staged rollout). Every number on the page is from committed material -- the README's 0.36/0.35/0.766 and 4-5 h runtime, the notebook's own cells (slice sizes, hyper-parameters, the 4-6 minute demo run, the fixed comparison rows) -- or a public issue number. Nothing is estimated, and no training was run to write it. ## What the page covers - **The notebook end to end**, as a table: what each step does, the Kaggle settings, and where the outputs land. - **The training recipe**: gold distributions as targets, the GRPO-style policy gradient with proper-scoring rewards, the soft cross-entropy term, and the knob table for a 16 GB card. - **Calibration is part of the run**, the page's centre: the slice is held out *before* the shard (fitting on trained items returns a degenerate scale), the fit is per question type and clamped, and the inherited `temperature_by_options` is removed in the same write because it would take precedence at inference. `tests/test_calibration_persistence.py` is the regression. - **Evaluating**: the test-split pass, the metric list, the fixed reference rows, and the two habits -- hold out your own slices, report calibration next to accuracy. - **Pushing to the Hub**, and **what to watch** (targets as the ceiling, the small calibration slice, primitive limits, config persistence). Deliberate choices tested: - **One nav edit.** The Fine-tuning section exists and already holds the browser-agent page, so `finetune.md` is listed beside it rather than left to the Guides glob -- #390's rule is that an unlisted page shows up under Guides and can be "placed in another section when needed". - **No link out of the docs tree, and none into pages still in review.** `BENCHMARKS.md` is plain text for the same reason as the Routing guide (zensical `--strict` resolves relative links), and links to the Questions and answers guide (#418, in review) failed both gates -- zensical aborted, and `test_packaging.py` named the missing target -- so the primitives' limits are linked as issues (#394, #377) instead. - **It expands the README's Fine-Tuning section rather than copying it**, per #390 ("the README summaries kept in place"): the walkthrough, the calibration order and the publish cell are notebook-level detail the README does not carry. ``` $ zensical build --strict --clean exit 0, no griffe lines $ python tests/test_packaging.py all packaging tests passed $ python tests/test_doc_tables.py 54 passed, 0 failed $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! $ python -m compileall -q laya/ tests/ ``` Known limitation: this is the fine-tuning walkthrough only. The other guides #390 lists -- HTTP API (which should wait for #304/#281 to settle), command line and MCP (@Bruce-Yii; #415 landed for staged rollout), benchmarks -- are not covered here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
docs/staged-adoption.mdguide for shadow observation, incumbent comparison, held-out threshold policy, and bounded promotion.docs/index.md.Why
Laya already exposes the decision evidence and hooks needed to evaluate adoption, but the path from shadow observation to a bounded real-world slice is not documented. This guide makes the application-owned rollout boundary explicit: high confidence is not execution permission, and irreversible/high-cost actions retain a review boundary.
How it was verified
python tests/test_packaging.py— 42 passed, 0 failedpython tests/test_doc_tables.py— 54 passed, 0 failedpython -m compileall -q laya tests— passedpython -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120— passedzensical build --strict --clean— no issues foundgit diff --check— passedBase:
970dc8c5f63d7b886a68409493f37d569424f933Branch:
docs/staged-adoption-guideCommit:
f8b3d8071f9c5f27260c2c35ab618ff8927a4053Closes #367
Checklist
main