Align dormant lab/evolution skills with the pipeline: when-to-use guidance, map-debt bridge, description trim, analysis.ipynb repair #642

Summary

The skill-level "evolution lab" tier of agent-agentic-os (os-eval-lab-setup, os-eval-runner, os-eval-backport, os-skill-improvement, os-improvement-loop, os-evolution-planner, os-evolution-verifier, os-experiment-log, os-improvement-report, plus os-architect/evo-smoketest) was built to run single-variable experiments on skills or scripts with agent loops and evidence. In day-to-day work the pipeline (work-intake, worktree, control-plane gates, map-debt.md, GitHub issues, transition-guidance-tester) has become the real improvement loop, and the lab skills are largely dormant. Their guidance does not say when they still apply, they add always-loaded routing surface, and one asset is broken. This issue tracks aligning them with current usage. It is the skill-level counterpart to #548 (pipeline-level learning) and does not replace it.

Observed Behavior

Findings from a review during the AUTH-CIBA-INCREMENT-B session (2026-09-19); there is no invocation telemetry, so "rarely used" is the owner's recollection and is not measured:

• Experiment log: context/experiment-log/ has 6 entries; the five listed were dated 2026-04-25 to 2026-04-27 (RSVP speed-reader lab, ecosystem validation, verifier round 1). Nothing since.
• Tests: none of the lab skills is referenced by any test in plugins/agent-agentic-os/tests/ (compare work-intake 12 files, os-init 6, self-evolution 6). Their only regression net is their own evals.
• Commits: 3-6 commits each since July 2026, but they look like bulk maintenance (renames, symlink migrations, the python3 -> python sweep), not use.
• Context footprint: the nine lab skills add about 3,500 characters of always-loaded descriptions (plus os-architect, evo-smoketest), which works against P1: Reduce static context usage through progressive elaboration #593 (reduce static context usage).
• map-debt relation: os-evolution-planner and os-evolution-verifier mention map-debt only as a completion-checklist / Evolution Integrity Gate item; nothing in the lab reads references/map-debt.md to choose or verify experiments. Root references/map-debt.md has 40 entries.
• Broken asset: plugins/agent-agentic-os/scripts/analysis.ipynb (the interactive companion to generate_report.py, spoked into os-improvement-report) is invalid JSON since commit 06c3766 ("updating references from python3 to python"), which mangled a line into python "python" at line 186. The two prior commits are valid.
• Not corruption (avoid false alarms): a scan of 212 non-symlink JSON/ipynb files under plugins/ found 13 more "invalid JSON" files; they are one-line text stand-ins (e.g. ../evals/evals.json), not damage.
• Dependencies: pandas and matplotlib are still imported by generate_report.py and plot_eval_progress.py (the reporting skill); numpy and pillow are transitive. No other plugin uses them.

Expected Behavior

Each lab skill states when it should be used and when it should not; the always-loaded description footprint is smaller; the lab is connected to the friction loop actually in use; and broken assets are repaired. No skill is removed, renamed or consolidated without an explicit per-path instruction.

Recommendation

1. Guidance-only refactor first. Add a "When to use / when NOT to use" block to each lab skill: use it when a measurable skill/script quality (trigger routing, eval score) is in question and a single-variable experiment with evidence is possible; do not use it for control-plane bugs or friction (those go to TDD/pytest, then map-debt.md or a GitHub issue).
2. Trim descriptions so they route only on lab intent, and add should-not-trigger eval cases (relates to Skill routing eval is pure keyword-overlap — no semantic/paraphrase tolerance #461: routing eval is keyword-overlap).
3. Connect the lab to map-debt. Friction is logged as map debt; when the fix is a measurable skill/script change, run a single-variable lab experiment (os-eval-runner) and record the evidence in the map-debt entry as Status: RESOLVED. os-evolution-planner's gap-detection lens can read open entries (by Severity, Repeat: YES) instead of a fixed checklist; os-experiment-log records those experiments.
4. Pilot on 3-4 skills, measure routing evals, then decide on any rename (e.g. a lab- prefix) or consolidation into one umbrella skill with progressive disclosure, from evidence. Rename cost is high (symlinks.json, plugin.yaml, evals, os-*-detailed-reference.md names, installed copies).
5. Repair analysis.ipynb (one-line JSON fix; do not delete).
6. Consider lightweight invocation telemetry so "rarely used" becomes measurable (coordinate with Design a two-tier evolution/improvement architecture: skill-level (existing) vs. pipeline-level (new) learning #548 / control_plane.db is gitignored with no durable export — approval/receipt audit trail is local-only, not git-trackable #519 local-records-first approach).

Acceptance Criteria

• Every lab skill has a when-to-use / when-not-to-use block referencing the pipeline alternative.
• Description footprint reduced with should-not-trigger evals passing (python3 plugins/agent-agentic-os/scripts/eval_runner.py --skill <skill-path> exits 0, no regression vs baseline).
• A documented map-debt -> experiment -> RESOLVED-with-evidence procedure exists and is exercised once on a real open entry.
• analysis.ipynb opens as valid JSON.
• Any rename/move/delete/consolidation happens only with explicit user instruction naming the exact path (destructive-action guard; routing identity, eval contract and methodology are not interchangeable between skills).

Evidence

• git log --follow -- plugins/agent-agentic-os/scripts/analysis.ipynb; python3 -c "import json; json.load(open(...))" fails at line 186 col 13.
• Per-skill description lengths, commit counts and test references measured with git log and grep -rl <skill> plugins/agent-agentic-os/tests.
• ls context/experiment-log/ (6 entries, April 2026).
• grep -rn "generate_report\|plot_eval_progress" plugins/agent-agentic-os --include='*.py' shows the only importers of pandas/matplotlib.

Impact

Not blocking. Ongoing cost: unclear guidance on when to use the lab versus the pipeline, extra always-loaded routing text, a broken notebook, and an evidence-based improvement path (lab experiments) that is disconnected from the map-debt loop that is actually used. Out of scope for #639 (AUTH-CIBA-INCREMENT-B); tracked separately.

Related

#548 (two-tier evolution/improvement architecture: this is the skill-level tier), #593 (static context reduction), #461 (routing eval is keyword-overlap), #537 (self-evolution vs control-plane duplication).