fix(engine): batch 4 report-quality bugs from session weakness analysis#21
Conversation
…is [skip ci] Path forward for weaknesses identified in session 2026-05-22 (paths #1 + #4 verification). 4 small additive fixes shipped together; 1 deferred to issue #20 (artifacts wiring is structural, not a 5-line fix). ## Fixes ### C2: CLI --plugins comma-separated parsing (run.ts) Before: `--plugins cwv,axe,third-parties` → '[ohmyperf] WARN unknown --plugins value; defaulting to all'. After: comma-separated tokens parsed individually. Individual plugin names (cwv, axe, third-parties, example) now accepted as standalone arg values. Preset names (all, cwv+axe, none) preserved. Duplicate-token elimination via Set on plugin.id. Implementation: split resolvePluginSet into resolveSinglePluginToken (per- token resolver returning null on unknown) + comma-split outer function that aggregates + deduplicates. Updated --plugins description to document both forms. ### C3: AuditResult.status field (types.ts + axe.ts + third-parties.ts) Before: `audits[i]` JSON has only `passed: boolean` and `score: number|null`. LLM/dashboard consumers reading `.status` got undefined (which displays as null in some JSON tooling). After: new `AuditStatus = "pass" | "fail" | "na"` exported type; new `status: AuditStatus` required field on AuditResult. Populated by both built-in plugins (axe derives from violation count, third-parties always passes for now). Test fixture in plugin-runtime.test.ts updated. Schema 1.0.0 compat: this is a *new required field* on AuditResult. Strictly speaking it breaks the schema, but AuditResult is only constructed by plugins (not consumed external to engine) and all 3 in-tree plugins + test fixture updated. External plugins compiled against old types will fail TS — desired behavior so they get the fix. ### C4: fixPlan grouped view (types.ts + fix-plan.ts + tests) Before: 18 entries all archetype `render-blocking-stylesheet-media-print`, one per URL. LLM agents reading top-N saw same-archetype noise. After: collapse same-(archetype, expectedMetric, originClass) entries into 1 grouped entry. New OPTIONAL `targets: Array<{url, originClass?, expectedImpactMs}>` field on FixPlanEntry preserves all URLs. Primary `target` field unchanged (highest-ROI representative). `expectedImpactMs` now reflects SUM across all targets when grouped. Single-URL entries omit `targets` (undefined). Verified end-to-end against tradeit.gg: - before: 18 entries - after: 1 entry, targets_count=18, expectedImpactMs=1923 ### B6: Document dev keypair multi-developer/CI implications Extended setup-dev-extension.mjs final log with a clear warning block: each developer's keypair is unique → CI ID will differ → SPA NEXT_PUBLIC_EXTENSION_ID must match keypair's ID OR test:e2e:extension will fail on teammate machine. Documents bundle-extension.mjs's actual behavior (key field injected only when .dev-keys/extension.pem present), NOT a false "stripped at zip time" claim em initially wrote then corrected after grep verification. ## Deferred to issues - #14 A1 multi-run extension (4h, SW state persistence) - #15 B1 wire test:e2e:extension into CI (xvfb infra) - #16 B2 periodic live-deploy smoke (cron workflow) - #17 A4 polling fallback for mid-session extension install (UX) - #18 C1 host CPU profile in report.meta (calibration plumbing) - #19 C6 trustScore in UI (viewer) - #20 C5 artifacts wiring (engine structural, not 5-line accessor) ## Verification (Evidence Receipts) layer: validate:quick pnpm -r typecheck → clean pnpm -r lint → clean (No ESLint warnings or errors) layer: test:integration pnpm --filter @ohmyperf/core test → 94/94 PASS Updated W4 + W5 tests assert new grouped contract. layer: test:cross-cutting-allowlists scripts/check-cross-cutting-allowlists.sh → 3/3 PASS layer: real-world CLI execution ohmyperf run https://tradeit.gg --plugins "cwv,axe,third-parties" → no plugin-arg warn → audits[].status populated ('fail'/'pass') → fixPlan 1 entry (was 18) ## Counts 9 files changed, 143 insertions(+), 19 deletions(-) All schema-compat-safe (status field is new required but in-tree only; targets field is optional additive).
There was a problem hiding this comment.
Code Review
This pull request enhances the CLI to support comma-separated plugin lists, introduces an AuditStatus field to core types and built-in plugins, and implements logic to collapse similar fix plan entries into grouped targets. It also adds documentation regarding extension ID stability in development environments. Feedback from the review suggests dynamically generating the plugin help text to avoid staleness, reusing existing resolution logic for the default 'all' case, and ensuring that roiScore is aggregated alongside impact when grouping fix plan entries to maintain data consistency.
| } | ||
| } | ||
| if (unknown.length > 0) { | ||
| logger.warn(`unknown --plugins token(s) ${JSON.stringify(unknown)}; ignored. Known: cwv, axe, third-parties, example, all, cwv+axe, none.`); |
There was a problem hiding this comment.
| const resolved = resolveSinglePluginToken(trimmed); | ||
| if (resolved !== null) return resolved; | ||
| logger.warn(`unknown --plugins value '${name}'; defaulting to 'all'`); | ||
| return [cwvPlugin(), axePlugin(), thirdPartiesPlugin(), customMetricExamplePlugin()]; |
There was a problem hiding this comment.
To improve maintainability and reduce duplication, consider reusing resolveSinglePluginToken("all") instead of hardcoding the full plugin list again as a fallback.
| return [cwvPlugin(), axePlugin(), thirdPartiesPlugin(), customMetricExamplePlugin()]; | |
| return resolveSinglePluginToken("all")!; |
| const totalImpactMs = all.reduce((acc, x) => acc + x.expectedImpactMs, 0); | ||
| const targets = all | ||
| .map((x) => { | ||
| const t: { url: string; originClass?: OriginClass; expectedImpactMs: number } = { | ||
| url: x.target.url, | ||
| expectedImpactMs: x.expectedImpactMs, | ||
| }; | ||
| if (x.target.originClass !== undefined) t.originClass = x.target.originClass; | ||
| return t; | ||
| }) | ||
| .sort((a, b) => b.expectedImpactMs - a.expectedImpactMs); | ||
| return { | ||
| ...bucket.primary, | ||
| expectedImpactMs: totalImpactMs, | ||
| targets, | ||
| }; |
There was a problem hiding this comment.
The roiScore should be aggregated when collapsing entries, similar to how expectedImpactMs is summed. Currently, the grouped entry retains the roiScore of only the primary (highest-ranked) item, which makes the report's ROI data inconsistent with the total impact and potentially misleading for consumers (like LLMs or dashboards) that might use this score for secondary sorting or prioritization.
const totalImpactMs = all.reduce((acc, x) => acc + x.expectedImpactMs, 0);
const totalRoiScore = all.reduce((acc, x) => acc + x.roiScore, 0);
const targets = all
.map((x) => {
const t: { url: string; originClass?: OriginClass; expectedImpactMs: number } = {
url: x.target.url,
expectedImpactMs: x.expectedImpactMs,
};
if (x.target.originClass !== undefined) t.originClass = x.target.originClass;
return t;
})
.sort((a, b) => b.expectedImpactMs - a.expectedImpactMs);
return {
...bucket.primary,
expectedImpactMs: totalImpactMs,
roiScore: totalRoiScore,
targets,
};Oracle 3-Angle review (bg_38c12f04, 1m55s) returned 2 blockers + 1 important + 2 nits. All 5 addressed in this commit. ## Blockers fixed ### B1: AuditResult.status missing in 4 test fixture literals - packages/viewer/src/render.test.ts:75 — added status: "fail" - packages/reporter-markdown/src/render.test.ts:46 — added status: "fail" - packages/reporter-markdown/src/render.test.ts:103 — added status: "pass" - packages/core/src/llm-signals/llm-signals.test.ts:293 — added status: "fail" Oracle correctly noted these would fail TS compile after Fix #2 made status required. Em's initial gate-check (pnpm -r typecheck) succeeded because some test packages have their own tsc skip patterns (vitest transpiles on-the-fly), but the gate would have failed in CI without this fix. ### B2: collapseSameArchetype confidence data loss - packages/core/src/llm-signals/fix-plan.ts:228-263 Before: when 2 fixPlan drafts collapsed (same archetype+metric+originClass) the merged entry silently kept only bucket.primary.confidence. If siblings had been confidence-downgraded by Q10 (estimated wastedMs) while primary hadn't, the reported confidence was a lie. After: confidence demoted to WORST of all siblings (high < medium < low via confidenceRank dict). Same defensive treatment for applicability: if any sibling is third-party-cannot-apply, the grouped entry inherits that. Real-world scenario from Oracle: lcp-image-fetchpriority-high archetype is produced by both largest-contentful-paint-image AND preload-lcp-image opps; either could have estimation flags. ## Important fixed ### I1: MCP server doc strings update - apps/mcp-server/src/server.ts:453 (propose_patch description) - apps/mcp-server/src/server.ts:532 (get_fix_plan description) Both now explicitly say: "Same-archetype URLs are collapsed into a single grouped entry; when targets[] is present, expectedImpactMs is the SUM across all targets (per-URL impact lives in targets[i].expectedImpactMs) and confidence is the WORST among siblings." LLM agents reading these tool descriptions now correctly interpret the grouped data shape. ## Nits fixed ### N1: --plugins ",,," produces silent empty set - apps/cli/src/commands/run.ts (comma branch) Now warns: "--plugins resolved to empty set from ',,,'; defaulting to 'all'" then returns all plugins. Prevents silent zero-plugin runs. ### N2: --plugins "none,cwv" silently drops 'none' - apps/cli/src/commands/run.ts (comma branch) Now warns: "--plugins contains 'none' alongside other tokens ['cwv']; 'none' ignored." Continues to resolve cwv. Documents the precedence. ## Verification (Evidence Receipts) layer: validate:quick pnpm -r typecheck → 0 errors across 30+ packages pnpm -r lint (touched packages) → clean pnpm --filter core test → 94/94 PASS pnpm --filter viewer test → 98/98 PASS pnpm --filter reporter-markdown test → all pass layer: real-world CLI ohmyperf run https://tradeit.gg --plugins "cwv,axe,third-parties": fixPlan: 1 grouped entry, 18 targets, expectedImpactMs=1840 confidence: medium (BLOCKER 2 worst-of-siblings working) audits[].status: 'fail' + 'pass' (BLOCKER 1 fix verified end-to-end) ohmyperf run --plugins ",,,": WARN --plugins resolved to empty set from ',,,'; defaulting to 'all' (NIT 1 fix verified) layer: test:cross-cutting-allowlists → 3/3 PASS ## Counts 6 files changed, 26 insertions(+), 7 deletions(-) All 5 review findings resolved.
3-Angle Review (Oracle, normal-lane) — addressed in
|
| ID | Sev | Fix |
|---|---|---|
| B1 | blocker | Added status: "fail"/"pass" to 4 test fixture audit literals (viewer/render.test.ts:75, reporter-markdown/render.test.ts:46+103, llm-signals.test.ts:293) — em initially missed these because vitest transpiles on the fly; would have failed in CI |
| B2 | blocker | collapseSameArchetype now demotes confidence to WORST of siblings (high<medium<low), defensive same for applicability (any third-party → grouped third-party). Prevents silent data loss when 2 opps (e.g. largest-contentful-paint-image + preload-lcp-image) merge into one archetype entry |
| I1 | important | Updated MCP server propose_patch + get_fix_plan tool descriptions to explicitly document grouped semantics: "when targets[] is present, expectedImpactMs is the SUM... confidence is the WORST" |
| N1 | nit | --plugins ",,," now warns --plugins resolved to empty set from ',,,'; defaulting to 'all' instead of silently returning zero plugins |
| N2 | nit | --plugins "none,cwv" now warns 'none' ignored documenting the precedence |
What Oracle confirmed is NOT an issue
- Schema 1.0.0 freeze: No external in-repo consumer constructs AuditResult outside fixed fixtures
- 1000+ entries overflow:
1e6ms ≪ MAX_SAFE_INTEGER - Mixed originClass: separated correctly by
archetype|metric|originClasskey - third-parties.ts
status: "pass" as const: narrows to AuditStatus correctly - CLI comma parse for
"cwv,","cwv,cwv", whitespace: handled correctly
Verification
pnpm -r typecheck → 0 errors across 30+ packages
pnpm --filter core test → 94/94 PASS
pnpm --filter viewer test → 98/98 PASS
real-world CLI: tradeit.gg → fixPlan 1 grouped entry, 18 targets, confidence='medium' (BLOCKER 2 worst-of-siblings working)
real-world CLI: --plugins ',,,' → warn fired (NIT 1 working)
test:cross-cutting-allowlists → 3/3 PASS
Em proceeds to merge.
Summary
4 small additive fixes to engine/CLI/report output quality, batched per HARNESS pragmatism. Identified during session 2026-05-22 weakness analysis (paths #1 + #4 verification of PR #13 deploy).
Fixes
apps/cli/src/commands/run.ts--plugins cwv,axe,third-partiesnow parses correctly (was warning + defaulting to all). Individual plugin names (cwv,axe,third-parties,example) accepted standalone.packages/core/src/types.ts+axe.ts+third-parties.tsAuditResult.status: "pass" | "fail" | "na"field — LLM/dashboard consumers reading.statusgot undefined; now correctly populated.packages/core/src/types.ts+llm-signals/fix-plan.tstargets[]preserving all URLs + per-URL impact. 18→1 entries for tradeit.gg.apps/extension-chrome/scripts/setup-dev-extension.mjsDeferred to issues
Filed during this PR's planning:
test:e2e:extensioninto CIreport.metatrustScoresurfaced in UIartifactsfield wiring (scope-reduced out — structural, not 5-line)Schema compat
AuditStatus/statusfield is new required field onAuditResult. Strict reading: breaks schema 1.0.0. Pragmatic reading:AuditResultis constructed only by plugins, never by external consumers. All 3 in-tree plugins (axe,third-parties, custom-metric-example) + test fixture updated. External plugins compiled against old types will fail TS — desired so they pick up the fix. Em accept this trade-off because the alternative (optional field) leaves the null-status bug unfixed for half the codebase.targetsfield is OPTIONAL onFixPlanEntry. Single-URL entries omit it. Schema 1.0.0 backward compat preserved.Evidence Receipts
Risk
LOW. All fixes additive or strictly-better-than-before. 94 existing tests + 2 updated tests assert new grouped contract.
Multi-Layer Pre-Flight (per HARNESS Forbidden #20)
Not a multi-layer change — no chrome-ext-spa-allowlist, mv3-sw-port-lifecycle, or extension-e2e-test-infra globs touched.
3-Angle Review
Per HARNESS Forbidden #21, em launches 3-Angle review (oracle correctness + 2 explores: edge-cases + breaking-changes) as next step. Em awaits results before merging.
🤖 Generated by Sisyphus during session 2026-05-22 weakness-improve cycle.