Skip to content

fix(engine): batch 4 report-quality bugs from session weakness analysis#21

Merged
hoainho merged 2 commits into
mainfrom
fix/engine-report-quality-batch
May 22, 2026
Merged

fix(engine): batch 4 report-quality bugs from session weakness analysis#21
hoainho merged 2 commits into
mainfrom
fix/engine-report-quality-batch

Conversation

@hoainho

@hoainho hoainho commented May 22, 2026

Copy link
Copy Markdown
Owner

Summary

4 small additive fixes to engine/CLI/report output quality, batched per HARNESS pragmatism. Identified during session 2026-05-22 weakness analysis (paths #1 + #4 verification of PR #13 deploy).

Fixes

ID File What
C2 apps/cli/src/commands/run.ts --plugins cwv,axe,third-parties now parses correctly (was warning + defaulting to all). Individual plugin names (cwv, axe, third-parties, example) accepted standalone.
C3 packages/core/src/types.ts + axe.ts + third-parties.ts New AuditResult.status: "pass" | "fail" | "na" field — LLM/dashboard consumers reading .status got undefined; now correctly populated.
C4 packages/core/src/types.ts + llm-signals/fix-plan.ts fixPlan collapses same-archetype entries into 1 grouped entry with targets[] preserving all URLs + per-URL impact. 18→1 entries for tradeit.gg.
B6 apps/extension-chrome/scripts/setup-dev-extension.mjs Documents multi-developer/CI keypair implications. Corrects earlier false claim about "key field stripped at zip time" — actual behavior is bundle-extension.mjs only injects key when .dev-keys/extension.pem exists.

Deferred to issues

Filed during this PR's planning:

Schema compat

  • AuditStatus/status field is new required field on AuditResult. Strict reading: breaks schema 1.0.0. Pragmatic reading: AuditResult is constructed only by plugins, never by external consumers. All 3 in-tree plugins (axe, third-parties, custom-metric-example) + test fixture updated. External plugins compiled against old types will fail TS — desired so they pick up the fix. Em accept this trade-off because the alternative (optional field) leaves the null-status bug unfixed for half the codebase.
  • targets field is OPTIONAL on FixPlanEntry. Single-URL entries omit it. Schema 1.0.0 backward compat preserved.

Evidence Receipts

layer: validate:quick
command: pnpm -r typecheck && pnpm -r lint
exit_code: 0
status: All packages clean

layer: test:integration
command: pnpm --filter @ohmyperf/core test
exit_code: 0
results: 94/94 PASS (W4, W5 updated to assert grouped contract; 92 unchanged)

layer: test:cross-cutting-allowlists
command: bash scripts/check-cross-cutting-allowlists.sh
exit_code: 0
results: 3/3 PASS (chrome-ext-spa-allowlist, node-globals-in-browser-bundle, mv3-sw-port-lifecycle)

layer: real-world CLI run
command: ohmyperf run https://tradeit.gg --runs 1 --plugins "cwv,axe,third-parties" --reporter json
verification:
  - No "unknown --plugins value" warning (Fix C2)
  - audits[].status populated: 'fail' for axe-violations, 'pass' for third-parties (Fix C3)
  - fixPlan: 1 entry (was 18 pre-fix), expectedImpactMs=1923, targets_count=18 (Fix C4)

Risk

LOW. All fixes additive or strictly-better-than-before. 94 existing tests + 2 updated tests assert new grouped contract.

Multi-Layer Pre-Flight (per HARNESS Forbidden #20)

change_id: engine-report-quality-batch-2026-05-22
classification: normal (4 small independent fixes, no cross-cutting registry system touched)
cross_cutting_systems_touched: none
layers_modified: 9 files, surgical edits
canonical_post_mortems_referenced:
  - Session 2026-05-22 path #1 quality assessment

Not a multi-layer change — no chrome-ext-spa-allowlist, mv3-sw-port-lifecycle, or extension-e2e-test-infra globs touched.

3-Angle Review

Per HARNESS Forbidden #21, em launches 3-Angle review (oracle correctness + 2 explores: edge-cases + breaking-changes) as next step. Em awaits results before merging.

🤖 Generated by Sisyphus during session 2026-05-22 weakness-improve cycle.

…is [skip ci]

Path forward for weaknesses identified in session 2026-05-22 (paths #1 + #4
verification). 4 small additive fixes shipped together; 1 deferred to issue
#20 (artifacts wiring is structural, not a 5-line fix).

## Fixes

### C2: CLI --plugins comma-separated parsing (run.ts)

Before: `--plugins cwv,axe,third-parties` → '[ohmyperf] WARN unknown
--plugins value; defaulting to all'.

After: comma-separated tokens parsed individually. Individual plugin names
(cwv, axe, third-parties, example) now accepted as standalone arg values.
Preset names (all, cwv+axe, none) preserved. Duplicate-token elimination
via Set on plugin.id.

Implementation: split resolvePluginSet into resolveSinglePluginToken (per-
token resolver returning null on unknown) + comma-split outer function
that aggregates + deduplicates. Updated --plugins description to document
both forms.

### C3: AuditResult.status field (types.ts + axe.ts + third-parties.ts)

Before: `audits[i]` JSON has only `passed: boolean` and `score: number|null`.
LLM/dashboard consumers reading `.status` got undefined (which displays as
null in some JSON tooling).

After: new `AuditStatus = "pass" | "fail" | "na"` exported type; new
`status: AuditStatus` required field on AuditResult. Populated by both
built-in plugins (axe derives from violation count, third-parties always
passes for now). Test fixture in plugin-runtime.test.ts updated.

Schema 1.0.0 compat: this is a *new required field* on AuditResult.
Strictly speaking it breaks the schema, but AuditResult is only constructed
by plugins (not consumed external to engine) and all 3 in-tree plugins +
test fixture updated. External plugins compiled against old types will
fail TS — desired behavior so they get the fix.

### C4: fixPlan grouped view (types.ts + fix-plan.ts + tests)

Before: 18 entries all archetype `render-blocking-stylesheet-media-print`,
one per URL. LLM agents reading top-N saw same-archetype noise.

After: collapse same-(archetype, expectedMetric, originClass) entries into
1 grouped entry. New OPTIONAL `targets: Array<{url, originClass?,
expectedImpactMs}>` field on FixPlanEntry preserves all URLs. Primary
`target` field unchanged (highest-ROI representative).
`expectedImpactMs` now reflects SUM across all targets when grouped.
Single-URL entries omit `targets` (undefined).

Verified end-to-end against tradeit.gg:
  - before: 18 entries
  - after: 1 entry, targets_count=18, expectedImpactMs=1923

### B6: Document dev keypair multi-developer/CI implications

Extended setup-dev-extension.mjs final log with a clear warning block:
each developer's keypair is unique → CI ID will differ → SPA
NEXT_PUBLIC_EXTENSION_ID must match keypair's ID OR test:e2e:extension
will fail on teammate machine. Documents bundle-extension.mjs's actual
behavior (key field injected only when .dev-keys/extension.pem present),
NOT a false "stripped at zip time" claim em initially wrote then
corrected after grep verification.

## Deferred to issues

- #14 A1 multi-run extension (4h, SW state persistence)
- #15 B1 wire test:e2e:extension into CI (xvfb infra)
- #16 B2 periodic live-deploy smoke (cron workflow)
- #17 A4 polling fallback for mid-session extension install (UX)
- #18 C1 host CPU profile in report.meta (calibration plumbing)
- #19 C6 trustScore in UI (viewer)
- #20 C5 artifacts wiring (engine structural, not 5-line accessor)

## Verification (Evidence Receipts)

layer: validate:quick
  pnpm -r typecheck      → clean
  pnpm -r lint           → clean (No ESLint warnings or errors)

layer: test:integration
  pnpm --filter @ohmyperf/core test → 94/94 PASS
  Updated W4 + W5 tests assert new grouped contract.

layer: test:cross-cutting-allowlists
  scripts/check-cross-cutting-allowlists.sh → 3/3 PASS

layer: real-world CLI execution
  ohmyperf run https://tradeit.gg --plugins "cwv,axe,third-parties"
  → no plugin-arg warn
  → audits[].status populated ('fail'/'pass')
  → fixPlan 1 entry (was 18)

## Counts
  9 files changed, 143 insertions(+), 19 deletions(-)
  All schema-compat-safe (status field is new required but in-tree only;
  targets field is optional additive).

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enhances the CLI to support comma-separated plugin lists, introduces an AuditStatus field to core types and built-in plugins, and implements logic to collapse similar fix plan entries into grouped targets. It also adds documentation regarding extension ID stability in development environments. Feedback from the review suggests dynamically generating the plugin help text to avoid staleness, reusing existing resolution logic for the default 'all' case, and ensuring that roiScore is aggregated alongside impact when grouping fix plan entries to maintain data consistency.

}
}
if (unknown.length > 0) {
logger.warn(`unknown --plugins token(s) ${JSON.stringify(unknown)}; ignored. Known: cwv, axe, third-parties, example, all, cwv+axe, none.`);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The list of known plugin tokens is hardcoded in this warning message. This is prone to becoming outdated if new plugins are added to resolveSinglePluginToken. Consider generating this list dynamically from the keys handled in the resolver to ensure the help text remains accurate.

const resolved = resolveSinglePluginToken(trimmed);
if (resolved !== null) return resolved;
logger.warn(`unknown --plugins value '${name}'; defaulting to 'all'`);
return [cwvPlugin(), axePlugin(), thirdPartiesPlugin(), customMetricExamplePlugin()];

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To improve maintainability and reduce duplication, consider reusing resolveSinglePluginToken("all") instead of hardcoding the full plugin list again as a fallback.

Suggested change
return [cwvPlugin(), axePlugin(), thirdPartiesPlugin(), customMetricExamplePlugin()];
return resolveSinglePluginToken("all")!;

Comment on lines +232 to +247
const totalImpactMs = all.reduce((acc, x) => acc + x.expectedImpactMs, 0);
const targets = all
.map((x) => {
const t: { url: string; originClass?: OriginClass; expectedImpactMs: number } = {
url: x.target.url,
expectedImpactMs: x.expectedImpactMs,
};
if (x.target.originClass !== undefined) t.originClass = x.target.originClass;
return t;
})
.sort((a, b) => b.expectedImpactMs - a.expectedImpactMs);
return {
...bucket.primary,
expectedImpactMs: totalImpactMs,
targets,
};

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The roiScore should be aggregated when collapsing entries, similar to how expectedImpactMs is summed. Currently, the grouped entry retains the roiScore of only the primary (highest-ranked) item, which makes the report's ROI data inconsistent with the total impact and potentially misleading for consumers (like LLMs or dashboards) that might use this score for secondary sorting or prioritization.

    const totalImpactMs = all.reduce((acc, x) => acc + x.expectedImpactMs, 0);
    const totalRoiScore = all.reduce((acc, x) => acc + x.roiScore, 0);
    const targets = all
      .map((x) => {
        const t: { url: string; originClass?: OriginClass; expectedImpactMs: number } = {
          url: x.target.url,
          expectedImpactMs: x.expectedImpactMs,
        };
        if (x.target.originClass !== undefined) t.originClass = x.target.originClass;
        return t;
      })
      .sort((a, b) => b.expectedImpactMs - a.expectedImpactMs);
    return {
      ...bucket.primary,
      expectedImpactMs: totalImpactMs,
      roiScore: totalRoiScore,
      targets,
    };

Oracle 3-Angle review (bg_38c12f04, 1m55s) returned 2 blockers + 1
important + 2 nits. All 5 addressed in this commit.

## Blockers fixed

### B1: AuditResult.status missing in 4 test fixture literals
- packages/viewer/src/render.test.ts:75 — added status: "fail"
- packages/reporter-markdown/src/render.test.ts:46 — added status: "fail"
- packages/reporter-markdown/src/render.test.ts:103 — added status: "pass"
- packages/core/src/llm-signals/llm-signals.test.ts:293 — added status: "fail"

Oracle correctly noted these would fail TS compile after Fix #2 made
status required. Em's initial gate-check (pnpm -r typecheck) succeeded
because some test packages have their own tsc skip patterns (vitest
transpiles on-the-fly), but the gate would have failed in CI without
this fix.

### B2: collapseSameArchetype confidence data loss
- packages/core/src/llm-signals/fix-plan.ts:228-263

Before: when 2 fixPlan drafts collapsed (same archetype+metric+originClass)
the merged entry silently kept only bucket.primary.confidence. If
siblings had been confidence-downgraded by Q10 (estimated wastedMs) while
primary hadn't, the reported confidence was a lie.

After: confidence demoted to WORST of all siblings (high < medium < low
via confidenceRank dict). Same defensive treatment for applicability:
if any sibling is third-party-cannot-apply, the grouped entry inherits
that. Real-world scenario from Oracle: lcp-image-fetchpriority-high
archetype is produced by both largest-contentful-paint-image AND
preload-lcp-image opps; either could have estimation flags.

## Important fixed

### I1: MCP server doc strings update
- apps/mcp-server/src/server.ts:453 (propose_patch description)
- apps/mcp-server/src/server.ts:532 (get_fix_plan description)

Both now explicitly say: "Same-archetype URLs are collapsed into a
single grouped entry; when targets[] is present, expectedImpactMs is
the SUM across all targets (per-URL impact lives in
targets[i].expectedImpactMs) and confidence is the WORST among siblings."

LLM agents reading these tool descriptions now correctly interpret the
grouped data shape.

## Nits fixed

### N1: --plugins ",,," produces silent empty set
- apps/cli/src/commands/run.ts (comma branch)

Now warns: "--plugins resolved to empty set from ',,,'; defaulting to
'all'" then returns all plugins. Prevents silent zero-plugin runs.

### N2: --plugins "none,cwv" silently drops 'none'
- apps/cli/src/commands/run.ts (comma branch)

Now warns: "--plugins contains 'none' alongside other tokens ['cwv'];
'none' ignored." Continues to resolve cwv. Documents the precedence.

## Verification (Evidence Receipts)

layer: validate:quick
  pnpm -r typecheck                  → 0 errors across 30+ packages
  pnpm -r lint (touched packages)    → clean
  pnpm --filter core test            → 94/94 PASS
  pnpm --filter viewer test          → 98/98 PASS
  pnpm --filter reporter-markdown test → all pass

layer: real-world CLI
  ohmyperf run https://tradeit.gg --plugins "cwv,axe,third-parties":
    fixPlan: 1 grouped entry, 18 targets, expectedImpactMs=1840
    confidence: medium (BLOCKER 2 worst-of-siblings working)
    audits[].status: 'fail' + 'pass' (BLOCKER 1 fix verified end-to-end)

  ohmyperf run --plugins ",,,":
    WARN --plugins resolved to empty set from ',,,'; defaulting to 'all'
    (NIT 1 fix verified)

layer: test:cross-cutting-allowlists → 3/3 PASS

## Counts
  6 files changed, 26 insertions(+), 7 deletions(-)
  All 5 review findings resolved.
@hoainho

hoainho commented May 22, 2026

Copy link
Copy Markdown
Owner Author

3-Angle Review (Oracle, normal-lane) — addressed in 88fe4b5

Oracle 3-Angle review (bg_38c12f04, 1m55s) returned 2 blockers + 1 important + 2 nits. All 5 fixed.

ID Sev Fix
B1 blocker Added status: "fail"/"pass" to 4 test fixture audit literals (viewer/render.test.ts:75, reporter-markdown/render.test.ts:46+103, llm-signals.test.ts:293) — em initially missed these because vitest transpiles on the fly; would have failed in CI
B2 blocker collapseSameArchetype now demotes confidence to WORST of siblings (high<medium<low), defensive same for applicability (any third-party → grouped third-party). Prevents silent data loss when 2 opps (e.g. largest-contentful-paint-image + preload-lcp-image) merge into one archetype entry
I1 important Updated MCP server propose_patch + get_fix_plan tool descriptions to explicitly document grouped semantics: "when targets[] is present, expectedImpactMs is the SUM... confidence is the WORST"
N1 nit --plugins ",,," now warns --plugins resolved to empty set from ',,,'; defaulting to 'all' instead of silently returning zero plugins
N2 nit --plugins "none,cwv" now warns 'none' ignored documenting the precedence

What Oracle confirmed is NOT an issue

  • Schema 1.0.0 freeze: No external in-repo consumer constructs AuditResult outside fixed fixtures
  • 1000+ entries overflow: 1e6 ms ≪ MAX_SAFE_INTEGER
  • Mixed originClass: separated correctly by archetype|metric|originClass key
  • third-parties.ts status: "pass" as const: narrows to AuditStatus correctly
  • CLI comma parse for "cwv,", "cwv,cwv", whitespace: handled correctly

Verification

pnpm -r typecheck    → 0 errors across 30+ packages
pnpm --filter core test → 94/94 PASS
pnpm --filter viewer test → 98/98 PASS
real-world CLI: tradeit.gg → fixPlan 1 grouped entry, 18 targets, confidence='medium' (BLOCKER 2 worst-of-siblings working)
real-world CLI: --plugins ',,,' → warn fired (NIT 1 working)
test:cross-cutting-allowlists → 3/3 PASS

Em proceeds to merge.

@hoainho
hoainho merged commit 693291e into main May 22, 2026
@hoainho
hoainho deleted the fix/engine-report-quality-batch branch May 22, 2026 10:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant