Skip to content

Add reusable privacy-reviewed model evaluations - #170

Draft
srkirkland wants to merge 1 commit into
mainfrom
srk/model-evaluation
Draft

srkirkland wants to merge 1 commit into
mainfrom
srk/model-evaluation

Conversation

@srkirkland

Copy link
Copy Markdown
Member

PolicyWonk model changes need a repeatable comparison against real questions. Add a small offline runner that samples recent saved production questions, requires local privacy review, freezes the app's actual retrieval context, and compares model/reasoning settings without writing to production.

The runner provides resumable generation and blinded judging, a cumulative cost cap, US-endpoint enforcement, citation checks, latency/token/cost metrics, reversed-order judge checks, and a self-contained HTML report. Production data and private reports stay outside Git; the PR includes only synthetic cases and aggregate results.

The initial 100-production-question plus 10-edge-case comparison favors keeping gpt-5.2 under the current prompt. Terra is roughly twice as fast but loses more grounded-answer comparisons; uncached answer costs are nearly equal. The aggregate results also document missing Knowledge Base content and limitations of the automated judge. No application model setting or deployment changes are included.

Validation:

  • Eight Node tests and three Python privacy-filter tests pass.
  • 330 answers and 220 comparisons completed, with 24 reversed-order rechecks.
  • Source-based assistant review covered all 35 severe-flagged cases plus 10 seeded random cases (42 unique); this is not policy-owner validation.
  • Full cached replay reused all answers/comparisons without adding ledger entries.
  • Report rendering/filter checks and production database read-only-session confirmation passed.

@coderabbitai

coderabbitai Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant