Jump to text
On this page
Copy and source
Published source confirmed at last check

Source changed 2026-11-18 21:11:39 UTC · snapshot created 2026-11-18 21:12:35 UTC · last check 2026-11-18 22:15:20 UTC

C.41.QC:6 - Bias-Annotation

People tend to invent tests that confirm their implementation. Generated criteria and generated answers can share the same blind spot. Recover a failure from the receiving work, seek an unlike case where it matters, and distinguish the data used to tune a checker from the evidence used to judge its broader use.

A benchmark’s accessible cases can underrepresent the domain or participants who bear false failures. Make that limit part of the reliance decision.