The RSI Loop — from a verification demo to a Goodhart testbed

Status (September 2026): being rebuilt as an experiment. The original RSI Loop — a two-stage acceptance gate around a hand-written posture-risk detector — was audited in September 2026 and found to contain a gate but no optimiser, no hidden evaluation and no process boundary between the candidate and its verifier (see Known limitations). The repository is now being converted, milestone by milestone, into the preregistered follow-up experiment designed in docs/rsi-loop-2-research-design.md: an LLM optimiser rewrites the detector under four verification architectures while a seeded simulator holds the hidden ground truth.

python3 -m pytest -q                      # 91 tests: simulator, datasets, sandbox, evaluators, gates
python3 -m env.datasets --seed 1 --gen0   # generation-0 scores on seed 1

The original demo is unchanged and still runs — from the legacy/ directory, whose README is the historical write-up:

python3 -m pip install -r requirements.txt
cd legacy && python3 demo.py

Everything below this line describes that original demo and its audit. It is kept here, unedited apart from paths, because the audit is the reason the experiment exists.

Built for the pharmatools.ai portfolio.

The recording is the output of demo.py: cycle 1 (the original, flawed v1 detector → 90 % accuracy), a narrative panel explaining the fix, cycle 2 (the corrected v2 → 100 %), and the Stage 2 audit. The "self-improvement" panel is a description of a change I made, not the output of an optimiser. A higher-fidelity recording is committed as demo.cast — play it with asciinema play demo.cast.

python3 -m pip install -r requirements.txt
cd legacy
python3 demo.py            # narrated v1 → v2 walkthrough + final audit (start here)
python3 test_engine.py     # unnarrated run, machine-readable output
python3 auditor.py         # Stage-2 audit only

Self-improving systems have a well-known failure mode: specification gaming, also called reward hacking. If the only objective is "pass the test suite", a sufficiently flexible optimiser will find parameters that satisfy the metric while destroying real-world meaning. A wrist-deviation threshold of 999° passes any benchmark in which nothing trips it — and is useless on a real user.

The RSI Loop sketches one answer: a two-stage gate in which passing the benchmark is necessary but not sufficient, and the second stage is grounded in domain knowledge that lives outside the benchmark. It is a sketch of the gate, not a test of it under optimisation pressure — see Known limitations.

test_engine.py runs detector.assess against every labelled scenario in benchmarks.json and computes precision, recall, F1 and accuracy. Anything under the 90 % accuracy gate exits with code 1 (NOT_ACCURATE) and the auditor is not invoked.

Once Stage 1 passes, auditor.run_audit() reads each tunable constant out of detector.py (getattr against the live module) and compares it with the Clinical Gold Standard. Each threshold receives one of three statuses:

The final status is COMPLETE only when Stage 1 and Stage 2 are clean. (Note that WARNING and HARD FAILURE both reject; the band changes the label, not the decision.)

Hard-coded in auditor.py:

Both ranges follow ergonomic literature on craniovertebral angle and ulnar/radial wrist deviation.

The gate was designed against a hand-enumerated list of threshold attacks. Against that list it behaves as intended:

To see this interactively, set WRIST_DEVIATION_ANGLE_THRESHOLD_DEG = 67.0 in detector.py and re-run python3 test_engine.py: Stage 1 still passes, Stage 2 rejects.

Two things the table hides. First, the 999° and 0.0 attacks are caught by Stage 1, not Stage 2; the auditor's unique contribution is rejecting values in the 60°–70° band that Stage 1 assigns no advantage to anyway. Second, on this benchmark the two stages can never disagree: the Safe cases have wrist metrics ≤ 36.6° and forward-head ≤ 15.1°, the High Strain cases ≥ 67.5° and ≥ 31.6°, so any threshold inside the clinical ranges scores 100 %. There is no optimisation pressure toward 67°, and nothing here shows the gate holding under pressure.

An audit of this version (September 2026) found the following, each reproduced against an untouched copy of the code:

1. The verifier shares a process with the candidate. auditor.py reads thresholds from the live detector module, and detector.assess() runs before run_audit(). Three lines inside assess() that import auditor and rewrite CLINICAL_GOLD_STANDARD let a 67° threshold pass with COMPLETE. The gate can be rewritten by the thing it gates.
2. The evaluator cannot distinguish a lookup table from a classifier. A detector.py that opens benchmarks.json and returns the stored label on an exact landmark match scores 100 % and passes the audit with thresholds untouched.
3. The 90 % gate tolerates a known misclassification. A forward-head threshold of 15.0° (inside the clinical range) misclassifies scenario S08 (15.07°) and is still certified COMPLETE.
4. The shipped v2 detector is sensitive to camera roll. Forward-head angle is measured against the image vertical rather than the torso axis. Rotating the benchmark landmarks about the shoulder to simulate a tilted webcam gives 90 % at +5°, 80 % at +15° and 60 % at +20°. The benchmark, which has no roll, cannot see this.
5. There is no optimiser and no loop. No code path proposes, searches or selects anything. The v1 → v2 change was made by hand with full visibility of every benchmark case and every auditor interval.

None of these is fixed in this repository; fixing them properly is the follow-up experiment. The intended lesson of this version is narrower than the original README claimed: a domain-grounded second stage is a sensible shape for an acceptance gate. Whether it holds against an actual optimiser is an open question this code cannot answer.

All of these now live in legacy/.

v1 missed the radial-deviation scenario S06 because its wrist check inspected only one side of the offset. I replaced both crude offsets with trigonometric angles (direction-agnostic), and v2 reached COMPLETE. demo.py narrates this change; it does not generate it.

python3 -m pip install -r requirements.txt
cd legacy && python3 -c "from detector import assess_from_webcam; print(assess_from_webcam())"

The webcam pipeline is gated behind a lazy import, so the rest of the project runs without mediapipe or opencv-python installed. Note limitation 4 above: the current geometry assumes an un-rolled camera.

• Add benchmarks by editing benchmarks.json. Each scenario needs the six landmarks the detector uses (ear, shoulder, elbow, wrist, index_mcp, pinky_mcp) plus a Safe / High Strain label.
• Add ergonomic rules by adding a new pure-function angle helper in detector.py plus its threshold constant; then add a matching entry to CLINICAL_GOLD_STANDARD in auditor.py so the new constant is regulated from day one.
• Tighten the gold standard if you have stronger clinical evidence — narrow the expected range and the gate forces more conservative thresholds.
• Turn it into an experiment — see docs/rsi-loop-2-research-design.md.