scaldex measures whether a Codex instruction package saves or costs tokens without visibly degrading task quality.

Use it when you optimise AGENTS.md, AGENTS.override.md or support files and folders referenced by those instructions. scaldex gives you a task-level result that you can inspect yourself or hand to Codex for evidence-based follow-up.

scaldex runs each benchmark task in two isolated variants:

• agents: Codex runs with the instruction package you are measuring.
• control: Codex runs without that package and without your global ~/.codex config.

scaldex does not edit your instructions. It measures, reports and writes evidence.

• Python 3.11 or newer.
• Git on PATH. scaldex uses Git internally for temporary benchmark snapshots; your measured package does not need to be a Git repository.
• Codex CLI on PATH with codex exec support.
• A Codex API key for paid benchmark runs.

scaldex bench doctor, scaldex bench inspect-subject, scaldex bench inspect-suite, result replay and report summaries do not spend API money. Benchmark commands spend API money.

Download or clone the scaldex project from GitHub, then open a terminal in the downloaded scaldex folder. This is the folder that contains README.md, pyproject.toml and run_scaldex.py.

Install the scaldex command:

python3 -m pip install -e .

Check your local setup without spending API money:

scaldex bench doctor

If you do not want to install the command yet, stay in the downloaded scaldex folder and use the wrapper instead:

python3 run_scaldex.py --help

The examples below use the installed scaldex command. With the wrapper, replace scaldex with python3 run_scaldex.py.

For a Git checkout, run git pull in the scaldex source folder. For a downloaded archive, replace the old source folder with the new one.

If you installed the scaldex command, run this again in the updated folder:

python3 -m pip install -e .

Editable installs read code from the source folder, but reinstalling keeps command entry points, package metadata and dependencies aligned. If the source folder moved to a new path, reinstall from the new folder.

If you only use the wrapper, update or replace the source folder and run this no-cost check from the updated folder:

python3 run_scaldex.py --help

v2.0.0 adds versioned result schemas and mandatory benchmark provenance. It cannot replay v1 reports or include them in summaries, and there is no automatic migration.

Before the first v2 benchmark, preserve any existing v1 scaldex-run folder by moving it outside the new run path or choose a fresh folder with --run-dir. A v1 result.json in the default run folder can prevent v2 from archiving the previous run.

Keep a v1.0.1 installation or checkout if you need to replay old reports. Do not add v2 schema or provenance fields manually. Generate new v2 evidence before making task-level or global efficiency claims, and do not combine v1 and v2 reports in one summary.

Create a subject/ folder for the instruction package you want to measure:

subject/
  AGENTS.md
  xyz/

subject/ is the package under test. Keep it separate from the scaldex project folder. It must contain AGENTS.md or AGENTS.override.md.

Support files and folders are optional. You can use names such as .codex/, xyz/ or anything else your instruction entry file references.

Inspect the package before spending API money:

scaldex bench inspect-subject --subject-dir subject

By default, scaldex measures the whole subject/ package. Use --subject-mode agents-md only when you intentionally want to measure the entry file without its support material.

Run one smoke benchmark first:

scaldex --model gpt-5.5 --subject-dir subject --task-id login_test_failure --repeats 1

This starts two paid Codex runs: one control run and one agents run.

To measure your own workflow, use a strict task manifest against the built-in workspace or a portable suite with a synthetic workspace. Inspect it before any paid run:

scaldex bench inspect-suite --tasks-file tasks.json
scaldex bench inspect-suite --suite-dir suite

Then pass the same --tasks-file or --suite-dir flag to the benchmark command. Custom evidence remains task-specific and never contributes to the built-in global efficiency claim. See the custom task guide for the manifest, workspace and safety contracts.

After the run, read What this means and What to do now in the terminal output. scaldex-run/RESULT.md contains the same human-readable interpretation and next-step guidance.

The run folder contains:

• scaldex-run/RESULT.md: human-readable report.
• scaldex-run/CODEX_HANDOFF.md: Codex-facing follow-up brief.
• scaldex-run/result.json: machine-readable report.

You have three safe paths:

• read RESULT.md and improve the instruction package manually
• give CODEX_HANDOFF.md and the measured contents of subject/ to Codex with a clear optimisation request
• stop changing the package and collect stronger decision-grade evidence when scaldex tells you to

CODEX_HANDOFF.md is a brief, not a self-contained request. If you want Codex-assisted follow-up, give Codex the handoff, the measured contents of subject/ and an explicit task. The Codex instruction workflow guide includes a copy-paste prompt.

Do not improve the measured package from a smoke win alone. Smoke evidence only tells you whether a decision-grade run is worth the cost.

• Codex instruction workflow: how scaldex, Codex and the end user work together.
• Measurement model: metrics, quality gates, smoke evidence, decision-grade evidence and global claims.
• Built-in task guide: task selection and the eight v1.0 benchmark workflows.
• Custom task guide: strict manifests, portable suites, no-cost preflight, provenance and safety boundaries.
• Contributing guide: development checks and contribution boundaries.

Read CONTRIBUTING.md before changing the package. Release, tag, push, and publication actions always require explicit approval.

This project uses AGPL-3.0-only. You may use, change and distribute it under the licence terms.

The project name, branding and public presentation are not granted as trade marks or endorsements by the software licence. See the project notice for trade mark, service, API-key and benchmark-evidence boundaries.