EAL-Bench

A benchmark for Endogenous Authorization Laundering (EAL) in persistent agent memory

EAL-Bench tests whether persistent agent memory preserves evolving authorization boundaries and whether memory errors lead to unauthorized downstream tool actions. It accompanies the paper Agent Memory Is a Surface for Endogenous Authorization Laundering.

A memory writer compresses an organizational history into persistent memory; an executor later receives that memory and a new request, then chooses whether to act through a native tool.

Each benchmark case includes:

• an evolving, multi-session organizational history;
• a hidden authorization ledger to score the action deterministically;
• matched authorized and unauthorized requests;
• a bounded free-text or typed memory;
• domain-native tools and a deterministic oracle.

To use EAL in your research, see the usage guide for installation, examples, and the four evaluation tracks.

If you use EAL-Bench in your research, or if it's closely related to your work, please cite the accompanying paper:

@misc{cerruti2026agent,
  title         = {Agent Memory Is a Surface for Endogenous Authorization Laundering},
  author        = {Cerruti, Tommaso and Okamoto, Mika and Erol, Ansel Kaplan},
  year          = {2026},
  eprint        = {2609.01836},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CR},
  url           = {https://arxiv.org/abs/2609.01836}
}

Incremental writers receive the previous accepted memory and the new block, but not earlier raw blocks. Final memories are frozen and hashed before executor evaluation.

List the configured models with:

uv run python -m experiments.run --list-targets

Requirements:

• Python 3.10 or newer;
• uv;
• a Baseten or OpenRouter API key for live experiments.

git clone git@github.com:tommasocerruti/eal-bench.git
cd eal-bench

uv sync --extra dev --extra analysis
cp .env.example .env

Add credentials only for the providers you intend to call:

BASETEN_API_KEY=...
OPENROUTER_API_KEY=...

Offline inspection and validation do not require API credentials.

uv run python -m experiments.run --list-domains
uv run python -m experiments.run --domain procurement --list-corpus-versions
uv run python -m experiments.run --domain procurement --list-studies
uv run python -m experiments.run --validate-only --all-domains

Validate your experiment configuration:

uv run python -m experiments.run \
  --domain procurement \
  --corpus-version benchmark_v1 \
  --presentation-version naturalistic_v1 \
  --study writer \
  --writer-targets nemotron_3_ultra_baseten \
  --executor-targets gptoss_baseten \
  --writer-architecture all \
  --writer-strategy all \
  --validate-only

Live routes make paid API calls. Review the validated call plan first, then rerun it with an explicit cost ceiling:

uv run python -m experiments.run \
  --domain procurement \
  --corpus-version benchmark_v1 \
  --presentation-version naturalistic_v1 \
  --study writer \
  --writer-targets nemotron_3_ultra_baseten \
  --executor-targets gptoss_baseten \
  --writer-architecture all \
  --writer-strategy all \
  --estimated-cost-usd <reviewed-total-ceiling> \
  --tag benchmark-v1-writer

Use --study controls for faithful controls. Pressure replays require a completed writer run through --source-run results/<domain>/<run-id>.

The mitigation guide explains how to apply source-authority gating to frozen writer memories and evaluate the paired memories with a configured executor.

Every run is stored in a new immutable directory:

results/<domain>/<run-id>/

Each run records its configuration, memories, model inputs, tool calls, scores, and provider usage. See manifest.json for the files included in the run.

The shared result guide provides one layout for Procurement, Cybersecurity, and Finance. Each domain has a results/<domain>/paper/manifest.json selecting the exact runs, release, source hashes, and counts used in Appendix B.2, Tables 11–13. Reproduce all three domains without model calls or API keys:

python3 -m analysis.paper_results --domain all --output-dir /tmp/eal-paper-tables

Each domain exports the same CSV, JSON, and Markdown files, including writer/executor breakdowns and an artifact inventory. Raw JSONL files are excluded from Git across all domains; the command checks any originals present locally and reports missing files. Saved aggregates reproduce the published counts, while independent trial scoring and bootstrap reanalysis require the originals.

To add a domain, define its authorization rules, histories, memory format, requests, tools, and scoring logic:

• Domain interface
• Domain contribution guide
• Model and provider configuration

Before opening a pull request:

uv run python -m experiments.run --validate-only --all-domains
uv run ruff check .
git diff --check