Skip to content

docs(demo): real AI-agent usage scenarios; deprecate stale sales kit - #4

Merged
xerj-team merged 1 commit into
xerj-org:mainfrom
xerj-team:docs/agent-scenarios
Jul 23, 2026
Merged

docs(demo): real AI-agent usage scenarios; deprecate stale sales kit#4
xerj-team merged 1 commit into
xerj-org:mainfrom
xerj-team:docs/agent-scenarios

Conversation

@xerj-team

Copy link
Copy Markdown
Collaborator

The existing demo/ kit is April-2026 sales-engineer material — DEMO_RUNBOOK.md and README.md pitch XERJ to enterprise buyers and cite v1.0.0-rc.1, conformance 1305/1329 (now 1360/1363), a since-fixed RC1 ingest race, and "hybrid not yet producing fused rankings" (run_hybrid does RRF today). It is not a guide to what this product is FOR: an AI coding agent's tool.

This replaces the framing with measured, reproducible agent scenarios and flags the old material rather than deleting it.

Added

  • SCENARIOS.md — six real an AI coding agent tasks with measured before/after and copy-paste reproduction, every number from a run in this repo:

    1. Orient in unfamiliar data — autoindex map, ~890 tokens flat vs growing
    2. Where/why in a large repo — 5.3x fewer tokens at 170k LOC; recall 6/7 vs grep 3/7 on prose (grep is lexical, misses reworded answers)
    3. Analytics over logs — 6/6 vs grep 4/6 correctness (the gate caught the baseline skipping a .gz and mis-parsing a decimal-comma CSV)
    4. Incident drill-down — indexed lookup 0.007s vs grep 7.3s at 1.2 GB (1043x), and flat as data grows
    5. Relational without JOIN — the denormalize pattern: 36 bytes vs 15,914 (442x) for an identical answer
    6. Cross-session memory — 328-byte semantic recall, a capability grep lacks Each scenario states plainly when XERJ does NOT help.
  • gate_retrieval.py + make_corpus_prose.py — the retrieval regime, the half the analytics gate could not show. On a 92%-prose corpus XERJ scores 6/7 vs grep 3/7; token parity at 53 KB, widening to 5.3x fewer at 170k LOC. Same anti-cheating discipline as gate.py.

  • RESULTS_analytics.txt / RESULTS_retrieval.txt — committed reference runs so a reader sees both regimes without running anything.

The through-line

One law predicts every result and is printed next to every ratio: retrieval savings scale with the prose fraction; analytics savings scale with record count but analytics answers are small either way, so there the win is correctness and latency, not tokens. A single headline number across both regimes is dishonest, and neither gate will produce one.

Deprecated (kept, bannered)

DEMO_RUNBOOK.md and README.md now carry a stale-material banner pointing to SCENARIOS.md and docs/TOKEN_USAGE.md. Refresh their numbers before any external use.

Not yet audited: demo/playbooks/ (ES-vs-XERJ perf battle plans) and demo/usecases/ — a follow-up pass. The autoindex usecase evaluation there remains the most current of the old material.

The existing demo/ kit is April-2026 sales-engineer material — DEMO_RUNBOOK.md
and README.md pitch XERJ to enterprise buyers and cite v1.0.0-rc.1, conformance
1305/1329 (now 1360/1363), a since-fixed RC1 ingest race, and "hybrid not yet
producing fused rankings" (run_hybrid does RRF today). It is not a guide to
what this product is FOR: an AI coding agent's tool.

This replaces the framing with measured, reproducible agent scenarios and
flags the old material rather than deleting it.

## Added

* `SCENARIOS.md` — six real an AI coding agent tasks with measured before/after and
  copy-paste reproduction, every number from a run in this repo:
    1. Orient in unfamiliar data — autoindex map, ~890 tokens flat vs growing
    2. Where/why in a large repo — 5.3x fewer tokens at 170k LOC; recall 6/7
       vs grep 3/7 on prose (grep is lexical, misses reworded answers)
    3. Analytics over logs — 6/6 vs grep 4/6 correctness (the gate caught the
       baseline skipping a .gz and mis-parsing a decimal-comma CSV)
    4. Incident drill-down — indexed lookup 0.007s vs grep 7.3s at 1.2 GB
       (1043x), and flat as data grows
    5. Relational without JOIN — the denormalize pattern: 36 bytes vs 15,914
       (442x) for an identical answer
    6. Cross-session memory — 328-byte semantic recall, a capability grep lacks
  Each scenario states plainly when XERJ does NOT help.

* `gate_retrieval.py` + `make_corpus_prose.py` — the retrieval regime, the
  half the analytics gate could not show. On a 92%-prose corpus XERJ scores
  6/7 vs grep 3/7; token parity at 53 KB, widening to 5.3x fewer at 170k LOC.
  Same anti-cheating discipline as gate.py.

* `RESULTS_analytics.txt` / `RESULTS_retrieval.txt` — committed reference runs
  so a reader sees both regimes without running anything.

## The through-line

One law predicts every result and is printed next to every ratio: retrieval
savings scale with the prose fraction; analytics savings scale with record
count but analytics answers are small either way, so there the win is
correctness and latency, not tokens. A single headline number across both
regimes is dishonest, and neither gate will produce one.

## Deprecated (kept, bannered)

DEMO_RUNBOOK.md and README.md now carry a stale-material banner pointing to
SCENARIOS.md and docs/TOKEN_USAGE.md. Refresh their numbers before any external
use.

Not yet audited: demo/playbooks/ (ES-vs-XERJ perf battle plans) and
demo/usecases/ — a follow-up pass. The autoindex usecase evaluation there
remains the most current of the old material.
@xerj-team
xerj-team merged commit 88f21b8 into xerj-org:main Jul 23, 2026
3 of 4 checks passed
xerj-team pushed a commit that referenced this pull request Jul 27, 2026
…ates + trigger traces

Recovery from journal.jsonl after the usage-limit interruption: 177/177 review
batches completed -> ALL 883 files were analyzed. Only the VERIFY phase dropped 8
of 44 candidates (not files). Hand-verified the 8 by reading real code:
- cover.php:102 REFUTED (fixed iframe + esc_url, not raw oEmbed HTML)
- feed-rss2-comments.php:107 plausible low-med (]]> CDATA breakout into comments
  feed; affects feed consumers, not the site; unconfirmed)
- custom-css.php:258 plausible med (edit_css strip wired only to content_save_pre;
  block-widget REST save may bypass; needs path trace)
- wp-mail.php (by-design Post-by-Email), wp-trackback.php (the code IS the UTF-7
  mitigation) -> FP/by-design; 3 REST candidates low/unconfirmed.

Net: lead-confirmed new finding stays user-new.php role injection (#4); added the
exact HTTP trigger trace (POST action=adduser role=administrator -> stored option
-> GET /newbloguser/{key}/ -> set_role, with the multisite+filtered-editable_roles
conditions).

Added "How XERJ makes this discoverable vs a full-context search": (1) 5.2M-token
core doesn't fit a window -> chunking splits the cross-file role flow; (2) XERJ
hands the reviewer the 3 sibling role-sinks to compare (the bug is a missing
sibling guard, invisible without the map); (3) index-once vs re-read-everything.
Neither structural graph nor blind full-context alone found it; together they did.

Co-Authored-By: Xerj Squad A <noreply@xerj.org>
xerj-team pushed a commit that referenced this pull request Jul 27, 2026
…ction finding

ATTACK-SCENARIO-role-injection.md: end-to-end verified chain for FINDINGS #4 with
real code snippets and HTTP request templates -- threat model (restricted
multisite admin with filtered editable_roles), Step 1 POST action=adduser
role=administrator -> add_option('new_user_{key}') unguarded, Step 2 GET
/newbloguser/{key}/ -> maybe_add_existing_user_to_blog reads it, Step 3
add_user_to_blog -> set_role applies it with no re-check. Confirmation-side code
(ms-functions.php) read and verified. Honest severity (conditional escalation, not
RCE) + the one-line fix + how XERJ's sibling-sink query surfaced it.

Co-Authored-By: Xerj Squad A <noreply@xerj.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant