docs(demo): real AI-agent usage scenarios; deprecate stale sales kit - #4
Merged
Merged
Conversation
The existing demo/ kit is April-2026 sales-engineer material — DEMO_RUNBOOK.md
and README.md pitch XERJ to enterprise buyers and cite v1.0.0-rc.1, conformance
1305/1329 (now 1360/1363), a since-fixed RC1 ingest race, and "hybrid not yet
producing fused rankings" (run_hybrid does RRF today). It is not a guide to
what this product is FOR: an AI coding agent's tool.
This replaces the framing with measured, reproducible agent scenarios and
flags the old material rather than deleting it.
## Added
* `SCENARIOS.md` — six real an AI coding agent tasks with measured before/after and
copy-paste reproduction, every number from a run in this repo:
1. Orient in unfamiliar data — autoindex map, ~890 tokens flat vs growing
2. Where/why in a large repo — 5.3x fewer tokens at 170k LOC; recall 6/7
vs grep 3/7 on prose (grep is lexical, misses reworded answers)
3. Analytics over logs — 6/6 vs grep 4/6 correctness (the gate caught the
baseline skipping a .gz and mis-parsing a decimal-comma CSV)
4. Incident drill-down — indexed lookup 0.007s vs grep 7.3s at 1.2 GB
(1043x), and flat as data grows
5. Relational without JOIN — the denormalize pattern: 36 bytes vs 15,914
(442x) for an identical answer
6. Cross-session memory — 328-byte semantic recall, a capability grep lacks
Each scenario states plainly when XERJ does NOT help.
* `gate_retrieval.py` + `make_corpus_prose.py` — the retrieval regime, the
half the analytics gate could not show. On a 92%-prose corpus XERJ scores
6/7 vs grep 3/7; token parity at 53 KB, widening to 5.3x fewer at 170k LOC.
Same anti-cheating discipline as gate.py.
* `RESULTS_analytics.txt` / `RESULTS_retrieval.txt` — committed reference runs
so a reader sees both regimes without running anything.
## The through-line
One law predicts every result and is printed next to every ratio: retrieval
savings scale with the prose fraction; analytics savings scale with record
count but analytics answers are small either way, so there the win is
correctness and latency, not tokens. A single headline number across both
regimes is dishonest, and neither gate will produce one.
## Deprecated (kept, bannered)
DEMO_RUNBOOK.md and README.md now carry a stale-material banner pointing to
SCENARIOS.md and docs/TOKEN_USAGE.md. Refresh their numbers before any external
use.
Not yet audited: demo/playbooks/ (ES-vs-XERJ perf battle plans) and
demo/usecases/ — a follow-up pass. The autoindex usecase evaluation there
remains the most current of the old material.
xerj-team
pushed a commit
that referenced
this pull request
Jul 27, 2026
…ates + trigger traces Recovery from journal.jsonl after the usage-limit interruption: 177/177 review batches completed -> ALL 883 files were analyzed. Only the VERIFY phase dropped 8 of 44 candidates (not files). Hand-verified the 8 by reading real code: - cover.php:102 REFUTED (fixed iframe + esc_url, not raw oEmbed HTML) - feed-rss2-comments.php:107 plausible low-med (]]> CDATA breakout into comments feed; affects feed consumers, not the site; unconfirmed) - custom-css.php:258 plausible med (edit_css strip wired only to content_save_pre; block-widget REST save may bypass; needs path trace) - wp-mail.php (by-design Post-by-Email), wp-trackback.php (the code IS the UTF-7 mitigation) -> FP/by-design; 3 REST candidates low/unconfirmed. Net: lead-confirmed new finding stays user-new.php role injection (#4); added the exact HTTP trigger trace (POST action=adduser role=administrator -> stored option -> GET /newbloguser/{key}/ -> set_role, with the multisite+filtered-editable_roles conditions). Added "How XERJ makes this discoverable vs a full-context search": (1) 5.2M-token core doesn't fit a window -> chunking splits the cross-file role flow; (2) XERJ hands the reviewer the 3 sibling role-sinks to compare (the bug is a missing sibling guard, invisible without the map); (3) index-once vs re-read-everything. Neither structural graph nor blind full-context alone found it; together they did. Co-Authored-By: Xerj Squad A <noreply@xerj.org>
xerj-team
pushed a commit
that referenced
this pull request
Jul 27, 2026
…ction finding ATTACK-SCENARIO-role-injection.md: end-to-end verified chain for FINDINGS #4 with real code snippets and HTTP request templates -- threat model (restricted multisite admin with filtered editable_roles), Step 1 POST action=adduser role=administrator -> add_option('new_user_{key}') unguarded, Step 2 GET /newbloguser/{key}/ -> maybe_add_existing_user_to_blog reads it, Step 3 add_user_to_blog -> set_role applies it with no re-check. Confirmation-side code (ms-functions.php) read and verified. Honest severity (conditional escalation, not RCE) + the one-line fix + how XERJ's sibling-sink query surfaced it. Co-Authored-By: Xerj Squad A <noreply@xerj.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The existing demo/ kit is April-2026 sales-engineer material — DEMO_RUNBOOK.md and README.md pitch XERJ to enterprise buyers and cite v1.0.0-rc.1, conformance 1305/1329 (now 1360/1363), a since-fixed RC1 ingest race, and "hybrid not yet producing fused rankings" (run_hybrid does RRF today). It is not a guide to what this product is FOR: an AI coding agent's tool.
This replaces the framing with measured, reproducible agent scenarios and flags the old material rather than deleting it.
Added
SCENARIOS.md— six real an AI coding agent tasks with measured before/after and copy-paste reproduction, every number from a run in this repo:gate_retrieval.py+make_corpus_prose.py— the retrieval regime, the half the analytics gate could not show. On a 92%-prose corpus XERJ scores 6/7 vs grep 3/7; token parity at 53 KB, widening to 5.3x fewer at 170k LOC. Same anti-cheating discipline as gate.py.RESULTS_analytics.txt/RESULTS_retrieval.txt— committed reference runs so a reader sees both regimes without running anything.The through-line
One law predicts every result and is printed next to every ratio: retrieval savings scale with the prose fraction; analytics savings scale with record count but analytics answers are small either way, so there the win is correctness and latency, not tokens. A single headline number across both regimes is dishonest, and neither gate will produce one.
Deprecated (kept, bannered)
DEMO_RUNBOOK.md and README.md now carry a stale-material banner pointing to SCENARIOS.md and docs/TOKEN_USAGE.md. Refresh their numbers before any external use.
Not yet audited: demo/playbooks/ (ES-vs-XERJ perf battle plans) and demo/usecases/ — a follow-up pass. The autoindex usecase evaluation there remains the most current of the old material.