Fix/autoindex text family split - #3

Markdown documents were being split across two format families by line shape
alone, which put the same logical content type into two different datasets
with two different field names.

`txt_kind` decided TxtProse vs TxtLines on `avg_len > 60`, with a `<= 5 lines`
escape hatch. On a real mixed corpus that produced:

  docs/postmortem-2026-06-14.md   7 lines, avg 51.7  -> TxtLines -> ax-code (text)
  docs/postmortem-2026-05-03.md   7 lines, avg 38.6  -> TxtLines -> ax-code (text)
  docs/runbook-database.md        5 lines, avg 58.8  -> TxtProse -> ax-docs (body)
  docs/architecture.md            2 lines, avg 130.5 -> TxtProse -> ax-docs (body)

Adding `## Headings` to a markdown file lengthens the line count and shortens
the average line, so a *more* structured document was more likely to be
treated as a record stream. The consequences are not cosmetic: the two
datasets carry different field names, so BM25 statistics are computed over
different corpora and a caller has to query both `body` and `text` to search
one logical collection.

Sentence density is the property that actually separates a document from a
record stream — prose lines end in terminal punctuation, log lines, CSV rows
and source code do not. Measured over the mixed corpus:

  markdown documents .......... 0.43 - 0.57
  syslog ...................... 0.20
  nginx access logs ........... 0.00
  Rust / Python / JS source ... 0.00 - 0.10

A 0.40 threshold separates them cleanly with margin on both sides, and the
existing length rules are kept ahead of it so nothing that classified
correctly before changes.

Verified on the same 35 MB / 458,012-record corpus (22 files: nginx logs plain
and gzipped, syslog, JSONL, two CSV dialects, SQLite, markdown, Python/Rust/JS,
HTML, YAML, TOML, binary junk):

  before   ax-docs 5 docs, ax-code 5 docs   (2 postmortems misfiled as code)
  after    ax-docs 7 docs, ax-code 3 docs   (all markdown / only source)

Ingest unchanged at 9.6 s for 458,012 records across 11 datasets.

Tests: 6 covering both directions — markdown with headings and a short runbook
classify as prose; access logs and syslog are asserted through the real
`classify_text` entry point (the Logs family claims them before `txt_kind` is
consulted, so asserting against `txt_kind` directly would be meaningless);
source code stays line records; long lines stay prose regardless of
punctuation.

ES-YAML conformance: 1360 passed / 0 failed / 3 skipped.
`cargo test -p xerj-autoindex`: 29 passed, 0 failed.

Does NOT fix retrieval ranking on small heterogeneous corpora. With ~11
documents spread over ax-docs / ax-code / ax-web, a `should` across `body` and
`text` still mis-ranks: scores are computed per index over very different
document lengths and are not comparable. Correct data layout is a
prerequisite for fixing that, not a fix for it. Candidate follow-ups: a single
field name for all text-derived chunks, or rank fusion across indices instead
of raw score comparison.

A repeatable gate that measures agent benefit and is built to REFUSE a win
XERJ did not earn. It is not a benchmark XERJ is supposed to pass.

Every number in this repo's history that later proved wrong was wrong in one
of five ways. Each now has an enforced countermeasure, because a rule in a
script survives and a rule in a reviewer's head does not:

* Cheap wrong answers — token savings are computed ONLY over tasks BOTH paths
  answered correctly and agreed on. A line-level index once looked 4.7x
  cheaper than grep while scoring 3/8 on recall.
* Steered baselines — baseline commands may use only vocabulary from the
  question, and every command sits next to its task so the steering is
  auditable.
* Cache mirages — queries vary per run; the first, uncached timing is reported.
* Silent truncation — every response is checked for `timed_out`, and
  aggregation buckets are cross-checked against `hits.total`. A truncated
  answer is scored WRONG, not fast.
* Grading your own homework — correctness is an assertion on CONTENT declared
  before the run, never on which file the answer came from. For value tasks
  the two paths must independently agree; disagreement fails BOTH rather than
  trusting either.

The gate found three real defects on its first run, two of them in itself:

1. Its own answer extractor turned "u4242 events: 2" into "42422" by stripping
   non-digits, inventing a disagreement. An extractor must not be able to
   invent an answer; it now reads the value after the final colon.
2. The BASELINE was wrong: `logs/access-*.log` silently skips
   access-2026-05-31.log.gz, so grep counted 394 where the engine counted 530.
   A real agent grepping `*.log` misses compressed input entirely. Fixed in
   the baseline and left documented, because that blind spot is the finding.
3. Both paths must be equally competent. The XERJ side originally returned
   full ES envelopes — 185 bytes to deliver one integer. `filter_path` cuts
   that to 30 and the run total from 18,920 to 4,062 tokens (4.7x). Comparing
   a tuned baseline against an untuned engine is its own dishonesty.

Result on the 36 MB mixed corpus committed here (22 files: nginx logs plain
and gzipped, syslog, JSONL, two CSV dialects, SQLite, markdown, Python/Rust/JS,
HTML, YAML, TOML, binary junk — 458,012 records):

  task        base  xerj  base tok  xerj tok   base s  xerj s
  lookup      OK    OK           4         7     0.14    0.02
  aggregate   OK    OK          24        60     0.16    0.01
  concept     BAD   OK          43       207     0.04    0.00
  join        OK    OK           8      3987     0.03    0.10
  drilldown   OK    OK           1         8     0.03    0.00
  orient      BAD   OK         172       223     0.00    0.00

  correct: baseline 4/6   xerj 6/6
  tokens over jointly-correct tasks: XERJ uses 109.78x MORE

Read honestly: on a corpus that is 0.01% searchable prose, XERJ wins
CORRECTNESS (the two questions where you do not already know where to look)
and loses TOKENS badly. A shell command that prints one integer cannot be
beaten on tokens by any engine that answers in JSON. The `join` row is the
bulk of the loss and is a real limitation — with no JOIN, one side of the
relation must round-trip through the client (3,987 tokens for one question).

The same gate on a prose/code-heavy corpus inverts this: 234 files / 170k LOC
measured 1,457 tokens for XERJ against 7,762 for grep at equal recall — 5.3x
FEWER. That is why the report prints corpus composition next to the ratio and
says so in words. Retrieval savings scale with the prose fraction; analytics
savings scale with record count. A single headline number across both is
meaningless, and this gate will not produce one.

Exit status is 0 when the REPORT is trustworthy, not when XERJ wins. It exits
non-zero only when a path produced an answer the gate could not verify — that
is, when the measurement itself is broken.

…s lookups

Answers the recurring question — "how does XERJ use 100x more tokens than
grep?" — with a measured model instead of hand-waving, and fixes one silent
wrong answer found while writing it.

## The guidebook (docs/TOKEN_USAGE.md)

The token cost of a XERJ answer decomposes into three parts the AGENT controls:

    tokens ~= envelope_overhead + answer + materialized_intermediate_data

Measured on the reference corpus (bytes on the wire = what becomes tokens):

    pattern                       answer   resp B  envelope B  answer B
    scalar count, trimmed         2            30          29         1
    scalar count, full envelope   2           185         184         1
    terms agg, 5 buckets          5 svcs      211         207         4
    retrieval, 1 hit + fragment   a path      279         269        10
    join, MATERIALIZED            966 ids   15914       15908         6
    join, DENORMALIZED            122          36          33         3

The "100x" is not the engine and not loops or API calls — it is
`materialized_intermediate_data`. The SAME join question costs 15,914 bytes
when the agent pulls one side of the relation into context, and 36 bytes when
the attribute is denormalized at ingest and the question becomes one filtered
aggregation. Identical answer (122). 442x, entirely from the query PATTERN.

Three rules turn 100x-worse into competitive, each measured:
  * `filter_path` to trim the JSON envelope — 6x (185 B -> 30 B)
  * denormalize at ingest instead of materializing a join side — 442x
  * `size:0` when only the aggregate is needed

Plus the corpus-composition law the gate already enforces: retrieval savings
scale with the prose fraction (5.3x FEWER tokens on 170k LOC of source), while
on a 0.01%-prose record corpus XERJ costs more but wins correctness. A single
headline number across both regimes is meaningless; the doc says so and shows
both.

## The fix: `_count` silently returned 0 for terms lookups

Writing the join section surfaced it. `_search` resolves ES `terms` lookups
(`{"terms":{f:{index,id,path}}}`) at its coordination step via
`resolve_terms_lookups`, substituting the concrete value array before parsing.
`_count` did NOT — it parsed the raw lookup object, the parser returned
`MatchNone` with only a `warn`, and `_count` reported 0 where `_search`
reported the true total. A filter that matched 4 documents counted as 0, with
a 200 and no error.

  * `count_docs` (both the single-index and multi-index branches) now calls
    `resolve_terms_lookups` before parsing, exactly like `_search`. Verified:
    `_count` with a lookup filter now returns 4, matching `_search`.
  * The parser's lookup-object arm no longer returns `MatchNone`. An
    unresolved lookup reaching the parser means some endpoint forgot to
    resolve — a bug that should be visible — so it is now a loud parse error
    naming the gap, not a silent empty match.

Corrects an earlier mis-diagnosis: terms lookup is NOT an unimplemented stub.
It is fully supported on `_search` (and now `_count`); an earlier `value:0`
was a nonexistent source-doc id, i.e. correct behaviour. The guidebook
documents it as a real token-efficient join primitive for the case where one
side fits in a single document's array field (allow-lists, cohorts), and keeps
denormalize-at-ingest as the answer for per-document joins.

Tests: `test_unresolved_terms_lookup_is_a_loud_error_not_silent_empty` and
`test_plain_terms_still_parses`. ES-YAML conformance 1360/0/3.
`cargo test -p xerj-query`: 124 passed. (`reindex_pages_past_10k_via_keyset`
fails identically with and without this change — a pre-existing environmental
failure, not a regression.)