feat(autoindex): estimate on the user's machine, ask before a long run, index what matters first - #281

landing/llms.txt already tells an agent driving XERJ on someone's laptop to estimate, tell them, ask if it is big, report progress. That was advice, and advice is ignorable. This makes the binary do it.

"search should be smart with ai agent harness getting real important things to index first and make it clear for user that indexing will take time if many files, so xerj autoindex if estimated more than 10min work needs to ask AI back what to do" — owner

Built on #240 (resource policy) and #241 (progress surface); neither is re-derived here.

(a) Estimate, measured on the user's machine

Phase A already reads and parses every file to sniff and sample it, so it now times that work per format family:

estimate: at least 64.2 s–64.2 s — a MEASURED FLOOR for client-side extraction,
          not a prediction of the whole run: server indexing, embedding and
          network time are not in it
estimate: basis — measured on this machine during phase A: 567 file(s) /
          793.4 MB read end to end across 3 family/families, scheduled over
          1 worker(s). Covers 793.4 MB of the 793.4 MB planned (100%).
  estimate: code   500 files 749.3 MB at 11.7 MB/s measured over 500 file(s) → 64.2 s
  estimate: yaml    12 files  23.8 KB at 13.1 MB/s measured over  12 file(s) → 0.0 s

A family is priced only from files phase A provably read end to end — decided from each extractor's own contract, not assumed (whole-file parsers always; streaming parsers only under the sampling byte cap and short of the record cap; never sqlite, never gzip). Everything else is named under unmeasured_families with its bytes rather than priced at another family's rate, and coverage reports the share of planned bytes covered. No measurement → no number and no gate; basis says why. The two ends are the list-scheduling bounds (Graham 1969) over the phase-B worker count #240 chose.

It is a floor, and is labelled as one everywhere (kind, headline, excludes, --help, llms.txt). It covers client-side extraction only, because measuring the server, the network or embedding means writing to the index the estimate exists to ask permission for. Measured here: 68 MB source tree → floor 0.1 s vs real 8.9 s; 793 MB → 64.2 s vs ~350 s. So the gate under-asks and never over-asks, and the "not triggered" line says in words that silence is not a promise the run is short. Closing that gap with a multiplier would mean gating on a number nobody measured.

(b) The 10-minute gate

Past --max-minutes (default 10) with no answer: nothing is indexed, a JSON decision request goes to stdout, and the process exits 4 — its own code, because 1 is already the catch-all for every real failure and an agent must be able to tell "choose something" from "your endpoint is down".

The payload carries the estimate and basis, file/byte counts, the per-band work order, the heaviest directories with real byte counts (flagged when they match the vendored/generated rule), and four options. narrower is re-costed with the same measured rates. fast states no speed-up factor — it reports which datasets and how many files it changes and says plainly that this run did not measure the factor.

Answer with --approve proceed|fast|cancel (--yes = proceed); --max-minutes 0 disables it. --approve fast really applies --no-semantic --no-graph; --approve narrower is refused with instructions rather than accepted and silently ignored (#204). A person at a terminal gets the same facts as prose plus a prompt; a piped or agent-driven run never touches stdin, and an unanswerable prompt is a cancel, never consent. The gate sits before the first remote mutation.

(c) Index what matters first

The queue was sorted by size alone: right about scheduling, silent about value — a user who stopped early got whatever was largest, which on a source tree is node_modules. Work now drains by value band (source and documents → configuration → structured data → logs and line files → vendored/generated/minified), with the old biggest-first rule kept inside each band so a large file runs alongside its band instead of becoming the tail; a single-worker run goes smallest-first. One exception: a file whose extraction outlasts everything ranked above it (size × workers > the rest) starts first regardless of band. Bands, counts and the reason each sits where it does are printed with the plan and shipped as priority_order — an unexplained order is indistinguishable from an arbitrary one. Bytes-based progress is unaffected.

Verified live (branch binary, real node)

Gates

• cargo fmt --all -- --check clean
• cargo clippy -p xerj-autoindex -p xerj-server --all-targets -- -D warnings clean
• cargo test -p xerj-autoindex — 390 passed / 0 failed
• cargo test -p xerj-server --bins — 30 passed / 0 failed
• ES-YAML conformance — 1366 passed / 0 failed / 3 skipped

Known limits (stated, not hidden)

1. The estimate is a floor, not a prediction; the gate therefore misses runs whose cost is server-side. A measured server term needs a probe write, which would cost the "nothing of yours has been touched" property that makes stopping safe. Follow-up, not silently assumed away.
2. A frozen-plan resume has no fresh measurement, so it prints "no estimate" and does not gate. Correct, but it means an interrupted long run is never re-gated.
3. xerj brain composes autoindex and has no --approve, so it sets max_minutes: 0 explicitly (one hunk in xerj-server/src/brain.rs) — arming the gate there would stop the run with instructions its caller cannot follow.
4. Band misclassification is possible (build, out, obj, dist are ambiguous names). The cost is a file's place in the queue and nothing else — it is still indexed, still complete.

Reference-coding note

The retrieval server was not reachable on localhost:9200, and the only live node carried an unrelated networking corpus, so I read the corpus clones in ~/.xerj-code/corpora/ directly rather than starting or reindexing anything: tantivy log_merge_policy.rs:94-99 (MIT — sorts Reverse(size) then buckets into levels; adapted, our levels are value bands), quickwit quickwit-cli/src/tool.rs:977-1012 and ClickHouse ProgressIndication.cpp:118-119,223-227 (both report measured bytes/s and stop short of an ETA), meilisearch milli::progress (named steps, no ETA at all). None of the four estimates a job before running it.

Rebased onto main (post-#259 incremental route)

main landed the generated/incremental reconcile path while this was in flight. Rebased and re-verified end to end on the rebased tree; all figures above were re-measured on it (exit 4, 0 gt-* indices, ordering unchanged, ES-YAML 1366/0/3, cargo test -p xerj-autoindex now 465 passed / 0 failed).

One interaction worth calling out: an incremental reconcile of an already committed generation (--no-graph re-run) publishes from a sealed snapshot and never reaches the phase-B queue the estimate prices, so --max-minutes cannot apply there. Rather than accept the flag and quietly not honour it (#204), that route now prints the exemption and points at --dry-run. Both phase-A routes feed the same throughput meter.

Pre-existing bug found while verifying (NOT from this PR)

A second --no-graph run over the same folder and state dir fails:

error: desired manifest digest does not match sync_begin payload   (exit 1)

Reproduced identically on unmodified origin/main @ 8fb08a9 with a binary built from a clean worktree — first run exits 0, second exits 1 — so it is a defect in the newly-landed incremental route, not a regression here. No flag of this PR is involved (--max-minutes 0, --max-minutes 10 and the default all reproduce, and none of the new fields enter prepared_records_identity). Worth its own issue.

Fix: --quiet made the gate's question invisible but still blocking

Found by measurement during the rebase verification above, and the single worst
failure mode this feature could have: an invisible prompt is indistinguishable
from a hang, in the one feature whose entire job is to say "this will take a
while".

The gate's prose and its prompt echo both go through the progress surface
(Progress::note), which --quiet / --progress none silences. But
can_prompt() only checked that stdin and stderr were TTYs — never that the
question had actually been printed. So a person at a terminal who passed
--quiet was asked nothing and then waited on read_line forever. Agent-driven
runs were never affected (stdin is not a TTY, so no prompt was attempted).

Measured on a real pty against a real node, 370 MB / 540 files,
--workers 1 --max-minutes 1:

No ax-* index was created by any run — the gate still sits before the first
remote mutation.

Why route (a), and not (b)

Both routes were on the table: (a) skip the prompt when the question cannot be
printed, or (b) route the gate's prose and prompt to stderr independently of
progress mode. (a).

1. --quiet has a documented contract in --help: "prints no progress and
   NO terminal line (only a fatal error: line)". Route (b) writes ~30 lines of
   prose plus an interactive prompt to a stream the user explicitly silenced.
   Accepting a flag and then not honouring it is the accepted-and-ignored class
   tracked in Tracking: accepted-and-ignored is the dominant bug class, six instances found in one review #204 — the same defect this PR already refuses elsewhere with
   --approve narrower.
2. (a) is not a silent early exit. The decision request goes to stdout,
   which --quiet never claimed to silence (stdout is the RESULT, stderr is PROGRESS — the help says so). The user still gets every fact, in
   machine-readable form, plus exit 4. The only thing lost is the prose
   rendering of information they still receive — and prose is precisely what
   they opted out of.
3. It makes quiet runs uniform with agent-driven ones. One rule to document
   and reason about, instead of a fourth distinct behaviour.
4. (b) would newly let a --quiet run block on stdin, enlarging the set of
   configurations that can wait for a human. That is the hazard itself.

The honest objection to (a) — "a user who asked for quiet and got a silent early
exit may be equally confused" — is answered rather than waved away: the payload
now carries prompt_offered and prompt_not_offered_because, and the prose
prints the same sentence wherever a surface exists to print it. Every reason
names the way out (--approve …), so no path ends in "wait". Under --quiet
the user sees, on stdout:

"prompt_offered": false,
"prompt_not_offered_because": "no terminal prompt: --quiet / --progress none
  silences the question, and this run will never wait on stdin for something it
  did not print. The decision request below is on stdout, which --quiet does not
  silence. Re-run with --approve proceed|fast|cancel, or without --quiet to be asked"

Implementation

• gate::NoPrompt + gate::prompt_blocked_by(stdin_tty, stderr_tty, question_visible) —
  pure, so the whole rule is testable without a terminal (same shape as
  progress::resolve). question_visible is the third condition that was missing.
• gate::answer_from_terminal(blocked, open_stdin, echo) takes stdin as a
  closure, so on a blocked path the process's stdin is never even opened.
  That is the invariant — not "no answer was returned" but "stdin was not read".
• can_prompt() is removed, not kept: a function with that name that ignores
  the surface is exactly the footgun that caused this.

Regression test

gate::tests::a_question_that_cannot_be_seen_is_never_asked hands in a reader
that panics if anything touches it. Verified to fail without the fix, on both
halves of it:

• classifier reverted to the tty-only rule → fails the QuestionSilenced assertion
• guard reverted → panics stdin was read for a question the user was never shown

Documented

--help (the --quiet entry and the ESTIMATE + DECISION GATE section) and
landing/llms.txt now state the three conditions, that --quiet is never
prompted, and that autoindex never waits on stdin for a question it did not
print. A help-text test pins all three strings.

Gates (re-run on the fixed tree)

• cargo fmt --all -- --check — clean
• cargo clippy -p xerj-autoindex --all-targets -- -D warnings — clean
• cargo test -p xerj-autoindex — 475 passed / 0 failed (473 before; +2 new)