Skip to content

Facts ingest: batch phase-B blob measuring, per-phase timing logs, and tighter ingest scope #862

Description

@LivioGama

Why

A cold pixel build-index --history . on a worktree of this repo took 253 s. Reproduced on a fresh clone of the same repo: 140.7 s, with 275 of 287 work ticks spent in phase B (blob measuring). Phase A (metadata) and phase C (diff text) are cheap; phase B is the bottleneck.

Evidence (measured 2026-10-06)

Phase Evidence Cost
Text index (1,028 files) printed early, base_files=1028 negligible
Phase A — commit metadata git log batches of 200 ~24 ms/batch
Phase B — blob measuring 275/287 ticks ~70–80 s of 140 s
Phase C — diff text (365-day window) 10 ticks ~15–20 s

Root cause: measure_commit_blobs (crates/pixel-facts/src/ingest.rs:803) measures one git ls-tree -l process per changed path (blob_size, ingest.rs:847), plus a second spawn per rename against the parent tree. The repo has ~9,170 changed-path rows across 1,024 commits → 10k+ git spawns at ~7 ms each. The comment above the function says cat-file --batch-check, but the implementation is per-path ls-tree.

Amplifier: ingest scope is all refs + stash + every reflog-only commit (rev-list --branches --remotes --tags --reflog, ingest.rs:487,512). A worktree's reflog enumeration picked up 2,382 commits vs 853 on HEAD (commits=2224 in the log), ~1.9× the work.

Ask

  1. Batch phase B: replace per-path ls-tree spawns with one batched cat-file --batch-check (or one ls-tree per commit listing all paths). Expected: phase B drops from ~70 s to seconds; cold --history build under a minute on this repo.
  2. Per-phase timing logs: report/log phases A, B and C separately (tick reports already carry phase; surface a per-phase duration breakdown in the build-index summary so regressions are visible without PIXEL_FACTS_DEBUG_TICKS=1).
  3. Scope control: don't index much more than needed — reflog-only commits inflate the work ~1.9× on worktrees and churn-heavy repos. Study what actually pays off in excavate/rescue/history queries and consider bounding reach (e.g. skip reflog-only by default, or an opt-in/out flag), keeping the age window + size budget limits.

Notes

  • PIXEL_FACTS_DEBUG_TICKS=1 exists but only prints raw tick reports.
  • Reproduction: wipe history.db, run pixel build-index --history ., watch ticks 2–277 sit in phase_b.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions