Skip to content

Make commit-history discovery distinct from full file-history reconstruction #1068

Description

@samuelstroschein

Context

While building a Git/Git LFS/Perforce/Lix benchmark, I needed the Lix equivalent of git log --all --format=%H and p4 changes: list commit records for reopen/history-query latency.

The initially discoverable query was:

SELECT path, lixcol_depth
FROM lix_file_history()
ORDER BY lixcol_depth, path;

That is not an equivalent commit-log operation. It reconstructs every historical file state, including semantic Markdown state. On a fixed 25-commit, 187-path wesnoth/wesnoth window containing a 991 KB structured changelog, this query was still running after 15 minutes at one saturated CPU core and about 2.0 GiB RSS when I stopped it.

The equivalent low-cost query for this benchmark is instead:

SELECT id FROM lix_commit ORDER BY id;

That returns the commit records quickly, but it was not obvious from the history/file APIs that lix_file_history() implied eager full-state reconstruction even when the SQL projection requested only path and lixcol_depth.

API friction

  • “history query” naturally led to the file-history surface, but commit history is a separate entity surface.
  • The cost model of lix_file_history() is not apparent at the call site.
  • Projecting metadata-only columns does not appear to avoid reconstructing file contents/semantic rows.
  • SELECT id FROM lix_commit lists commits, but discovering branch-reachable chronological history equivalent to git log is not self-evident from the surface alone.

Suggested improvements

  1. Document a canonical branch-history query (or expose a dedicated branch-reachable commit-history surface/helper).
  2. Document explicitly that lix_file_history() reconstructs historical file states and can be proportional to all semantic entities across revisions.
  3. Add projection/filter pushdown so metadata-only file-history queries do not materialize file bytes or semantic document state when those columns are not selected.
  4. Consider naming/docs that distinguish commit graph traversal from historical file reconstruction.

Reproduction window

This issue is about discoverability/cost transparency and possible metadata projection pushdown, not about making exhaustive historical reconstruction look equivalent to a commit-log query.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions