Skip to content

autoindex: AST code extraction for 21 more languages (13 → 34) - #317

Merged
xerj-org merged 3 commits into
mainfrom
feat/issue-295-autoindex-languages
Aug 13, 2026
Merged

autoindex: AST code extraction for 21 more languages (13 → 34)#317
xerj-org merged 3 commits into
mainfrom
feat/issue-295-autoindex-languages

Conversation

@xerj-org

Copy link
Copy Markdown
Owner

Fixes #295

xerj autoindex's tree-sitter registry covered 13 languages. Everything else fell through to the prose extractor — and an unclaimed extension is not code to the sniffer at all, so those files were chunked into several body-only records with no language, no defs, no symbols.

Reproduced at HEAD first

A folder of Kotlin/Swift/Lua/Elixir fixtures, indexed by a real xerj autoindex run against a live server on a private port:

util.lua    | language: None | symbols: None | defs: None
App.swift   | language: None | symbols: None | defs: None
server.ex   | language: None | symbols: None | defs: None
Greeter.kt  | language: None | symbols: None | defs: None
(+ 4 more body-only chunk records — 8 docs from 4 files)

What this adds

21 languages — one grammar dep + one registry row + one capture query each, which is what the registry was built for:

Tier Languages
1 Kotlin, Swift, Scala, Dart, Lua, Perl, R, Julia, Haskell, Elixir
2 Erlang, OCaml (+ .mli), Zig, Objective-C, Groovy, PowerShell, F#, Nix
3 Fortran (free-form), MATLAB, Solidity

35 registry rows / 34 languages — OCaml needs two rows because .ml and .mli are separate grammars. documented_language_count pins that number to the docs so the public claim can't drift.

Three registry mechanics were needed

  • Text-predicate evaluation. The core library exposes #eq?/#any-of? as general predicates and applies none of them. In Elixir def is itself a macro, so a definition is a call whose target spells a def-keyword — without predicate evaluation every function call in the file would land in defs. def() rejects unimplemented operators at registry-build time, so a bad predicate fails all_queries_compile rather than silently matching everything.
  • Extension-collision policy (issue design decision Welcome to xerj Discussions! #1), decided once and applied to all collisions: content probe where both owners are registered, dominant-owner-documented where only one is. .m is the only real case — Objective-C markers (#import/@interface/…) win, MATLAB is the probe-less fallback. .pl→Perl, .r→R, .sc→Scala, .h→C (unchanged) are documented dominant-owner rows.
  • _-prefixed captures are predicate-only and never emitted as symbols.

Query provenance

Where a grammar crate ships an author-written queries/tags.scm, the definition patterns are adapted from it and cited in-line (Swift, Scala, Dart, Lua, R, Elixir, OCaml, F#, Nix, Fortran, MATLAB, Solidity); reference-captures and doc-comment plumbing are dropped, as are val/var/property captures where they are the locals trap measured for C in #170. Where no tags.scm ships (Kotlin, Perl, Julia, Haskell, Erlang, Zig, Objective-C, Groovy, PowerShell), patterns come from the grammar's own node-types.json and are pinned by fixture tests.

Locals-trap anchoring is re-derived per grammar, not assumed — Haskell anchors to declarations (a where-bound helper is a local), F# keeps upstream's four top-level contexts (F# nests let constantly), Zig anchors top-level const to source_file. Each has a negative assertion.

Deliberately not added (reasons recorded in Cargo.toml)

  • Clojure — its only release hard-links core ^0.25.6, mutually exclusive with the core 0.26 that tree-sitter-perl forces.
  • source-SQLtree-sitter-sql 0.0.2 is immature and .sql routing must not steal sqldump files (issue design decision Fix/autoindex text family split #3).
  • Nim, Crystal — their only releases are toy grammars (~17 node types, no proc/type/module nodes, verified in node-types.json); no alternative crate exists.
  • fixed-form Fortran (.f) — the grammar is free-form; a fixed-form file would parse to garbage, and an unclaimed extension still indexes as text, which beats a wrong AST.
  • tree-sitter-kotlin 0.3.8 — hard-links core >=0.21,<0.23; the -ng fork of the same fwcd grammar is used instead.

Core bump

tree-sitter 0.25 → 0.26, forced by tree-sitter-perl 1.1.2, the only new grammar with a hard links dep on core. Every other grammar binds through tree-sitter-language and is version-agnostic. Same shape as the c-sharp crate's 0.24 → 0.25 bump.

Licences (acceptance criterion)

All 21 grammar crates are MIT, except tree-sitter-elixir which is Apache-2.0. Both are compatible with XERJ's Apache-2.0. Verified from each crate's own Cargo.toml in the registry cache — not from a manifest, per the repo's verify-before-copying rule.

Tests

  • issue_295_extensions_route — the regression test. Fails at origin/main 803c85b (kt must be recognised as code, verified by appending it to a clean worktree of main), passes here. Pins all 33 newly claimed extensions to their language and asserts the 5 deliberately-unclaimed ones stay unclaimed.
  • all_languages_load — the issue's ABI smoke test: instantiates every registered Language once, so an incompatible grammar fails in CI on the 3-OS matrix instead of at a user's first parse.
  • m_extension_content_probe — the .m decision, including that registry order (objc before matlab) is load-bearing.
  • One real-world-shaped fixture per language asserting ≥1 symbol of each captured kind, plus negative assertions for every locals trap.

cargo test -p xerj-autoindex --lib extract::code56 passed, 0 failed. cargo fmt clean.

Still to report on this PR

Binary-size delta and the local ES-YAML gate run are in progress and will be posted as a comment; CI's own 3-OS matrix, autoindex-fd-smoke, and the ES-YAML job gate the merge regardless.

@cla-bot cla-bot Bot added the cla-signed label Aug 12, 2026
xerj-org added a commit that referenced this pull request Aug 12, 2026
CI's Format + Clippy job failed on this branch with two
`clippy::manual_contains` errors (denied via `-D warnings`), which broke
`cargo clippy --workspace --all-targets` and left PR #317 unmergeable:

    error: using `contains()` instead of `iter().any()` is more efficient
      --> crates/xerj-autoindex/src/extract/code.rs:375:34
      --> crates/xerj-autoindex/src/extract/code.rs:376:43

Both sites evaluate a tree-sitter query predicate over `strings: Vec<&str>`,
testing whether the captured text `got: &str` is present. `manual_contains`
is a rust-1.97 lint; the crate builds clean on older toolchains, which is why
this only surfaced in CI.

Rewrite both to the form clippy suggests:

    "eq?" | "any-of?"         => strings.contains(&got)
    "not-eq?" | "not-any-of?" => !strings.contains(&got)

`<[&str]>::contains(&self, &&str)` compares by `PartialEq` on `&str`, which is
exactly what the closure `|s| *s == got` did — the rewrite is semantically
identical, not a behaviour change. No test expectations move.

Verified locally on clippy 0.1.97 (the same lint set CI runs):
  cargo fmt --all -- --check                              -> clean
  cargo clippy --workspace --all-targets -- -D warnings   -> exit 0
  cargo test -p xerj-autoindex                            -> 556 passed, 0 failed
Motivation
----------
`xerj autoindex`'s tree-sitter registry covered 13 languages. Every other
source tree fell through to the PROSE extractor -- and an unclaimed
extension is not code to the sniffer at all, so those files were chunked
into several body-only records with no `language`, no `defs` and no
`symbols`. Agent-facing code search over a Kotlin/Swift/Elixir/Zig repo
degraded to substring matching.

Reproduced at HEAD before the fix (real run, not inferred): a folder of
Kotlin/Swift/Lua/Elixir fixtures indexed by `xerj autoindex` against a
live server produced 8 documents from 4 files, every one of them with
`language: null, symbols: null, defs: null`.

What changed
------------
21 languages, one grammar dep + one registry row + one capture query each,
exactly as the registry was designed for:

  Tier 1  Kotlin, Swift, Scala, Dart, Lua, Perl, R, Julia, Haskell, Elixir
  Tier 2  Erlang, OCaml (+ .mli), Zig, Objective-C, Groovy, PowerShell,
          F#, Nix
  Tier 3  Fortran (free-form), MATLAB, Solidity

Registry is 35 rows / 34 languages -- OCaml needs two rows because `.ml`
and `.mli` are separate grammars. `documented_language_count` pins that
number to the docs so the public claim cannot drift.

Three registry mechanics were added to support them:

* Text-predicate evaluation (`predicates_hold`). The core library exposes
  `#eq?`/`#any-of?` as *general* predicates and applies NONE of them. In
  Elixir `def` is itself a macro, so a definition is a `call` whose target
  spells a def-keyword -- without predicate evaluation every function call
  in the file would land in `defs`. `def()` rejects operators this does
  not implement at registry-build time, so an unsupported predicate fails
  `all_queries_compile` rather than silently matching everything.
* Extension-collision policy (issue's design decision #1), decided once
  and applied to all collisions: content probe where BOTH owners are
  registered, dominant-owner-documented where only one is. `.m` is the
  only real case -- Objective-C markers (`#import`/`@interface`/...) win,
  MATLAB is the probe-less fallback row. `.pl` -> Perl, `.r` -> R,
  `.sc` -> Scala, and `.h` -> C (unchanged) are dominant-owner rows with
  the reasoning in-line.
* `_`-prefixed captures are predicate-only and never emitted as symbols.

Query provenance
----------------
Where the grammar crate ships an author-written `queries/tags.scm`, the
definition patterns are ADAPTED from it and cited in-line (Swift, Scala,
Dart, Lua, R, Elixir, OCaml, F#, Nix, Fortran, MATLAB, Solidity);
reference-captures and doc-comment plumbing are dropped, and `val`/`var`/
property captures are dropped where they are the locals trap measured for
C in #170. Where no tags.scm ships (Kotlin, Perl, Julia, Haskell, Erlang,
Zig, Objective-C, Groovy, PowerShell) patterns are derived from the
grammar's own `src/node-types.json` and pinned by fixture tests.

Locals-trap anchoring is re-derived per grammar, not assumed:
Haskell anchors to `declarations` (a `where`-bound helper is a local),
F# keeps upstream's four top-level contexts (F# nests `let` constantly),
Zig anchors top-level `const` to `source_file`. Each has a negative
assertion in its test.

Deliberately NOT added, each with the reason recorded in Cargo.toml:
* Clojure -- its only release hard-links core ^0.25.6, mutually exclusive
  with the core 0.26 that tree-sitter-perl forces.
* source-SQL -- `tree-sitter-sql 0.0.2` is immature and `.sql` routing
  must not steal `sqldump` files (issue's design decision #3).
* Nim, Crystal -- their only releases are toy grammars (~17 node types,
  no proc/type/module nodes, verified in node-types.json); no alternative
  crate exists.
* fixed-form Fortran (`.f`) -- the grammar is free-form; a fixed-form file
  would parse to garbage, and an unclaimed extension still indexes as
  text, which is strictly better than a wrong AST.
* `tree-sitter-kotlin` 0.3.8 -- hard-links core >=0.21,<0.23; the `-ng`
  fork of the same fwcd grammar is used instead.

Core bump 0.25 -> 0.26, forced by `tree-sitter-perl 1.1.2` (the only new
grammar with a hard `links` dep on core); every other grammar binds
through `tree-sitter-language` and is version-agnostic. Same shape as the
c-sharp crate's 0.24 -> 0.25 bump.

Licences (issue acceptance criterion): all 21 grammar crates are MIT
except tree-sitter-elixir (Apache-2.0). Both compatible with XERJ's
Apache-2.0. Verified from each crate's own Cargo.toml in the registry
cache, not from a manifest.

Tests
-----
* `issue_295_extensions_route` -- the regression test. FAILS at
  origin/main 803c85b ("kt must be recognised as code", verified in a
  clean worktree), passes here. Pins all 33 newly claimed extensions to
  their language, and asserts the 5 deliberately-unclaimed ones stay
  unclaimed.
* `all_languages_load` -- the issue's ABI smoke test: instantiates every
  registered `Language` once, so an ABI-incompatible grammar fails in CI
  on all three OSes instead of at a user's first parse.
* `m_extension_content_probe` -- the `.m` decision, including that
  registry order (objc before matlab) is load-bearing.
* One real-world-shaped fixture per language asserting >=1 symbol of each
  captured kind, plus negative assertions for every locals trap.
  56 tests in `extract::code`, 0 failed.

Files
-----
- engine/crates/xerj-autoindex/src/extract/code.rs (registry, probes,
  predicate evaluation, 22 queries, fixtures)
- engine/crates/xerj-autoindex/Cargo.toml (deps + exclusion reasons)
- ROADMAP.md, landing/index.html (13 -> 34, pinned by test)

Fixes #295
CI's Format + Clippy job failed on this branch with two
`clippy::manual_contains` errors (denied via `-D warnings`), which broke
`cargo clippy --workspace --all-targets` and left PR #317 unmergeable:

    error: using `contains()` instead of `iter().any()` is more efficient
      --> crates/xerj-autoindex/src/extract/code.rs:375:34
      --> crates/xerj-autoindex/src/extract/code.rs:376:43

Both sites evaluate a tree-sitter query predicate over `strings: Vec<&str>`,
testing whether the captured text `got: &str` is present. `manual_contains`
is a rust-1.97 lint; the crate builds clean on older toolchains, which is why
this only surfaced in CI.

Rewrite both to the form clippy suggests:

    "eq?" | "any-of?"         => strings.contains(&got)
    "not-eq?" | "not-any-of?" => !strings.contains(&got)

`<[&str]>::contains(&self, &&str)` compares by `PartialEq` on `&str`, which is
exactly what the closure `|s| *s == got` did — the rewrite is semantically
identical, not a behaviour change. No test expectations move.

Verified locally on clippy 0.1.97 (the same lint set CI runs):
  cargo fmt --all -- --check                              -> clean
  cargo clippy --workspace --all-targets -- -D warnings   -> exit 0
  cargo test -p xerj-autoindex                            -> 556 passed, 0 failed
@xerj-org
xerj-org force-pushed the feat/issue-295-autoindex-languages branch from f10d3ea to b8a9f9d Compare August 12, 2026 04:32
@xerj-org
xerj-org merged commit 468d74f into main Aug 13, 2026
16 checks passed
buger added a commit to probelabs/xerj that referenced this pull request Aug 13, 2026
main does not build:

    error: failed to select a version for `tree-sitter`.
    package `tree-sitter` links to the native library `tree-sitter`, but it
    conflicts with a previous package which links to `tree-sitter` as well
    note: only one package in the dependency graph may specify the same links
          value to ensure that only one copy of a native library is linked

xerj-engine asked for tree-sitter 0.25 and xerj-autoindex for 0.26. Cargo
permits exactly one crate to `links` a given native library, so the workspace
cannot resolve at all — every build, every test, every CI job.

Neither side is at fault on its own. xerj-org#327 brought code_blocks.rs, written and
verified against 0.25; xerj-org#317 moved autoindex to 0.26 for the Kotlin, Swift,
Scala and Dart grammars. Each merged green; together they do not compile. The
collision is invisible to a PR that only builds its own branch.

Aligning xerj-engine to 0.26 is the safe direction: both crates already pin
identical grammar versions (all 0.23), so only the core binding moves, and node
kind names come from the grammars rather than the core. That is the property
that makes this a one-line change instead of a re-verification of every table
in code_blocks.rs.

Cargo.lock is unchanged — 0.26.12 was already locked for autoindex, so the
engine now resolves to the copy that was being built anyway.

Verified on this branch:
  cargo check -p xerj-server                        builds (was a hard failure)
  cargo test -p xerj-engine --lib code_blocks       15 passed, 0 failed
  cargo test --lib (engine, autoindex, api, query, server)
                                                    1503 passed, 0 failed
  cargo fmt --all -- --check                        exit 0
  cargo clippy --workspace --all-targets -D warnings exit 0
  ES-YAML conformance                               1366 passed, 0 failed, 3 skipped

The code_blocks result is the one that mattered: those node-kind tables were
hand-checked against each grammar's node-types.json under the 0.25 bindings, so
a core bump that shifted any kind name would have surfaced there.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

autoindex: extend AST code extraction beyond 13 languages — 25 missing grammars, tiered worklist

1 participant