autoindex: AST code extraction for 21 more languages (13 → 34) - #317
Merged
Conversation
xerj-org
added a commit
that referenced
this pull request
Aug 12, 2026
CI's Format + Clippy job failed on this branch with two `clippy::manual_contains` errors (denied via `-D warnings`), which broke `cargo clippy --workspace --all-targets` and left PR #317 unmergeable: error: using `contains()` instead of `iter().any()` is more efficient --> crates/xerj-autoindex/src/extract/code.rs:375:34 --> crates/xerj-autoindex/src/extract/code.rs:376:43 Both sites evaluate a tree-sitter query predicate over `strings: Vec<&str>`, testing whether the captured text `got: &str` is present. `manual_contains` is a rust-1.97 lint; the crate builds clean on older toolchains, which is why this only surfaced in CI. Rewrite both to the form clippy suggests: "eq?" | "any-of?" => strings.contains(&got) "not-eq?" | "not-any-of?" => !strings.contains(&got) `<[&str]>::contains(&self, &&str)` compares by `PartialEq` on `&str`, which is exactly what the closure `|s| *s == got` did — the rewrite is semantically identical, not a behaviour change. No test expectations move. Verified locally on clippy 0.1.97 (the same lint set CI runs): cargo fmt --all -- --check -> clean cargo clippy --workspace --all-targets -- -D warnings -> exit 0 cargo test -p xerj-autoindex -> 556 passed, 0 failed
Motivation
----------
`xerj autoindex`'s tree-sitter registry covered 13 languages. Every other
source tree fell through to the PROSE extractor -- and an unclaimed
extension is not code to the sniffer at all, so those files were chunked
into several body-only records with no `language`, no `defs` and no
`symbols`. Agent-facing code search over a Kotlin/Swift/Elixir/Zig repo
degraded to substring matching.
Reproduced at HEAD before the fix (real run, not inferred): a folder of
Kotlin/Swift/Lua/Elixir fixtures indexed by `xerj autoindex` against a
live server produced 8 documents from 4 files, every one of them with
`language: null, symbols: null, defs: null`.
What changed
------------
21 languages, one grammar dep + one registry row + one capture query each,
exactly as the registry was designed for:
Tier 1 Kotlin, Swift, Scala, Dart, Lua, Perl, R, Julia, Haskell, Elixir
Tier 2 Erlang, OCaml (+ .mli), Zig, Objective-C, Groovy, PowerShell,
F#, Nix
Tier 3 Fortran (free-form), MATLAB, Solidity
Registry is 35 rows / 34 languages -- OCaml needs two rows because `.ml`
and `.mli` are separate grammars. `documented_language_count` pins that
number to the docs so the public claim cannot drift.
Three registry mechanics were added to support them:
* Text-predicate evaluation (`predicates_hold`). The core library exposes
`#eq?`/`#any-of?` as *general* predicates and applies NONE of them. In
Elixir `def` is itself a macro, so a definition is a `call` whose target
spells a def-keyword -- without predicate evaluation every function call
in the file would land in `defs`. `def()` rejects operators this does
not implement at registry-build time, so an unsupported predicate fails
`all_queries_compile` rather than silently matching everything.
* Extension-collision policy (issue's design decision #1), decided once
and applied to all collisions: content probe where BOTH owners are
registered, dominant-owner-documented where only one is. `.m` is the
only real case -- Objective-C markers (`#import`/`@interface`/...) win,
MATLAB is the probe-less fallback row. `.pl` -> Perl, `.r` -> R,
`.sc` -> Scala, and `.h` -> C (unchanged) are dominant-owner rows with
the reasoning in-line.
* `_`-prefixed captures are predicate-only and never emitted as symbols.
Query provenance
----------------
Where the grammar crate ships an author-written `queries/tags.scm`, the
definition patterns are ADAPTED from it and cited in-line (Swift, Scala,
Dart, Lua, R, Elixir, OCaml, F#, Nix, Fortran, MATLAB, Solidity);
reference-captures and doc-comment plumbing are dropped, and `val`/`var`/
property captures are dropped where they are the locals trap measured for
C in #170. Where no tags.scm ships (Kotlin, Perl, Julia, Haskell, Erlang,
Zig, Objective-C, Groovy, PowerShell) patterns are derived from the
grammar's own `src/node-types.json` and pinned by fixture tests.
Locals-trap anchoring is re-derived per grammar, not assumed:
Haskell anchors to `declarations` (a `where`-bound helper is a local),
F# keeps upstream's four top-level contexts (F# nests `let` constantly),
Zig anchors top-level `const` to `source_file`. Each has a negative
assertion in its test.
Deliberately NOT added, each with the reason recorded in Cargo.toml:
* Clojure -- its only release hard-links core ^0.25.6, mutually exclusive
with the core 0.26 that tree-sitter-perl forces.
* source-SQL -- `tree-sitter-sql 0.0.2` is immature and `.sql` routing
must not steal `sqldump` files (issue's design decision #3).
* Nim, Crystal -- their only releases are toy grammars (~17 node types,
no proc/type/module nodes, verified in node-types.json); no alternative
crate exists.
* fixed-form Fortran (`.f`) -- the grammar is free-form; a fixed-form file
would parse to garbage, and an unclaimed extension still indexes as
text, which is strictly better than a wrong AST.
* `tree-sitter-kotlin` 0.3.8 -- hard-links core >=0.21,<0.23; the `-ng`
fork of the same fwcd grammar is used instead.
Core bump 0.25 -> 0.26, forced by `tree-sitter-perl 1.1.2` (the only new
grammar with a hard `links` dep on core); every other grammar binds
through `tree-sitter-language` and is version-agnostic. Same shape as the
c-sharp crate's 0.24 -> 0.25 bump.
Licences (issue acceptance criterion): all 21 grammar crates are MIT
except tree-sitter-elixir (Apache-2.0). Both compatible with XERJ's
Apache-2.0. Verified from each crate's own Cargo.toml in the registry
cache, not from a manifest.
Tests
-----
* `issue_295_extensions_route` -- the regression test. FAILS at
origin/main 803c85b ("kt must be recognised as code", verified in a
clean worktree), passes here. Pins all 33 newly claimed extensions to
their language, and asserts the 5 deliberately-unclaimed ones stay
unclaimed.
* `all_languages_load` -- the issue's ABI smoke test: instantiates every
registered `Language` once, so an ABI-incompatible grammar fails in CI
on all three OSes instead of at a user's first parse.
* `m_extension_content_probe` -- the `.m` decision, including that
registry order (objc before matlab) is load-bearing.
* One real-world-shaped fixture per language asserting >=1 symbol of each
captured kind, plus negative assertions for every locals trap.
56 tests in `extract::code`, 0 failed.
Files
-----
- engine/crates/xerj-autoindex/src/extract/code.rs (registry, probes,
predicate evaluation, 22 queries, fixtures)
- engine/crates/xerj-autoindex/Cargo.toml (deps + exclusion reasons)
- ROADMAP.md, landing/index.html (13 -> 34, pinned by test)
Fixes #295
CI's Format + Clippy job failed on this branch with two `clippy::manual_contains` errors (denied via `-D warnings`), which broke `cargo clippy --workspace --all-targets` and left PR #317 unmergeable: error: using `contains()` instead of `iter().any()` is more efficient --> crates/xerj-autoindex/src/extract/code.rs:375:34 --> crates/xerj-autoindex/src/extract/code.rs:376:43 Both sites evaluate a tree-sitter query predicate over `strings: Vec<&str>`, testing whether the captured text `got: &str` is present. `manual_contains` is a rust-1.97 lint; the crate builds clean on older toolchains, which is why this only surfaced in CI. Rewrite both to the form clippy suggests: "eq?" | "any-of?" => strings.contains(&got) "not-eq?" | "not-any-of?" => !strings.contains(&got) `<[&str]>::contains(&self, &&str)` compares by `PartialEq` on `&str`, which is exactly what the closure `|s| *s == got` did — the rewrite is semantically identical, not a behaviour change. No test expectations move. Verified locally on clippy 0.1.97 (the same lint set CI runs): cargo fmt --all -- --check -> clean cargo clippy --workspace --all-targets -- -D warnings -> exit 0 cargo test -p xerj-autoindex -> 556 passed, 0 failed
xerj-org
force-pushed
the
feat/issue-295-autoindex-languages
branch
from
August 12, 2026 04:32
f10d3ea to
b8a9f9d
Compare
buger
added a commit
to probelabs/xerj
that referenced
this pull request
Aug 13, 2026
main does not build:
error: failed to select a version for `tree-sitter`.
package `tree-sitter` links to the native library `tree-sitter`, but it
conflicts with a previous package which links to `tree-sitter` as well
note: only one package in the dependency graph may specify the same links
value to ensure that only one copy of a native library is linked
xerj-engine asked for tree-sitter 0.25 and xerj-autoindex for 0.26. Cargo
permits exactly one crate to `links` a given native library, so the workspace
cannot resolve at all — every build, every test, every CI job.
Neither side is at fault on its own. xerj-org#327 brought code_blocks.rs, written and
verified against 0.25; xerj-org#317 moved autoindex to 0.26 for the Kotlin, Swift,
Scala and Dart grammars. Each merged green; together they do not compile. The
collision is invisible to a PR that only builds its own branch.
Aligning xerj-engine to 0.26 is the safe direction: both crates already pin
identical grammar versions (all 0.23), so only the core binding moves, and node
kind names come from the grammars rather than the core. That is the property
that makes this a one-line change instead of a re-verification of every table
in code_blocks.rs.
Cargo.lock is unchanged — 0.26.12 was already locked for autoindex, so the
engine now resolves to the copy that was being built anyway.
Verified on this branch:
cargo check -p xerj-server builds (was a hard failure)
cargo test -p xerj-engine --lib code_blocks 15 passed, 0 failed
cargo test --lib (engine, autoindex, api, query, server)
1503 passed, 0 failed
cargo fmt --all -- --check exit 0
cargo clippy --workspace --all-targets -D warnings exit 0
ES-YAML conformance 1366 passed, 0 failed, 3 skipped
The code_blocks result is the one that mattered: those node-kind tables were
hand-checked against each grammar's node-types.json under the 0.25 bindings, so
a core bump that shifted any kind name would have surfaced there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #295
xerj autoindex's tree-sitter registry covered 13 languages. Everything else fell through to the prose extractor — and an unclaimed extension is not code to the sniffer at all, so those files were chunked into several body-only records with nolanguage, nodefs, nosymbols.Reproduced at HEAD first
A folder of Kotlin/Swift/Lua/Elixir fixtures, indexed by a real
xerj autoindexrun against a live server on a private port:What this adds
21 languages — one grammar dep + one registry row + one capture query each, which is what the registry was built for:
.mli), Zig, Objective-C, Groovy, PowerShell, F#, Nix35 registry rows / 34 languages — OCaml needs two rows because
.mland.mliare separate grammars.documented_language_countpins that number to the docs so the public claim can't drift.Three registry mechanics were needed
#eq?/#any-of?as general predicates and applies none of them. In Elixirdefis itself a macro, so a definition is acallwhose target spells a def-keyword — without predicate evaluation every function call in the file would land indefs.def()rejects unimplemented operators at registry-build time, so a bad predicate failsall_queries_compilerather than silently matching everything..mis the only real case — Objective-C markers (#import/@interface/…) win, MATLAB is the probe-less fallback..pl→Perl,.r→R,.sc→Scala,.h→C (unchanged) are documented dominant-owner rows._-prefixed captures are predicate-only and never emitted as symbols.Query provenance
Where a grammar crate ships an author-written
queries/tags.scm, the definition patterns are adapted from it and cited in-line (Swift, Scala, Dart, Lua, R, Elixir, OCaml, F#, Nix, Fortran, MATLAB, Solidity); reference-captures and doc-comment plumbing are dropped, as areval/var/property captures where they are the locals trap measured for C in #170. Where no tags.scm ships (Kotlin, Perl, Julia, Haskell, Erlang, Zig, Objective-C, Groovy, PowerShell), patterns come from the grammar's ownnode-types.jsonand are pinned by fixture tests.Locals-trap anchoring is re-derived per grammar, not assumed — Haskell anchors to
declarations(awhere-bound helper is a local), F# keeps upstream's four top-level contexts (F# nestsletconstantly), Zig anchors top-levelconsttosource_file. Each has a negative assertion.Deliberately not added (reasons recorded in
Cargo.toml)^0.25.6, mutually exclusive with the core 0.26 thattree-sitter-perlforces.tree-sitter-sql 0.0.2is immature and.sqlrouting must not stealsqldumpfiles (issue design decision Fix/autoindex text family split #3).node-types.json); no alternative crate exists..f) — the grammar is free-form; a fixed-form file would parse to garbage, and an unclaimed extension still indexes as text, which beats a wrong AST.tree-sitter-kotlin0.3.8 — hard-links core>=0.21,<0.23; the-ngfork of the same fwcd grammar is used instead.Core bump
tree-sitter0.25 → 0.26, forced bytree-sitter-perl 1.1.2, the only new grammar with a hardlinksdep on core. Every other grammar binds throughtree-sitter-languageand is version-agnostic. Same shape as the c-sharp crate's 0.24 → 0.25 bump.Licences (acceptance criterion)
All 21 grammar crates are MIT, except
tree-sitter-elixirwhich is Apache-2.0. Both are compatible with XERJ's Apache-2.0. Verified from each crate's ownCargo.tomlin the registry cache — not from a manifest, per the repo's verify-before-copying rule.Tests
issue_295_extensions_route— the regression test. Fails atorigin/main803c85b (kt must be recognised as code, verified by appending it to a clean worktree of main), passes here. Pins all 33 newly claimed extensions to their language and asserts the 5 deliberately-unclaimed ones stay unclaimed.all_languages_load— the issue's ABI smoke test: instantiates every registeredLanguageonce, so an incompatible grammar fails in CI on the 3-OS matrix instead of at a user's first parse.m_extension_content_probe— the.mdecision, including that registry order (objc before matlab) is load-bearing.cargo test -p xerj-autoindex --lib extract::code→ 56 passed, 0 failed.cargo fmtclean.Still to report on this PR
Binary-size delta and the local ES-YAML gate run are in progress and will be posted as a comment; CI's own 3-OS matrix,
autoindex-fd-smoke, and the ES-YAML job gate the merge regardless.