syntext

A faster grep for agentic AI. ~20X faster than ripgrep when indexed.

Hybrid code search index for agent workflows, built in Rust. Indexes repositories using sparse n-grams, then narrows to a small candidate set before verification. Drop-in replacement for rg in AI agent loops where grep is called repeatedly and in parallel.

Status: stable (v1.1).

Installation

Quick install (macOS and Linux)

curl -fsSL https://raw.githubusercontent.com/whit3rabbit/syntext/main/install.sh | sh

Installs st to /usr/local/bin. On macOS, uses Homebrew cask if brew is available. On Debian/Ubuntu (x86_64), installs the .deb package. All other Linux targets get the raw binary. Checksums are verified against SHA256SUMS from the release.

Override defaults with environment variables:

INSTALL_DIR=~/.local/bin SYNTEXT_VERSION=1.1.1 \
  curl -fsSL https://raw.githubusercontent.com/whit3rabbit/syntext/main/install.sh | sh

brew tap whit3rabbit/tap
brew install --cask whit3rabbit/tap/syntext

VERSION=1.1.1

# Debian/Ubuntu (x86_64)
curl -L "https://github.com/whit3rabbit/syntext/releases/download/v${VERSION}/syntext_${VERSION}_amd64.deb" \
  -o "syntext_${VERSION}_amd64.deb"
sudo dpkg -i "syntext_${VERSION}_amd64.deb"

# Any Linux (x86_64 or arm64)
ARCH=amd64   # or arm64
curl -L "https://github.com/whit3rabbit/syntext/releases/download/v${VERSION}/st-${VERSION}-linux-${ARCH}" -o st
chmod +x st && sudo mv st /usr/local/bin/

iwr -useb https://raw.githubusercontent.com/whit3rabbit/syntext/main/install.ps1 | iex

powershell -ExecutionPolicy Bypass -File install.ps1

cargo install wasm-pack
wasm-pack build --target bundler -- --features wasm --no-default-features
# output: pkg/  (JS glue + .wasm + TypeScript types)

From source

cargo install syntext

Benchmarks

Search latency across five real-world repositories (v1.0, macOS, Apple Silicon).

Average speedup across five presets: 20.1x versus rg. Search time excludes index build time.

See docs/BENCHMARKS.md for methodology, index build times, query discipline, and historical runs.

Usage

# Build the index (run once per repo, then only after large changes)
# Index is stored in .syntext/ at the repo root (nearest .git ancestor).
# Not run automatically -- you must run this before the first search.
st index
st index --stats                    # show file count and index size after build

# Override where the index is stored or which root to index
st --repo-root /path/to/repo index
st --index-dir /tmp/my-index index

# After editing files, sync the index incrementally (faster than full rebuild)
st update

# Search the whole repo (index must exist)
st "fn parse_query"                 # regex
st -F "parse_query("                # literal (metacharacters stay literal)
st -i "parsequery"                  # case-insensitive
st -x "TODO"                        # whole-line match
st -n "impl.*Iterator"              # force line numbers

# Restrict search scope with positional paths
st "needle" src/                    # search one directory
st "needle" src/lib.rs              # search one file
st "needle" src/lib.rs tests/       # search multiple files/directories

# Additional filters and output modes
st -t rs "impl.*Iterator"           # restrict to Rust files
st -g "src/" "TODO"                 # restrict by glob
st -c "parse_query" src/lib.rs      # count matches in one file
st -l "parse_query"                 # print matching file paths
st --json "TODO"                    # NDJSON output for tooling

# Status
st status

Notes:

• Search is the default command, there is no st search subcommand.
• Like ripgrep, file names are shown by default when searching a directory, the whole repo, or multiple positional paths.
• Like ripgrep, line numbers are off by default when stdout is not a TTY. Use -n to force them on.

Agent harness install

st can install RTK-style agent harness integrations. Programmatic hooks rewrite safe agent shell searches from rg or grep to st only when a .syntext/ index exists. Human shells, scripts, pipes, CI, and unsupported search forms are left alone. Hooks never run st index or st update automatically.

Quick installs:

# Claude Code project instructions only
st init

# Claude Code global Bash hook plus Grep blocker
st init -g

# RTK-style agent selectors
st init -g --agent cursor
st init -g --gemini
st init --copilot        # project hook; `st init -g --copilot` is also accepted
st init --codex          # project rules
st init -g --codex       # global Codex rules

Explicit install, show, and uninstall commands are also available:

st agent install claude --global
st agent show claude --global
st agent uninstall claude --global

Supported harnesses:

Each install is idempotent, preserves unrelated settings, writes a timestamped backup before editing an existing file, and only removes syntext-owned entries on uninstall.

Architecture

Query -> Router -> [Literal | Indexed Regex | Full Scan]
                        |
                   Gram extraction
                        |
                   Posting list intersection (smallest-first)
                        |
                   Candidate file IDs
                        |
                   Verifier (memchr or regex against file content)
                        |
                   Results

Three index components:

• Content index: sparse n-gram posting lists. Trigram augmentation ensures no false negatives for token-aligned queries.
• Path index: Roaring bitmap component sets for path/type filtering.
• Symbol index (optional): Tree-sitter extraction into SQLite.

Segments are immutable single-file mmap structures (SNTX format). Updates go through an in-memory overlay with atomic batch commit via ArcSwap.

See docs/ARCHITECTURE.md for the full quantitative analysis: selectivity math, index size estimates, posting list encoding tradeoffs.

WASM

The wasm Cargo feature compiles syntext to a fully in-memory index with no filesystem access. See the releases page for prebuilt syntext-wasm-<version>.tar.gz, or build from source:

wasm-pack build --target bundler -- --features wasm --no-default-features
# output: pkg/  (JS glue + .wasm + TypeScript types)

Project status

All phases complete (v1.1). Core st index && st "pattern" workflow validated against ripgrep. Symbol search available behind --features symbols.

Known limitations

1. Crash recovery: Overlay state is lost on unclean shutdown. Run st update or st index after a crash.
2. Non-aligned substring coverage: ~16% false-negative rate for queries that don't align with token boundaries. Token-aligned queries (identifiers, keywords) have 0% false negatives.
3. Network filesystems: Index directory must be on local filesystem. NFS/SMB behavior is undefined.
4. Case-insensitive overhead: ~15-20% more candidates due to lowercase normalization. Correct results guaranteed by verifier.
5. \r-only line endings: Treated as a single line (matches ripgrep behavior).
6. Symbol search accuracy: Tier 3 (heuristic) results are approximate. Tree-sitter failures fall back silently.
7. One root per index: Each index covers exactly one --repo-root. There is no way to merge multiple directories into a single index. To search across two repos, build and query each index separately with --repo-root. st update requires a git repo; non-git directories must be re-indexed with st index.

Design documents

• docs/ARCHITECTURE.md -- Quantitative analysis: selectivity math, index size estimates, posting list encoding, design tradeoffs
• specs/001-hybrid-code-search-index/spec.md -- Feature specification with user stories and acceptance criteria
• specs/001-hybrid-code-search-index/research.md -- 19-section architecture research covering every subsystem
• specs/001-hybrid-code-search-index/data-model.md -- Entity definitions and relationships
• specs/001-hybrid-code-search-index/contracts/ -- Library API, CLI, and segment format contracts
• specs/001-hybrid-code-search-index/tasks.md -- Implementation plan with dependency graph

License

MIT