Skip to content
dean0xPublic

About

The most intelligent context optimization engine for coding agents. Code-aware AST parsing across 17 languages. Command rewriting. Test, build, and git output compression. Token budget cascading. Built in Rust.

Topics

Resources

Contributing

Security policy

Stars

31 stars

Watchers

0 watching

Forks

Repository files navigation

Skim: The Most Intelligent Context Optimization Engine for Coding Agents

Code skimming. Command rewriting. Test, build, and git output compression. Codebase heatmap. Token budget cascading. 18 languages. 14ms for 3,000 lines. Built in Rust.

Other tools filter terminal noise. Skim understands your code. It parses ASTs across 18 languages, strips implementation while preserving architecture, then optimizes every other type of context your agent consumes: test output, build errors, git diffs, raw commands, and codebase heatmaps. 14ms for 3,000 lines. 48x faster on cache hits.

Website CI Crates.io npm Downloads License: MIT

Why Skim?

Context capacity is not the bottleneck. Attention is. Every token you send to an LLM dilutes its focus. Research consistently shows attention dilution in long contexts -- models lose track of critical details even within their window. More tokens means higher latency, degraded recall, and weaker reasoning. Past a threshold, adding context makes outputs worse. While other tools stop at filtering command output, Skim parses your actual code structure and optimizes the full spectrum of agent context: code, test output, build errors, git diffs, and commands. Deeper, broader, and smarter than anything else available.

Take a typical 80-file TypeScript project: 63,000 tokens. That contains maybe 5,000 tokens of actual signal. The rest is implementation noise the model doesn't need for architectural reasoning.

80% of the time, the model doesn't need implementation details. It doesn't care how you loop through users or validate emails. It needs to understand what your code does and how pieces connect.

That's where Skim comes in.

Mode Tokens Reduction Use Case
Full 63,198 0% Original source code
Structure 25,119 60.3% Understanding architecture
Signatures 7,328 88.4% API documentation
Types 5,181 91.8% Type system analysis

For example:

// Before: Full implementation (100 tokens)
export function processUser(user: User): Result {
    const validated = validateUser(user);
    if (!validated) throw new Error("Invalid");
    const normalized = normalizeData(user);
    return await saveToDatabase(normalized);
}

// After: Structure only (12 tokens)
export function processUser(user: User): Result { /* ... */ }

One command. 60-90% smaller. Your 63,000-token codebase? Now 5,000 tokens. Fits comfortably in a single prompt with room for your question.

That same 80-file project that wouldn't fit? Now you can ask: "Explain the entire authentication flow" or "How do these services interact?" β€” and the AI actually has enough context to answer.

Features

Code Skimming (the original, still unmatched)

  • 18 languages including TypeScript, JavaScript, Python, Rust, Go, Java, C, C++, C#, Ruby, SQL, Kotlin, Swift, Bash, Markdown, JSON, YAML, TOML
  • 6 transformation modes from full to minimal to pseudo to structure to signatures to types (15-95% reduction)
  • 14.6ms for 3,000-line files. 48x faster on cache hits
  • Token budget cascading that automatically selects the most aggressive mode fitting your budget
  • Parallel processing with multi-file globs via rayon

Command Rewriting (skim init)

  • PreToolUse hook rewrites ls, grep, gh, cargo test, vitest, git diff into skim equivalents
  • File reads (cat, head, tail on code files) are rewritten into direct skim reads (e.g. cat file.ts β†’ skim file.ts --mode=pseudo); output is a structured view, not raw bytes β€” skim emits a one-line stderr notice whenever the served view differs from raw file contents. head -20 file.ts becomes skim file.ts --mode=full --max-lines 20 (verbatim lines; when the file is longer, one slot goes to the elision marker, so the bound is N lines total β€” with one documented exception at N=1, which emits 1 content line plus the marker), and tail -5 file.ts likewise with --last-lines 5; bare head/tail default to 10 lines
  • Two-layer rule system with declarative prefix-swap and custom argument handlers
  • One command installs the hook for automatic, invisible context savings
  • Round-trip safe: commands with newlines, heredocs, or command substitution are never rewritten, and neither is a command whose target binary skim cannot resolve on PATH (a shell function or alias is invisible to a PATH probe by construction, so the real command runs untouched); piped commands are refused too, with one exception β€” <cmd> | cat (bare cat, sole consumer, no redirects) rewrites its source, because bare cat renders a stream for a reader rather than consuming its bytes
  • npx tsc, npx eslint, npx prettier, npx biome and npx oxlint are rewritten only when the tool is in node_modules/.bin of the current folder or of a parent folder, up to the repository root. Skim then runs npx --no-install <tool>, so the project's own copy runs. The rewritten command has --via-npx after the tool words, for example skim prettier --write --via-npx src. When the tool is not in the project, the npx command runs unchanged
  • Dev installs (skim init --dev): while working on skim itself, the hook's commit pin goes stale the moment you rebuild at a new commit, which forces a full reinstall on the next skim init and makes skim doctor exit 1 for being behind HEAD β€” neither of which rebuilding can fix, because HEAD keeps moving. --dev marks the installed hook as dev-pinned: the real commit is still recorded, and only the commit-staleness check is waived, so repeat installs take the already-up-to-date fast path and doctor stops failing. Version and binary-path checks are unchanged, and the waiver applies only to a hook whose SHA-256 manifest verifies. Dev mode is a property of the command, not stored state β€” running skim init without --dev reverts the hook to a strict install and reports dev-pinned -> pinned, so there is no --undev; skim init --dev --force re-stamps one in place
  • PATH wrappers (skim init --wrappers) now apply a force-raw sidecar marker (set by the PreToolUse hook when it identifies a pipeline shape where compression would cause byte loss) and an fstat/isatty gate (which serves raw bytes when stdout is a regular file, socket, or non-terminal character device). Together these partially close the compression-into-piped-consumer window (#319). Two holes remain (#514): (1) a same-tool concurrent command can clear a live marker, and (2) when no PreToolUse hook fires at all the wrapper falls back to fstat-only behaviour with no pipeline-shape awareness. Use SKIM_PASSTHROUGH=1 when byte-exact output is required in those cases
  • A link or copy of skim that has a tool name and sits outside the wrappers folder finds the real tool behind it. If skim finds only itself, it prints one error line and exits with status 126. It does not start itself again. When a program that skim started reports this error, skim starts no other program. It prints the error line once and exits with status 126, so the number of starts is in proportion to the depth of the chain

After-Run Compression (Claude Code auto mode)

In auto permission mode, Claude Code runs a command without the PreToolUse rewrite. skim init also installs a PostToolUse hook, which compresses the output after the command ran.

  • Commands: cat of one code file, cargo test, pytest (also python -m pytest), eslint, ruff check, mypy, golangci-lint run, rubocop, oxlint, biome check/biome lint and swiftlint. Only a successful run is compressed.
  • Marker: skim adds one [skim] line to stderr, for example [skim] compressed view of 'pytest': parsed summary, 494 B β†’ 30 B β€” SKIM_PASSTHROUGH=1 for full output. A cat view shows [skim] transformed view (cat β†’ skim --mode=pseudo): … instead.
  • Raw output: run the command again with SKIM_PASSTHROUGH=1 in front.
  • Scope: the hook never runs a command, never reads a file and never returns a permission decision. It changes only the output text it receives.
  • Limits: a failed run stays raw, because Claude Code does not let a hook replace its output. Lint tools exit non-zero when they find problems, so only clean or warning-only lint runs can compress. Output of 29,000 characters or more, compound commands, pipes, redirects (including 2>&1) and env-prefixed commands stay raw. ruff check in its default full format stays raw.
  • Existing installs: run skim init again to add the hook entry. skim doctor notes a missing entry.

Test Output Compression

  • skim cargo test, skim pytest, skim vitest, skim jest, skim go test
  • Extracts failures, assertions, pass/fail counts while stripping noise
  • Three-tier degradation from structured parse to regex fallback to passthrough

Build Output Compression

  • skim cargo build, skim cargo clippy, skim make, skim tsc
  • Extracts errors, warnings, and summaries

Lint Output Compression

  • skim eslint, skim ruff, skim mypy, skim golangci-lint, skim prettier, skim rustfmt, skim biome, skim dprint, skim oxlint, skim black, skim gofmt
  • Extracts errors and warnings with severity grouping
  • Three-tier degradation from structured parse to regex fallback to passthrough

Package Manager Compression

  • skim npm, skim pnpm, skim pip, skim cargo audit
  • Extracts vulnerabilities, version conflicts, and dependency issues

Git Output Compression (skim git)

  • skim git diff -- AST-aware: renders hunk-scoped context (AST breadcrumb β€” prefixed ~, marking a declaration header skim pulled in from outside the hunk rather than a line git printed β€” plus the changed lines), bounded by a guardrail so output never exceeds the raw diff size; strips diff noise
    • --mode structure adds unchanged functions as signatures for context
    • --mode full shows entire files with change markers
    • Supports --staged, commit ranges (HEAD~3, main..feature)
    • Line numbers: one column, two coordinate spaces β€” the prefix says which. - numbers index the old file; + and space numbers index the new one (~ breadcrumb lines carry no number). So the column repeats, and runs backward across a -/+ boundary: that is the two axes interleaving, not a misnumbering. A second column is not emitted β€” it needs 76 B against a measured raw-diff margin of 68 B or less β€” and the @@ header is withheld on single-hunk files, so there the prefix is the only cue
  • skim git show -- two modes, only one of which compresses by default:
    • Commit mode (default, e.g. skim git show HEAD): strips commit header noise and renders the diff with the AST-aware pipeline
    • File-content mode (e.g. skim git show HEAD:src/main.rs): <rev>:<path> is git's blob-extraction syntax, so skim serves the blob's exact bytes β€” no transform. --mode=<m> opts into a transformed view and a lossy one announces itself on stderr (ADR-022). Caveat: skim reads the child's stdout as UTF-8 with lossy replacement, so a blob that is not valid UTF-8 arrives with U+FFFD substituted
    • Three-tier degradation (commit mode only): AST-aware render β†’ raw hunk render β†’ guardrail passthrough
  • Machine-contract flags serve git's own bytes verbatim, on every git subcommand skim compresses, ahead of and independent of the net-savings guard (ADR-022): --porcelain, --null/-z, --stat, --shortstat, --numstat, --name-only, --name-status, --raw, --check, --quiet/-q, --exit-code, --graph, --format, --pretty. So git log --stat reaches the reader, --porcelain output is byte-exact, and git push -q / git commit --quiet stay silent
  • Otherwise compresses git status and git log
  • All subcommands support --json for machine-readable output

Codebase Heatmap (skim heatmap)

  • Short ranked default view (30 lines or fewer): a header line with the repository name, the since..until dates, the commit count and the window, then the top 10 hotspots by commit count, the top 5 code-to-code coupling pairs and the top 5 fix-risk files above the repository base rate, each row with its number and support. A capped section ends with one exact N more (--detail <section>) line. The last line points to the per-file card (skim heatmap <file>)
  • --detail <section> shows 25 rows of one section (--top N raises it), then one exact N more (raise --top) line: hotspots, coupling, fix-risk, modules (crate, package or top-folder encapsulation) or bus-factor (one author with more than 80% of a file's commits). Module health and bus factor show only with --detail
  • --top N sets the rows of each list in every view: the default view, --detail, --insights, the cards and JSON
  • skim heatmap <file>... prints a "before you edit" card per file (at most 5): co-change partners with 3 or more shared commits (tests first), up to 3 fix commits from history (at most 5 at any --top), and owners when the file has 2 or more authors. --insights <file> prints the same card
  • Noise rules: fix risk and bus factor need 5 commits of a file, coupling needs 3 shared commits, modules need 5 commits; a fix-risk row needs the lower bound of its fix rate above the repository base rate; bus factor needs 2 or more authors in the window. Modules come from tracked Cargo.toml and package.json folders, else the top folder
  • Default excludes: lock files, build output, logs, vendored code, and at any depth CHANGELOG.md, *.snap, snapshots/, *.pb.go, *_pb2.py and generated/ folders (--no-exclude keeps them; a file marked "generated" only in a header line stays in)
  • Non-code paths: doc files (*.md, *.rst, *.txt and similar) and the top folders docs, doc, .github, .devflow, tests and test stay out of the hotspots, the modules and the insights. The N more line counts them, and JSON gives left_out; --include-docs keeps them. --diff, file arguments and a --path that is a non-code folder keep every path
  • Fix commits: a commit counts as a fix only when its subject says so: a [fix] tag, a fix: or fix(scope): type (also bug, hotfix, patch, revert), or one of these words first in the subject (Revert "..."). A word inside a description does not count.
  • --insights gives the top 20 CRITICAL/WARNING findings, ranked by severity, score, support and path, then one N more (raise --top) line
  • --json gives JSON version 2: one compact line, each array of rows capped like its section, with counts.<section>.{total,shown}; file_targets[] holds at most 20 targets, warnings[] has at most one line for each kind of warning, and excluded_patterns[] is the fixed pattern list
  • Configurable time windows: --window sprint (14d), --last 200, --since 30d, or full history; --path scopes the git log to a subdirectory, and --diff <base> gives the default view for the changed files

Code Search (skim search)

  • Scan engine β€” skim search NAME reads the files at query time. There is no index, no build step, no cache and no git hook. Each query lists or counts every matching line. Every bound is a named constant, and each bound that fires prints a marker with exact counts. skim search records no analytics row.
  • Labelled view β€” a bare identifier (skim search launchWorker) gives the definition first, with its line range. Each call follows, with the function that holds it. Imports, strings, comments, text and partial matches are counted by role (--all lists them, --full prints the text of long sections). When a name has two or more definitions, each definition is listed and a note gives the count. Calls are not split by definition. Roles come from syntax, not from types.
  • Word view β€” a quoted argument with several words (skim search "temporal staleness check") lists the files that hold all the words first, ranked by file kind (source, test, docs, changelog, generated) and then by count. It lists 10 files by default (--limit N lists more). If fewer than 3 files hold all the words, files with fewer words follow.
  • Flags in the rg style β€” -i, -S (smart case), -w, -F, -E (regex), -g GLOB and -t TYPE (repeat them; a leading ! leaves files out), -l (paths), -c (counts per file), -C N (context lines), plus --all, --full, --json, --limit N, --offset N and --root DIR. Short boolean flags can join (-iwF). The pattern is a literal and is case sensitive by default. The flag set is closed: an unknown flag exits 2, and skim search starts no other program. -l and -c print in path order.
  • Universe β€” every tracked text file, plus the untracked files that the ignore files allow. There is no language list and no file cap. A file that is binary, not UTF-8, above 5 MiB, unreadable, missing, a link, or outside the root is not read. It is counted with its reason in the last line of the output (total: …; N files searched, M left out).
  • Limit while a query runs β€” a query checks each folder before it reads the files in it. A folder that another process replaces with a link while a query runs can make the query read outside the root. rg and grep have the same limit. Do not change the work tree while a query runs.
  • Hidden and secret-bearing files β€” tracked hidden files and tracked files that may hold secrets (.env, key files) are searched and listed like other tracked files. Use -g '!NAME' to leave one out. A later release can change this default to leave such files out; the rule sits in one function (sensitive_admission) and one constant (SENSITIVE_PATH_POLICY) in crates/rskim/src/cmd/search/universe.rs. Untracked hidden files are not searched.
  • Root β€” without --root, the root is the repository that holds the current folder. The lookup stops at GIT_CEILING_DIRECTORIES and never takes the home folder or / as the root. It adopts a repository above the current folder only when you own its .git entry, as git does by default; otherwise it searches the current folder only and prints a note. When GIT_CEILING_DIRECTORIES has more than 64 entries, the lookup reads the first 64, does not go up, and prints a note with the number of entries dropped. Outside a repository, only the current folder is searched. A subfolder given with --root limits the search to that subtree; the same rules and notes apply to it, and its tracked hidden and ignored files are searched.
  • Output and exit codes β€” --json prints one compact object on one line (an error is {"error":"…"}). --limit N and --offset N page the files in rank order. Exit codes: 0 a line matched, 1 no match, 2 an error (an unknown or removed flag, a bad value, an invalid regex, a bad --root, a git index or repository state that cannot be read), and 141 on unix when the reader of stdout stops.
  • Old search data β€” skim 2.x built an index and installed git hooks. skim init and skim init --uninstall remove the old hook block and keep every other line of the hook files. skim init needs an installed agent and changes nothing without one; skim init --uninstall removes the block with no agent, but it removes skim completely. A hook that cannot be read gets its own note and the other hooks are still cleaned. skim --clear-cache removes the old index data under the cache folder, including the journal and temp files that a crash left, and prints a note with the count of every entry it leaves in place. It does not enter a folder that another user owns. skim doctor reports both as advisory lines, and says so when it cannot read a hook or the old index folder. A query never deletes a file.
  • Removed flags β€” each exits 2 with a message that names the replacement, and prints nothing on stdout. --ast: use ast-grep. --hot, --cold, --risky and --blast-radius: use skim heatmap. --phrase and --near: use -F 'exact words' or -E. --lang: use -t TYPE. -j: use --json. --weights, --build, --rebuild, --update, --stats, --install-hooks and --remove-hooks have no replacement, because there is no index.

LLM Proxy (skim proxy)

  • Resident loopback HTTP/1.1 reverse proxy that sits between your agent and the LLM API
  • Availability: prebuilt release binaries (GitHub Releases, npm, Homebrew) include skim proxy. Source builds opt in: cargo build -p rskim --features proxy or cargo install --path crates/rskim --features proxy. Default source builds are HTTP/TLS-free. Note: cargo install rskim (crates.io) does not currently include the proxy subcommand (tracked in #453).
  • Lossless-only egress guarantee β€” all compression on the proxy path is information-preserving:
    • JSON minification (structural whitespace removal; value-equivalent, dup-key-safe)
    • Log deduplication with Γ—N counts and timestamp min–max ranges (all unique content preserved)
    • Code blocks always pass through byte-identical (no tree-sitter rewriting on the egress path)
  • Two policy tiers: LosslessOnly (byte-exact passthrough for subscription/Bearer auth) and Default (lossless re-encoding allowed for API key auth)
  • Per-engine runtime certification gate β€” every engine must prove value-equivalence before the modified bytes leave the proxy; fail-open to byte-identical passthrough on any uncertainty
  • SKIM_PASSTHROUGH=1 substitutes the identity pipeline, so no proxy-side compression is applied (the proxy builds its own pipeline and does not go through the subcommand passthrough gate, whose exceptions are listed under Common options below); --port / --bind configure the listener

Intelligence

  • skim discover scans agent session history for optimization opportunities
  • skim learn detects CLI error-retry patterns and generates correction rules
  • Output guardrail ensures compressed output is never larger than the original

Installation

Try it (no install required)

npx rskim file.ts

Install globally (recommended for regular use)

# Via Homebrew (macOS/Linux)
# Homebrew 7 loads a formula from a third-party tap only after you trust it.
# An older Homebrew has no `brew trust` command. Skip that line there.
brew trust --formula dean0x/tap/skim
brew install dean0x/tap/skim

# Via npm
npm install -g rskim

# Via Cargo
cargo install rskim

Note: Use npx for trying it out. For regular use, install globally to avoid npx overhead (~100-500ms per invocation).

From Source

git clone https://github.com/dean0x/skim.git
cd skim
cargo build --release
# Binary at target/release/skim

Quick Start

# Try it with npx (no install)
npx rskim src/app.ts

# Or install globally for better performance
npm install -g rskim

# Extract structure from single file (auto-detects language)
skim src/app.ts

# Process entire directory recursively (auto-detects all languages)
skim src/

# Process current directory
skim .

# Process multiple files with glob patterns
skim 'src/**/*.ts'

# Process all TypeScript files with custom parallelism
skim '*.{js,ts}' --jobs 4

# Get only function signatures from multiple files
skim 'src/*.ts' --mode signatures --no-header

# Extract type definitions
skim src/types.ts --mode types

# Print the raw source lines of one declaration, not the whole file
skim src/app.ts --symbol parseConfig
skim src/lib.rs --symbol Point::norm   # only norm inside the impl of Point

# Extract markdown headers (H1-H3 for structure, H1-H6 for signatures/types)
skim README.md --mode structure

# Pipe to other tools
skim src/app.ts | bat -l typescript

# Read from stdin (--language recommended; shebang auto-detected, else lossless passthrough)
cat app.ts | skim - --language=typescript

# Override language detection for unusual file extensions
skim weird.inc --language=typescript

# Clear the parser cache and old search index data
skim --clear-cache

# Disable caching for pure transformation
skim file.ts --no-cache

# Show token reduction statistics
skim file.ts --show-stats

Usage

# Basic usage (auto-detects language)
skim file.ts                    # Single file
skim src/                       # Directory (recursive)
skim 'src/**/*.ts'             # Glob pattern

# With options
skim file.ts --mode signatures  # Different mode
skim src/ --jobs 8             # Parallel processing
skim - --language typescript   # Stdin (recommended; shebang auto-detected)

Common options:

  • -m, --mode - Transformation mode: structure (default), signatures, types, full, minimal, pseudo
  • -l, --language - Override auto-detection (recommended for stdin; else shebang is tried, then lossless passthrough)
  • -j, --jobs - Parallel processing threads (default: CPU cores)
  • -n, --line-numbers - Annotate output with original source line numbers (prefix {line}\t{content})
  • --max-lines N - Emit at most N lines total; when content is elided, one slot goes to the elision marker. One documented exception at N=1: 1 content line plus the marker (2 lines), because spending the only slot on the marker returns a view containing no code, and dropping the marker to fit is silent loss
  • --last-lines N - Emit at most N lines from the tail; when content is elided, one slot goes to the elision marker. The same N=1 exception applies β€” the carve-out is a property of the bound, not of the direction
  • --tokens N - Fit output within N tokens by escalating through modes, then line-truncating with a marker
  • --symbol NAME - Print the raw source lines of one named declaration of one file. Type::name or Type.name selects a declaration inside one container. If two or more declarations have the name, skim prints all of them, each after a header line. One stderr notice line states how many lines are not shown. --max-lines, --last-lines and --line-numbers apply to the selected lines. Exit code 1: no declaration has the name. Exit code 2: the command is not a symbol read (stdin, two or more files, a glob, a directory, an empty name, --tokens, or a --mode other than full). Exit code 3: Markdown, JSON, YAML, TOML, or a file with no detected language. Known limits: raw source only. Functions in C and C++ files are not found, because their name is inside the declarator. Attributes, doc comments and decorators above a declaration are not in the printed range. The structure mode still parses .tsx files with the TypeScript grammar. See the usage guide
  • --passthrough - Bypass compression and exec the real tool with raw argv (equivalent to SKIM_PASSTHROUGH=1). Not universal: env/printenv keep credential redaction regardless of passthrough state, and the multi-level dispatchers (cargo, dotnet, go, swift) still serve the compressed summary whenever stdin is not a TTY β€” which is every agent harness, CI job, and pipeline. Run the tool directly when you need raw bytes from a build tool
  • --debug - Enable raw-fallback diagnostic banners on stderr (loss-bearing elision markers are always emitted regardless)
  • --no-cache - Disable caching
  • --show-stats - Show token reduction stats
  • --disable-analytics - Disable analytics recording

πŸ“– Full Usage Guide β†’

Transformation Modes

Skim offers six modes with different levels of aggressiveness:

Mode Reduction What's Kept Use Case
Full 0% Everything (original source) Testing/comparison
Minimal 15-30% † All code, doc comments Light cleanup
Pseudo 30-50% † Logic flow, names, values, return types; parameter types preserved for TypeScript and Rust, as are its interface / type-literal member annotations LLM context with logic
Structure 60-80% ‑ Signatures, types, classes, imports Understanding architecture
Signatures 85-92% Only callable signatures API documentation
Types 90-95% Only type definitions Type system analysis

† Unverified targets, not measured figures. These two numbers were never derived from a measurement, and no CI gate defends them. Both modes have since been widened to preserve more content β€” module header comments are now kept in every language (#476), and Rust's pseudo mode strips only statement semicolons β€” declaration terminators and the array-length ; are preserved β€” plus non-doc comments; TypeScript now keeps its type-level member annotations too β€” so treat these two rows as aspirational until they are re-measured.

‑ Measured basis: 60.3%. Structure mode's only measured figure is the 60.3% in the reduction table at the top of this README and in the Real-World Token Reduction table under Performance β€” both on the same production TypeScript codebase. The range on this row is stated wide enough to contain it. No CI gate defends the range either way: the only reduction ratio any test asserts is > 0.30, on the JSON and YAML structure-mode fixtures.

skim file.ts --mode structure   # Default
skim file.ts --mode pseudo      # Pseudocode (preserves return types, parameter types for TS/Rust; strips decorators)
skim file.ts --mode signatures  # More aggressive
skim file.ts --mode types       # Most aggressive
skim file.ts --mode full        # No transformation

Note on JSON/YAML/TOML files: JSON, YAML, and TOML always use structure extraction regardless of mode. Since they are data (not code), there are no "signatures" or "types" to extractβ€”only structure. All modes produce identical output for these file types.

πŸ“– Detailed Mode Guide β†’

Supported Languages

Language Status Extensions Notes
TypeScript βœ… .ts, .tsx Excellent grammar
JavaScript βœ… .js, .jsx Full ES2024 support
Python βœ… .py, .pyi Complete coverage
Rust βœ… .rs Up-to-date grammar
Go βœ… .go Stable
Java βœ… .java Good coverage
C βœ… .c, .h Full C11 support
C++ βœ… .cpp, .hpp, .cc, .hh, .cxx, .hxx C++20 support
Markdown βœ… .md, .markdown Header extraction
JSON βœ… .json Structure extraction (serde)
YAML βœ… .yaml, .yml Multi-document support (serde)
TOML βœ… .toml Structure extraction (toml)
C# βœ… .cs Full grammar, structs/interfaces
Ruby βœ… .rb Classes, modules, methods
SQL βœ… .sql DDL/DML via tree-sitter-sequel
Kotlin βœ… .kt, .kts Data classes, coroutines, sealed classes
Swift βœ… .swift Protocols, generics, SwiftUI structs
Bash βœ… .sh, .bash Functions + shebang auto-detect; also #!/bin/sh, zsh, ksh

Examples

TypeScript

// Input
class UserService {
    async findUser(id: string): Promise<User> {
        const user = await db.users.findOne({ id });
        if (!user) throw new NotFoundError();
        return user;
    }
}

// Output (structure mode)
class UserService {
    async findUser(id: string): Promise<User> { /* ... */ }
}

Python

# Input
def process_data(items: List[Item]) -> Dict[str, Any]:
    """Process items and return statistics"""
    results = {}
    for item in items:
        results[item.id] = calculate_metrics(item)
    return results

# Output (structure mode)
def process_data(items: List[Item]) -> Dict[str, Any]: { /* ... */ }

JSON

// Input
{
  "user": {
    "profile": {
      "name": "Jane Smith",
      "age": 28,
      "tags": ["admin", "verified"]
    },
    "settings": {
      "theme": "dark",
      "notifications": true
    }
  },
  "items": [
    {"id": 1, "price": 100},
    {"id": 2, "price": 200}
  ]
}

// Output (structure mode)
{
  user: {
    profile: {
      name,
      age,
      tags
    },
    settings: {
      theme,
      notifications
    }
  },
  items: {
    id,
    price
  }
}

YAML (Multi-Document)

# Input (Kubernetes manifests)
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  database_url: postgres://localhost:5432
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 3

# Output (structure mode)
apiVersion
kind
metadata:
  name
data:
  database_url
---
apiVersion
kind
metadata:
  name
spec:
  replicas

πŸ“– More Examples (All Languages) β†’

Use Cases

LLM Context Optimization

Reduce codebase size by 60-90% to fit in LLM context windows:

skim src/ --no-header | llm "Analyze this codebase"
skim src/app.ts | llm "Review this architecture"

API Documentation

Extract function signatures for documentation:

skim src/ --mode signatures > api-docs.txt
skim 'lib/**/*.py' --mode signatures > python-api.txt

Type System Analysis

Focus on type definitions and interfaces:

skim src/ --mode types --no-header
skim 'src/**/*.ts' --mode types

Code Navigation

Quick overview without implementation details:

skim large-file.py | less
skim src/auth/ | less

πŸ“– 10 Detailed Use Cases β†’

Caching

Caching is enabled by default for 40-50x faster repeated processing.

Performance Impact

Scenario Time Speedup
First run (no cache) 244ms 1.0x
Second run (cached) 5ms 48.8x faster!

Cache Management

ls ~/.cache/skim/           # View cache
skim --clear-cache          # Clear the parser cache and old search index data
skim file.ts --no-cache     # Disable for one run

How it works:

  • Cache key: SHA256(file path + mtime + mode)
  • Automatic invalidation when files change
  • Platform-specific cache directory

When to disable caching:

  • One-time LLM transformations
  • Stdin processing
  • Disk-constrained environments

πŸ“– Caching Internals β†’

Token Counting

See exactly how much context you're saving with --show-stats:

skim file.ts --show-stats
# [skim] 1,000 tokens β†’ 200 tokens (80.0% reduction)

skim 'src/**/*.ts' --show-stats
# [skim] 15,000 tokens β†’ 3,000 tokens (80.0% reduction) across 50 file(s)

Uses OpenAI's tiktoken (cl100k_base for GPT-3.5/GPT-4). Output to stderr for clean piping.

Analytics

Skim automatically tracks token savings from every invocation in a local SQLite database (~/.cache/skim/analytics.db). View your savings with the stats subcommand:

skim stats                       # All-time dashboard (cost estimates always shown)
skim stats --since 7d            # Last 7 days
skim stats --format json         # Machine-readable output
skim stats --verbose             # Include parse quality details
skim stats --clear               # Reset analytics data

Environment variables:

Variable Description
SKIM_DEBUG Set to 1 (or use --debug) to enable raw-fallback diagnostic banners on stderr; loss-bearing elision markers are always emitted regardless of this setting (ADR-011)
SKIM_DISABLE_ANALYTICS Set to 1, true, or yes to disable recording
SKIM_INPUT_COST_PER_MTOK Override $/MTok for cost estimates (default: 3.0)
SKIM_ANALYTICS_DB Override analytics database path
SKIM_DEPTH Set by skim. Do not set it yourself. It counts the skim processes in one chain of starts. When the count is more than 32, skim prints one error line and exits with status 126

Analytics recording is fire-and-forget (non-blocking) and does not affect command performance. Data is automatically pruned after 90 days.

Security

Skim includes built-in DoS protections:

  • Max recursion depth: 500 levels
  • Max input size: 50MB per file
  • Max AST nodes: 100,000 nodes
  • Path traversal protection: Rejects malicious paths
  • No code execution: Only parses, never runs code

πŸ“– Security Details & Best Practices β†’ πŸ”’ Vulnerability Disclosure β†’

Architecture

Skim uses a clean, streaming architecture:

Language Detection β†’ tree-sitter Parser β†’ Transformation β†’ Streaming Output

Design principles:

  • Streaming-first: Output to stdout, no intermediate files
  • Zero-copy: Uses &str slices to minimize allocations
  • Error-tolerant: Handles incomplete/broken code gracefully
  • Type-safe: Explicit error handling, no panics

πŸ“– Architecture Deep Dive β†’

Performance

Target: <50ms for 1000-line files βœ… Exceeded (14.6ms for 3000-line files)

Benchmark Results

File Size Lines Time Speed
Small 300 1.3ms 4.3Β΅s/line
Medium 1500 6.4ms 4.3Β΅s/line
Large 3000 14.6ms 4.9Β΅s/line

Real-World Token Reduction

Production TypeScript Codebase:

Mode Tokens Reduction LLM Context Multiplier
Full 63,198 0% 1.0x
Structure 25,119 60.3% 2.5x more code
Signatures 7,328 88.4% 8.6x more code
Types 5,181 91.8% 12.2x more code

πŸ“– Full Performance Benchmarks β†’

Development

Quick Start

# Build and test
cargo build --release
cargo test --all-features

# Lint
cargo clippy -- -D warnings
cargo fmt -- --check

# Benchmark
cargo bench

Adding New Languages

~30 minutes per language:

  1. Add tree-sitter grammar to Cargo.toml
  2. Update Language enum in src/types.rs
  3. Add file extension mapping
  4. Add test fixtures
  5. Run tests

πŸ“– Development Guide β†’

Project Status

Current: v3.0.0 β€” Stable

βœ… Core β€” Code Reading (18 languages):

  • TypeScript/JavaScript/Python/Rust/Go/Java/C/C++/C#/Ruby/SQL/Kotlin/Swift/Bash/Markdown/JSON/YAML/TOML
  • 6 transformation modes: structure, signatures, types, minimal, pseudo, full
  • Token budget (--tokens N), max lines (--max-lines N), last lines (--last-lines N)
  • Multi-file glob support, parallel processing, caching (40-50x speedup)

βœ… Command Output Compression:

  • Test runners: cargo test, pytest, vitest/jest, go test
  • Build tools: cargo build, cargo clippy, make, tsc
  • Git: status, diff, log
  • File tools: find, ls, tree, grep, rg
  • Log: JSON structured + plaintext dedup, debug filtering, stack trace collapsing
  • Infra: gh, aws, curl, dig, nslookup, wget, docker, kubectl, terraform
  • Database: psql, mysql, sqlite3
  • Three-tier degradation: Structured β†’ Regex β†’ Passthrough

βœ… Agent Integration:

  • skim init β€” hook installation for Claude Code, Cursor, Codex, Gemini, Copilot, Crush; optional consent-gated permission seeding (--permissions); dev-pinned installs for skim development (--dev)
  • skim doctor β€” install health check: which skim on $PATH wins, per-agent hook state (version, commit, binary pin, tamper verdict), wrapper targets, and staleness vs. repo HEAD. Exit 0 healthy / 1 on drift, so it works as a CI pre-flight. A dev-pinned hook renders as ⚠ … dev-pinned (binary commit <sha>) β€” never βœ“ β€” printing both the commit the install froze and the commit the running binary was built from, so skim doctor | grep dev-pinned is a one-line guard against shipping a dev install. It also prints an advisory note (no change to the exit code) for each PostToolUse hook of another tool that can select Bash, found in either scope or in settings.local.json; Claude Code gives the model one PostToolUse result per call, so such a hook can cancel skim's output. Plugin hooks are not checked.
  • skim rewrite β€” command rewriting engine with --hook mode
  • MCP server mode for agent-native workflows

βœ… Analytics & Intelligence:

  • skim heatmap β€” git history risk analysis: a short ranked view (churn, coupling, fix risk), --detail sections and per-file "before you edit" cards
  • skim stats β€” persistent SQLite dashboard with cost estimation
  • skim discover β€” missed optimization finder across agent sessions
  • skim learn β€” CLI error pattern detection and correction rules

βœ… Distribution:

  • cargo (cargo install rskim), npm (npx rskim), Homebrew (brew trust --formula dean0x/tap/skim, then brew install dean0x/tap/skim; an older Homebrew without brew trust skips that step)
  • 14.6ms performance (3x under target)

See CHANGELOG.md for version history.

Documentation

Comprehensive guides for all aspects of Skim:

  • πŸ“– Usage Guide - Complete CLI reference and options
  • 🎯 Transformation Modes - Detailed mode comparison and examples
  • πŸ’‘ Examples - Language-specific transformation examples
  • πŸš€ Use Cases - 10 practical scenarios with commands
  • ⚑ Caching - Caching internals and best practices
  • πŸ”’ Security - DoS protections and security best practices
  • πŸ—οΈ Architecture - System design and technical details
  • ⏱️ Performance - Benchmarks and optimization guide
  • πŸ› οΈ Development - Contributing and adding languages

Part of the AI Development Stack

Tool Role What It Does
Skim Context Optimization Code-aware AST parsing across 18 languages, command rewriting, test/build/git output compression
DevFlow Quality Orchestration 18 parallel reviewers, working memory, self-learning, composable plugin system
Autobeat Agent Orchestration Autonomous orchestration. Eval loops, multi-agent pipelines, DAG dependencies, crash-proof persistence

Skim optimizes every byte of context. DevFlow enforces production-grade quality. Autobeat scales execution across agents. No other stack covers all three.

Contributing

Contributions welcome! Please:

  1. Check issues for existing work
  2. Open an issue to discuss major changes
  3. Follow existing code style (cargo fmt, cargo clippy)
  4. Add tests for new features
  5. Update documentation

πŸ“– See Development Guide for detailed instructions

Project Structure

skim/
β”œβ”€β”€ crates/
β”‚   β”œβ”€β”€ rskim-core/    # Core library (language-agnostic)
β”‚   └── rskim/         # CLI binary (I/O layer)
β”œβ”€β”€ tests/fixtures/    # Test files for each language
└── benches/           # Performance benchmarks (planned)

License

MIT License - see LICENSE for details.

Acknowledgments

  • tree-sitter - Fast, incremental parsing library
  • clap - Command-line argument parsing
  • ripgrep, bat, fd - Inspiration for Rust CLI design

Links


Built with ❀️ in Rust

About

The most intelligent context optimization engine for coding agents. Code-aware AST parsing across 17 languages. Command rewriting. Test, build, and git output compression. Token budget cascading. Built in Rust.

Topics

Resources

Contributing

Security policy

Stars

31 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages