ken is a local context-rank index for coding agents such as Claude Code and Codex CLI. It indexes each project into a local SQLite database, watches for changes, and gives the assistant a ranked <context-rank> block before each prompt so it can start from the most relevant files instead of exploring from scratch.

Everything runs locally: project metadata, embeddings, the daemon, and the SQLite index stay on your machine.

ken is not only a code embedding index. It builds several local signals and ranks files by combining them:

• Structural code index: each file is hashed and parsed when possible. The AST parser extracts symbols, line ranges, imports, module docstrings, and symbol docstrings. Files that cannot be parsed are still tracked by path, mtime, and lightweight text intent when useful.
• Semantic index: ken embeds files, symbols, and explicit purpose text. File embeddings are based on language, filename, and top symbol names. Symbol embeddings include kind, name, and docstring. Docstrings are also stored as separate intent sources, so a prompt can find code by what it is for, not only by what it is named.
• Live task memory: hooks record local interactions in the project SQLite DB: prompts, reads, edits, writes, dismissals, and the files touched in each turn. Recent interactions are weighted more heavily, and useful patterns such as read-then-edit score higher than repeated reads with no follow-through.
• Predictive memory: at the end of a session, ken snapshots which files were productive for that task. Later, when a new prompt is semantically similar to a previous one, those files get a predictive boost.
• Relationship boosts: after the main channels rank candidates, ken applies conservative boosts for things like recently modified files, symbols pointing to their containing file, source/test counterparts, imports, and files that often co-occurred in past similar sessions.

The result is closer to a local project memory than a plain search tool. Raw text search can find exact strings. Embeddings can find semantic neighbors. AST indexing can find named symbols. ken combines all of that with how the assistant actually used the project over time. That is why results are often better after a few real sessions: the database accumulates project-specific evidence about which files matter for which kinds of tasks.

All of these signals stay local in .ken/ken.db. They are not sent to a any server.

ken is published on PyPI as ken-rank. The distribution name is ken-rank, but it installs the ken command. Install it with pipx (recommended, isolates the tool) or uv:

pipx install ken-rank
# or
uv tool install ken-rank
# or
pip install ken-rank

Verify the install:

ken --version

To install from a local clone (for development or an unreleased build):

./install.sh

The installer uses uv to install the ken command into your user-local tool directory. If uv is not present, the script bootstraps it with Astral's official installer unless you pass --no-bootstrap-uv. You can also install the checkout directly with uv:

uv tool install --editable . --force --reinstall --refresh

Run this once from the project you want ken to index:

ken install .

By default, ken detects the assistant setup you use and wires itself into the supported local agent config. The install creates .ken/, adds it to .gitignore, installs hooks/MCP config where applicable, and performs the initial structural code index.

Embeddings are lazy by default. That means ken install . does not eagerly embed the whole repository; the daemon warms embeddings as they are needed. This keeps install fast and avoids doing expensive work before the project needs it.

ken also gets better with use. The first run starts from the project index, names, symbols, docstrings, and any available embeddings. As you and the assistant read files, edit code, dismiss weak context, and save findings, ken records those interactions as local ranking signals. Results should improve after real sessions because the system learns which files were useful for similar work in this project.

Use --claude when you specifically want Claude Code wiring:

ken install --claude .

Use --codex when you specifically want Codex CLI wiring:

ken install --codex .

You can pass both when a project uses both assistants:

ken install --claude --codex .

For better first-run semantic ranking, especially on projects where initial context quality matters, force the initial embedding pass:

ken install --embed .

This is recommended when you can afford the extra install time. The cost depends on repository size. On very large projects, cap the eager pass while still structurally indexing the whole repo:

ken install --embed --embed-limit 5000 .

You can combine assistant selection and eager embeddings:

ken install --codex --embed .
ken install --claude --embed .

From a ken checkout, install.sh can install the CLI and then run ken install for a project:

./install.sh --project /path/to/my-project
./install.sh --project /path/to/my-project --codex --embed

New projects use ken/static-qwen3-r512-v2 (1024-dim), a fitted token table that ships inside the wheel. It is not a neural network. A text is tokenised, one vector per token is looked up and added up, and the sum goes through a single matrix:

ids  = tokenize(text)          # Qwen3's own tokenizer, bundled with the table
vec  = normalise( Σ table[ids] · projection )

That is ~1000 arithmetic operations per token instead of the 880 million a transformer spends, so it encodes about 10 000 texts a second on one CPU thread — three orders of magnitude faster than running a model — with no GPU, no ONNX, no torch, and nothing to download on first use. The table is 23 MB and the arithmetic is numpy.

It is fitted, not designed: a ridge regression against Qwen/Qwen3-Embedding-0.6B over 964 475 texts of the exact shapes ken stores, parsed with ken's own parsers from the-stack-smol and balanced across sixteen languages.

How good is it? On 4 351 retrieval queries over code held out of its training — one thousand files per language, split off before a single training text was built — it reaches 97% of its teacher's recall@10:

Per language, table against teacher: typescript 0.92/0.93 · go 0.85/0.87 · rust 0.82/0.87 · csharp 0.79/0.79 · php 0.78/0.76 · java 0.78/0.82 · powershell 0.84/0.87 · python 0.74/0.80 · ruby 0.72/0.73 · bash 0.70/0.66 · c++ 0.66/0.65 · sql 0.25/0.31 · css 0.22/0.23 · html 0.20/0.14. CSS, HTML and SQL are low for both — those files carry few docstrings, so the task itself is hard there, not the table.

And on ranking, which is what ken actually does with it: over 637 real prompts from two projects ranking 2 627 files, asking whether each encoder surfaces the teacher's own top pick in its top ten —

+0.517 in English (95% CI [+0.461, +0.574]); in Spanish the difference is indistinguishable from zero ([−0.091, +0.046]). The table is fitted on code-shaped text, so that is where it is strongest — but it does not cost you anything on conversational prompts in other languages either.

sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 (384-dim) remains available and is the automatic fallback if an install has no table.

A project's model is pinned to its index. Upgrading ken never re-encodes an existing project behind your back — cosine similarity across two models is meaningless, so switching always requires a deliberate re-encode:

# Re-encode every stored embedding with a new model (no re-index; uses the
# source text ken already keeps). Records the model so it stays pinned.
ken reembed --model ken/static-qwen3-r512-v2

Any fastembed model works out of the box too.

To choose the default for future projects (leaving existing ones untouched), set a user-level default:

ken default-model                     # show the current default for new projects
ken default-model BAAI/bge-m3         # every project installed from now on uses this
ken default-model --clear             # back to ken's built-in default

This only affects projects created afterwards; switch an existing one with ken reembed --model <name>.

The default needs none. A table lookup and a sum have nothing to offload — the static table runs on the CPU at ~10 000 texts a second and there is no GPU path to want. This section is for the model backends.

The embedder auto-detects a GPU and uses it, falling back to CPU when none is usable — no configuration. What counts as a GPU depends on the backend:

Override detection with KEN_EMBED_DEVICE=auto|cpu|gpu|cuda|mps (and KEN_EMBED_DEVICE_ID=0 to pick a CUDA index). auto and gpu prefer CUDA, then MPS, then CPU; naming one accelerator restricts the choice to that one, so cuda on a Mac lands on the CPU rather than quietly substituting the other GPU.

On Apple Silicon, only the torch backend reaches the GPU — ONNX Runtime has no MPS provider, and [gpu] is not installable on macOS at all (onnxruntime-gpu ships no Mac wheels), so the default models stay on the CPU where they are already fast. [torch] alone is enough: the stock PyPI torch wheel for macOS arm64 carries MPS.

MPS is used more defensively than CUDA, because it fails in ways CUDA doesn't:

• It can lie quietly. A batch can come back as NaN, or as rows the GPU never filled in — which survive normalisation as all-zero vectors that are perfectly finite and score 0.0 against every query forever. ken checks that each batch is finite and unit-norm; if it isn't, or the GPU raises, it re-encodes on the CPU and stays there for the rest of the process. One reload, versus unusable vectors in ken.db that nobody notices until retrieval has quietly degraded.
• It won't tell you it's out of memory. torch's MPS allocator ceiling is set from a fraction of RAM and lands above the RAM that exists, so instead of an out-of-memory error macOS just swaps. ken caps it (KEN_MPS_MEMORY_FRACTION, default 0.7) so an oversized batch raises and hits the fallback above.
• torch 2.9+ is required to use the GPU (macOS 14+). Below that, a PyTorch bug could return embeddings that are finite, unit-norm and wrong — the one failure no runtime check can catch — so ken refuses MPS on older torch and says so in the log rather than trusting an index it can't verify. The [torch] extra pins this for you on Apple Silicon.

Is the GPU worth it? For most people, not much — and that's fine. The device only affects how fast a query is embedded; the rest of a rank (loading the stored vectors, the cosine sweep, the lexical/import channels) is CPU work either way. With the default table a query embeds in well under a millisecond, and with a fastembed model in ~30–40 ms on CPU, so the GPU barely moves the needle for inline ranking. Even with the heavy Qwen/Qwen3-Embedding-0.6B on a large repo, a full rank was ~1.1 s of which the query embed is ~250 ms on CPU vs ~50 ms on GPU (measured on CUDA — Apple Silicon is unmeasured) — the GPU saves ~200 ms of a second-plus that's dominated by device-independent work. So CPU-only is perfectly usable, including with Qwen3.

Where the GPU genuinely pays off: bulk work — a full ken reembed or the first index of a big repo re-encodes thousands of texts at once, and there the GPU is several times faster. And if you simply want to shave every last millisecond off inline ranking, turn it on. Otherwise, don't sweat it.

Some of the best open-source embedding models are not shipped by fastembed. The optional torch extra adds a sentence-transformers backend so you can use them — most notably Qwen/Qwen3-Embedding-0.6B, the top scorer in ken's own retrieval benchmark, and BAAI/bge-m3:

pip install 'ken-rank[torch]'
ken reembed --model Qwen/Qwen3-Embedding-0.6B

ken selects the backend automatically from the model name — just point ken reembed --model at it.

How good is Qwen3? On ken's benchmark (100 labeled prompt→file queries over a real project with Spanish prompts and English code), Qwen3-0.6B was clearly the strongest model tested:

Those numbers come from a different, older benchmark than the ones at the top of this section, so read them against each other and not across — what they establish is the ordering among model backends. The default table is distilled from the winner of this table and reaches 97% of it on held-out code, so [torch] is worth installing when you want the last few points of recall and can afford 1.2 GB and ~250 ms per query, and is otherwise unnecessary.

That is roughly +40% Recall@5 and +87% MRR over the old default, and it more than doubles retrieval quality on non-English prompts.

The costs. Qwen3 is not free to run:

• Heavier install. The [torch] extra pulls PyTorch + sentence-transformers — hundreds of MB, versus the small ONNX-only default. The model itself is ~1.2 GB.
• Bigger index. It is 1024-dimensional; stored vectors are ~2.7× the size of the 384-dim default, so ken.db grows accordingly.
• Slower on CPU. It shines on a GPU — the torch backend picks up CUDA or Apple Silicon's MPS on its own (the [gpu] extra is for the fastembed path, not this one). On CPU the per-prompt embedding is noticeably slower than the fastembed default, which matters because the daemon embeds inline on every prompt. On a Mac, [torch] alone is enough: the stock PyPI torch wheel for macOS arm64 ships MPS support, no extra index or extra to install.

So it is worth it when you have a GPU (or don't mind the CPU latency) and want the best retrieval; otherwise the lightweight multilingual default is the better trade-off. The fastembed default always stays the no-torch path.

ken works through hooks automatically, but assistants behave better when your project instructions tell them when to reach for ken and, just as importantly, to write back what they learn. The MCP server already describes all 30 tools to the assistant, so the block below deliberately carries only what a tool description cannot: where to start, when to stop, and what to record. Add it to the agent instruction file for the tool you use: AGENTS.md for Codex, CLAUDE.md for Claude Code, or both.

## Code intelligence: ken

**When a prompt arrives with a `<context-rank>` block**, that is ken's ranked guess
for this request: `Files:` best first, `Symbols:`, and `Notes:` — finding *topics*
from past sessions, so `ken_recall` one to read its body. If it names what you need,
open that file and skip searching. If a listed file was irrelevant, `ken_dismiss(path,
reason)` — the ranker's only negative signal, and only useful while you can still see
it. Thin or missing? `ken_rank(verbose=2)`, or `ken_intent_history("<task>")` for the
files past tasks like this one actually touched.

**Before the first search**, don't reach for `rg`, `find`, or open files you are
guessing at. One ken call, matched to the question:
- exact string or identifier (`MY_ENV_VAR`, `os.path`) → `ken_grep`, not `rg`
- which file implements X → `ken_search_files`
- where is the function/class that does X → `ken_search_symbols`
- how a route / CLI command / env var reaches its handler → `ken_wiring`

Then read what ken named: it narrows the search space, it doesn't replace reading
code. Two ken calls is normal, five means you should have opened the file already;
trivial or in-context questions need none. Use `rg` when ken comes back empty — it
searches indexed files, so something created moments ago may be missing.

**Before editing an unfamiliar file**, `ken_file_findings(path)` — what past sessions
learned here. If the change isn't local: `ken_blast_radius` (what it breaks),
`ken_cochange` (what changes with it that imports don't show), `ken_find_tests`.

**Before finishing, write back — the step agents skip, and the reason ken stops
improving.** Learned something non-obvious that cost real effort (a root cause, a
constraint, where a subsystem actually lives)? `ken_remember(topic, content)`, so the
next session starts where this one ended. Durable facts, not a session log; same
topic overwrites.

The shell commands (ken rank, ken search-files, ken search-symbols, and ken explain) expose the same ideas for humans or agents without MCP, and ken tools <name> runs any MCP tool directly (see Run MCP tools from the shell). For assistants with MCP available, the MCP tools are the preferred path because they return structured results and feed ken's local task memory.

The hooks run automatically when the assistant is active, but the CLI is useful for checking what ken sees:

ken status .
ken rank "where is codex install wiring handled"
ken rank --verbose 2 "how does predictive ranking work"
ken explain "why did src/ken/cli.py appear"
ken search-files "semantic file retrieval"
ken search-symbols "merge codex hooks"

You can save and recall project-specific findings:

ken remember "codex wiring" "Use ken install --codex . to repair invalid hooks."
ken recall "codex hook repair"

Every tool the ken MCP server exposes is also runnable directly with ken tools, so you can use the structured code-intelligence tools (call graph, blast radius, co-change, wiring, clones, …) without an assistant in the loop. The list, descriptions, and parameters are read live from the same MCP surface, so ken tools never drifts from what the agent sees.

ken tools                                   # list every tool with a one-line summary
ken tools grep --help                       # show one tool's parameters
ken tools grep "MY_ENV_VAR" --mode bm25     # required params are positional, options are --flags
ken tools blast_radius src/ken/cli.py
ken tools file_symbols src/ken/search.py --no-include-docstrings

The tool name may be given with or without the ken_ prefix (grep or ken_grep). Results print as JSON (--compact for a single line). Point at another checkout with ken tools --path /repo <name> ... (the flag comes before the tool name).

After ken install --codex ., start Codex in the project and run /hooks to enable the project hooks. Codex only loads project-local hooks after the project is trusted, so also approve the trust prompt when Codex asks.

You can mark the project as trusted manually in ~/.codex/config.toml:

[projects."/abs/path/to/my-project"]
trust_level = "trusted"

ken uninstall .
ken uninstall --keep-db .

ken uninstall . removes hooks, MCP entries, and the local .ken/ index. Use --keep-db if you want to keep .ken/ken.db for later.

ken is released under the MIT License. See LICENSE.