日本語 · Documentation

A cross-workspace knowledge search engine built in Rust. Indexes Markdown documents across multiple workspaces and provides hybrid search combining FTS5 full-text search with vector semantic search (ruri-v3-30m, 256-dim).

This repository ships only the tsm / tsmd binaries. The Claude Code plugin (skills, agents, hooks) lives in a separate repository, key/tsm-plugin-cc, which doubles as its own plugin marketplace:

/plugin marketplace add key/tsm-plugin-cc
/plugin install the-space-memory@tsm-plugin-cc

The plugin calls the tsm CLI, so install tsm separately (see below) and ensure it is on your PATH.

• Cross-workspace search — Index and search across multiple repositories (personal notes, work projects, tech notes) from a single orchestration repo
• Sub-100ms local search — Both indexing and search run locally with no network overhead, completing in under 100ms
• Transparent Claude Code integration — Hooks intercept prompts, search the knowledge base, and inject relevant context automatically

• Hybrid search — FTS5 + vector search fused via Reciprocal Rank Fusion (RRF)
• Morphological analysis — Japanese tokenization via lindera (IPADIC)
• Semantic search — ruri-v3-30m embeddings computed locally with candle (no ONNX Runtime)
• Entity graph — Automatic entity extraction and link inference
• Synonym expansion — WordNet + user-defined CSV for query expansion
• Session ingestion — Index Claude Code session transcripts as searchable knowledge
• Single binary — No Python, no external runtime dependencies

File watching uses inotify (Linux) / FSEvents (macOS).

# 1. Build
cargo build --release

# 2. Download external resources (ruri model + Japanese WordNet DB).
#    System-wide; run once per machine.
tsm setup

# 3. Initialize the workspace from your notes directory:
#    DB schema, default scaffold files (tsm.toml, .tsmignore,
#    .tsm/{user_dict.simpledic,custom_terms.toml, synonyms.csv,
#    hooks/extract/10-md_frontmatter.lua, hooks/score/10-default.lua}),
#    and WordNet/synonym imports. Idempotent — existing
#    user-customized files are never overwritten.
cd ~/my-notes
tsm init

# 5. Start the daemon (embedder + file watcher)
tsm start

# 6. Index your documents
tsm index

# 7. Search
tsm search -q "query" -k 5

# Scope to a directory (absolute or relative to the current directory)
tsm search -q "query" --path notes/

tsm setup sets HF_HUB_CACHE automatically; override it to redirect the Hugging Face model cache.

tsm setup populates a machine-wide cache (~/.cache/tsm) once per machine; tsm init then materializes the model and WordNet DB into the workspace's .tsm/ as a symlink (default) or copy. See Resource Layers and link_mode.

tsm recursively scans the project root (directory containing tsm.toml) for .md files. A typical directory layout:

~/my-notes/              ← project root (contains tsm.toml)
├── projects/
│   ├── project-a.md
│   └── project-b.md
├── research/
│   └── notes.md
└── journal/
    └── 2026-04.md

By default (no content_dirs configured) every Markdown file under the project root is indexed. Set content_dirs in tsm.toml to scope indexing to specific directories — each listed directory is still scanned recursively, but anything outside them is skipped. The file watcher detects additions, modifications, and deletions in real time.

# Re-index while daemon is running (non-destructive, background)
tsm reindex all       # FTS + vectors
tsm reindex fts       # FTS only (after dictionary changes)
tsm reindex vectors   # Vectors only (after model changes)

# Rebuild from scratch (destructive, requires daemon stopped)
tsm rebuild           # Dry run (shows DB stats)
tsm rebuild --apply   # Delete DB and rebuild

Use tsm doctor to check system health and daemon status.

Text is normalized to Unicode NFC (canonical composition) on index and query, so visually identical text with different byte encodings (e.g. a precomposed vs. decomposed katakana dakuten) always matches. This applies going forward only — content indexed before this was added may still hold non-normalized text. Run tsm rebuild --apply to renormalize existing content, then tsm dict import to resync dictionary verdicts from user_dict.simpledic / reject_words.txt.

tsm uses embedded Lua (mlua, lua54) for user-editable metadata extraction and result scoring. Hooks live in .tsm/hooks/ and are scaffolded by tsm init.

.tsm/hooks/
├── extract/
│   └── 10-md_frontmatter.lua   ← extract hook (index time)
└── score/
    └── 10-default.lua          ← score hook (search time)

• Only .lua files are loaded; non-.lua files are ignored.
• Multiple hooks in the same directory are executed in sorted file-name order.
• To disable a hook without deleting it, rename it away from .lua (e.g. 10-default.lua.disabled).
• An empty or absent directory falls back to the embedded default hook.

Called at index time for each document chunk. Receive a context table:

-- ctx fields: path, body, frontmatter (top-level YAML keys), metadata (accumulated so far)
function extract(ctx)
  local fm = ctx.frontmatter or {}
  return { status = fm.status, effective_date = fm.updated }
end

ctx.frontmatter exposes the top-level YAML keys: scalars (string/number/bool) and sequences (e.g. tags, as a Lua array). Nested mappings are not passed.

Return a flat table of scalar values (string, number, boolean). The results of all extract hooks are shallow-merged (last-wins for duplicate keys) and stored in documents.metadata (JSON column).

Called at search time for each result. Receive a context table:

-- ctx fields: metadata (from extract), rrf, source_type, path, half_life_days
-- builtins: decay(date, half_life_days), today()
function score(ctx)
  local m = ctx.metadata or {}
  local penalty = ({ outdated = 0.4 })[m.status] or 1.0
  return penalty * decay(m.effective_date, ctx.half_life_days)
end

The snippet above is a minimal example. The shipped default (score/10-default.lua) uses the full status→penalty table: superseded=0.2, rejected=0.3, dropped=0.3, deprecated=0.3, outdated=0.4, proposed=0.7, everything else 1.0.

Each hook returns a multiplier (>= 0). The final score is rrf × weight × Π(score hooks). Invalid returns (negative, NaN, ±Inf) are treated as 1.0 with a warning.

• Lua VMs are sandboxed: no standard library (io/os/package absent), 64 MiB memory ceiling. Hooks cannot touch the filesystem, network, or processes.
• The daemon validates and loads all hooks at startup (fail-fast on syntax error or missing entrypoint). The CLI uses a lazy per-process cache.
• Editing a hook requires tsm restart — hooks are not reloaded at runtime.
• The metadata column is added automatically (idempotent migration on connect); existing rows have NULL metadata and the searcher synthesizes scoring from status/updated, so scoring is unaffected. To populate metadata for existing documents after writing custom extract hooks, run tsm reindex — a full destructive rebuild is not required.

Every setting below can also be set in tsm.toml (the tsm.toml key column); the environment variable takes precedence. See docs/configuration.md for the full reference.

tsm also honors RUST_LOG (log level, default info) and NO_COLOR (disable colored output).

Performance benches for the search/index pipeline, plus a CI regression gate that runs on every PR touching src/, benches/, or Cargo.toml. Only embedder call counts are regression-gated (deterministic, exact equality); indexing throughput and search latency are recorded for trend visibility only. See docs/benchmarks.md for the full design rationale, how to run the benches locally, and the CI workflow internals.

Precision@5 / MRR / nDCG@5 measured against a hand-graded golden corpus, gated in CI against a committed baseline — catches quality regressions from dictionary, scoring, or hook changes that don't break any existing test but silently make search worse. Run it locally with bash tests/quality_bench.sh.

See Search Quality for the full guide: design rationale, the golden corpus/query format, running and updating the baseline, and how the regression gate works.

• Command Reference — CLI commands, flags, and usage examples
• Architecture — Process architecture and component responsibilities
• Data Flow — Indexing and search flow diagrams
• Configuration — Environment variables and config file reference
• User Dictionary — Custom dictionary management
• Benchmarks — Perf benches, CI regression gate, baseline lifecycle
• Search Quality — Precision/MRR/nDCG regression gate: golden corpus, gate design, updating the baseline
• Design Decisions — ADR (Architecture Decision Records)

The Space Memory was inspired by sui-memory, which introduced the idea of indexing Claude Code session transcripts into a searchable database. tsm extends this concept from session records to entire document repositories, enabling cross-workspace knowledge search.

Existing tools each had critical gaps for this use case:

• Notion / GitHub search — Network-bound; too slow for real-time prompt injection
• grep — Sequential scan with no semantic correlation between terms
• Obsidian — Excellent Markdown editor, but not designed for AI agent integration

tsm was built to fill these gaps: a local-first, sub-100ms hybrid search engine that integrates transparently with Claude Code via hooks. The combination of FTS5 and vector search bridges vocabulary gaps (e.g., matching "shooting" with "firearms"), and Japanese tokenization via lindera/IPADIC was a key reason to build a custom solution rather than adapting English-centric tools.

The name follows sui-memory's naming pattern (prefix + memory), with "space" representing the multiple repositories treated as a unified search space. The cover image is an homage to Hydlide 3, whose subtitle is The Space Memories.

MIT