Skip to content

Repository files navigation

lwc

Persistent, source-grounded wikis for LLM agents.

License: Apache-2.0

English · 简体中文

lwc is an agent-first CLI that turns curated documents into a durable Wiki. The agent reasons and synthesizes; lwc preserves sources, pages, citations, links, indexes, and history so knowledge compounds instead of being rediscovered from raw chunks on every query.

LWC Is Agent Memory, Not RAG

RAG and LWC can both help an LLM work with external documents, but they keep state in different places. A typical RAG request retrieves raw chunks and builds one answer at query time:

query -> retrieve chunks -> generate answer

LWC keeps the useful work between requests:

task -> recall maintained Wiki -> reason from sources and prior synthesis
     -> write durable improvements back

Retrieval is one operation inside LWC, not its organizing principle. The durable artifact is a source-grounded Wiki whose pages, citations, links, contradictions, and history are revised as knowledge changes. LWC therefore does not require embeddings or a vector database, and it does not discard each synthesis after answering. It can complement RAG, but it is not query-time RAG.

The Agent operates LWC

lwc is a machine interface for Agents, not a human-facing note-taking app. In normal use, a human selects sources, states goals, asks questions, and reviews answers or the projected Markdown. The Agent runs the CLI, manages scope, integrates sources, maintains citations and links, and decides what is worth recalling or writing back.

Do not manually drive the routine lwc workflow unless you are developing or debugging the tool. Ask your Agent to activate the bundled using-lwc Skill instead—usually as $using-lwc. The setup below also registers $using-wiki where the Agent runtime supports named Skill commands.

Recommended: Ask Your Agent to Set Up LWC

Paste this prompt into the Agent you use. It uses that Agent's own native settings to install the CLI and user-level Skills, initialize global memory, and add a minimal session-start reminder when Hooks are available—all without overwriting existing configuration.

Copy the complete setup prompt
Configure LWC completely for the current user and the Agent runtime executing
this prompt. Perform the work and verify it; do not merely describe commands
for me to run.

Source of truth:
- https://github.com/JanYork/llm-wiki-cli
- https://github.com/JanYork/llm-wiki-cli/tree/main/skills/using-lwc

Requirements:
1. Read the repository README, SECURITY.md, the complete
   skills/using-lwc/SKILL.md, and every directly required script/reference
   before executing it. Record the source commit SHA used for installation.
2. Use your own native user-level locations and mechanisms for Skills, global
   instructions, and lifecycle Hooks. Do not ask the user to identify your
   configuration files, assume another Agent's filenames, or configure other
   installed Agent runtimes unless explicitly requested.
3. Preserve all existing user configuration. Before changing an existing file
   or Skill, create a timestamped backup; make every edit idempotent and do not
   duplicate blocks when this prompt is run again.
4. Install or update `skills/using-lwc` as the canonical user-level Skill. If
   the runtime supports named Skill commands, register `$using-lwc` from that
   canonical Skill and `$using-wiki` as a thin alias that delegates to it. Do
   not copy the implementation into two independently maintained Skills. If the
   runtime uses different invocation syntax, expose the closest native aliases
   and report their exact names. Verify both entry points when supported.
5. From the current host-authorized workspace root, run the canonical Skill's
   bootstrap exactly as its instructions require. Let it install the official
   SHA-256-verified LWC release when needed and initialize ~/.lwc global memory.
   Validate the returned JSON, the absolute lwc path, `lwc --version`, and the
   global Wiki. Do not initialize any project Wiki unless the user explicitly
   requests it for that project; never use global memory as a fallback for
   project-specific writes.
6. Apply the smallest complementary integration through your own native
   configuration:
   - Add a concise LWC routing rule to your additive user-level global
     system/developer instructions. Do not replace your built-in prompt.
   - Merge one LWC section into your user-level global instruction file,
     whatever that file is named. Enclose it with the exact comments
     `<!-- LWC_START -->` and `<!-- LWC_END -->`. On later runs, replace only
     the content inside those markers and leave every user-owned line outside
     them untouched. If only one marker exists, stop rather than guessing where
     user content ends. The section should require use of the canonical LWC
     Skill for substantive project, research, planning, debugging, decision,
     or document-ingest work; recall before re-deriving; write back durable
     findings; keep project and global scopes separate; respect the authorized
     workspace boundary; and never store secrets, raw chain-of-thought,
     transient logs, or unsupported guesses.
   - When you support lifecycle Hooks, create or merge one native user-level
     session-start Hook. It should only add a brief reminder to evaluate and use
     the LWC Skill at suitable times. The Hook must not read, initialize, or
     mutate a project Wiki itself. Do not add a per-prompt Hook, replace
     unrelated Hooks, or bypass Hook trust review. If you do not support Hooks,
     rely on the global instructions and report that limitation instead of
     inventing a mechanism.
7. Keep the Skill, global instructions, and Hook short and complementary instead
   of repeating the full policy. The canonical Skill remains authoritative.
8. Validate every changed config file and Hook executable, confirm existing
   config is preserved, and run the smallest safe smoke checks. Use only your
   native supported configuration rather than inventing keys or filenames.

Finish with a concise report containing: detected Agent runtime, installed LWC
version and path, source commit, installed Skill and alias paths, global Wiki
path, Hook/config files changed, backup paths, validation performed, unsupported
integrations, and anything that requires a new Agent session or normal Hook
trust approval.

Origin and Acknowledgements

lwc implements the LLM Wiki pattern proposed by Andrej Karpathy: an LLM incrementally builds and maintains a persistent, interlinked Wiki instead of reconstructing knowledge from raw documents for every query. The CLI architecture and selected implementation details also draw inspiration from nashsu/llm_wiki.

This project adapts those ideas into an agent-first Rust CLI backed by SQLite.

Core Design

+-----------------------------------------------------------------------+
|                              AGENT PLANE                              |
+-----------------------------------------------------------------------+
| User Task -> LLM Agent -> using-lwc Skill                             |
|                         trigger | bootstrap | recall | write-back     |
+-----------------------------------------------------------------------+
                                   |
                          JSON / stdin / files
                                   v
+-----------------------------------------------------------------------+
|                               CLI LAYER                               |
+-----------------------------------------------------------------------+
| clap command router                                                   |
| init | schema | purpose | source | page | ingest | search | context   |
| graph | lint | changeset | maintenance | checkpoint | log             |
+-----------------------------------------------------------------------+
                                   |
                                   v
+----------------------------------+------------------------------------+
| SCOPE RESOLVER                   | IMPORT / VALIDATION                |
| project | global | all (merge)   | UTF-8 | size | ext | symlink       |
+----------------------------------+------------------------------------+
                                   |
                                   v
+-----------------------------------------------------------------------+
|                             SQLITE STORE                              |
+-----------------------------------------------------------------------+
| Canonical | WAL | foreign keys | transactions | migrations            |
| meta | sources | pages | page_sources | links | ingest_jobs           |
| operations | changesets | search_fts                                  |
+-----------------------------------------------------------------------+
                                   |
                                   v
+-----------------------+-----------------------+-----------------------+
| SEARCH PIPELINE       | GRAPH ENGINE          | MARKDOWN PROJECTION   |
| CJK n-grams + Latin   | links + citations     | raw/ + wiki/          |
| contentless FTS5/BM25 | structural evidence   | index/log/overview    |
+-----------------------+-----------------------+-----------------------+

The persistent knowledge model has three logical layers:

Layer Contents Contract
Raw sources Immutable snapshots of curated input Add through source; never rewrite source truth.
Wiki Agent-maintained pages, citations, links, and provenance Update through page; cite sources and classify durable non-source knowledge.
Schema and purpose Maintenance rules and project intent Guide every future ingest and revision.

SQLite is canonical. The Markdown tree is a rebuildable projection for people and tools such as Obsidian. Agents mutate knowledge through lwc, not by editing .lwc/wiki.db or projected Markdown directly. Successful commands return JSON on stdout; failures return structured JSON on stderr.

Read commands keep current-format stores read-only. When an older writable store is opened by a newer CLI, its schema is migrated transactionally once before the read proceeds.

Hierarchical Recall and Knowledge Graph

Every current Source and Wiki page is deterministically indexed as passages, sentences, and normalized terms. SQLite remains authoritative; span FTS and the typed canonical graph are rebuilt indexes. Existing search stays document-only unless a granularity is requested:

lwc search "projection consistency" --granularity sentence --type page
lwc search "projection consistency" --granularity passage
lwc search "projection consistency" --granularity all --group-by document
lwc span get <SPAN_ID>
lwc span expand <SPAN_ID> --before 1 --after 1 --children 20

Span locators contain the document fingerprint and segmentation version. A locator from a replaced body fails with stale_span and reports prior/current metadata; LWC never silently remaps it to similar text.

Use the bounded, typed graph API for exploration without requiring keywords:

lwc graph explore                         # representative macro view
lwc graph explore term:consistency --depth 2 --edge-type CO_OCCURS
lwc graph node page:projection-policy
lwc graph neighbors page:projection-policy --direction outgoing
lwc graph path page:implementation page:policy --max-depth 6
lwc graph impact page:policy --max-depth 4
lwc graph overview
lwc graph status
lwc graph verify

Automatic edges are limited to structural/evidential facts. Semantic claims must be explicit and auditable:

lwc graph relation set page:implementation DEPENDS_ON page:policy \
  --provenance source-grounded --source 12 \
  --reason "Source 12 states the required policy" --confidence 0.95
lwc graph relation list --from page:implementation
lwc graph relation retract page:implementation DEPENDS_ON page:policy \
  --reason "The dependency was superseded"

Relation reasons are durable content: never put credentials, secrets, or raw chain-of-thought in them.

The physical graph is enabled by default. auto selects the pinned embedded GraphQLite runtime on supported macOS/Linux targets and rslg elsewhere. Windows uses rslg and does not embed or load GraphQLite. Configuration is layered from built-in defaults through global and project files; project values can inherit:

lwc config show
lwc config set --physical enabled --engine auto
lwc config set --engine rslg
lwc config unset --physical --engine

The canonical schema stores exact term positions once per term/document in a compact posting blob. Passage/sentence OCCURS_IN edges and structural CONTAINS/NEXT/PREVIOUS edges are derived deterministically from those postings and span locators, so public graph behavior stays exact without triplicating row/index storage. Co-occurrence evidence is stored in compact per-document blobs plus incremental totals; normalized edge weights are persisted to six decimal places while exact totals remain available for rebuilds.

GraphQLite is a disposable, checksummed sidecar projection. Canonical mutations commit first; a projection failure returns graph_projection_failed, marks the projection stale, and graph reads fail closed. A subsequent writable open or lwc config set --engine graphqlite resumes/rebuilds it. Superseded sidecars are retained and reported by graph status for manual review rather than deleted automatically. Draft changesets always query their candidate canonical graph with rslg and never import deployment-local projection state.

Installation

Most users should use the Agent setup prompt above. The manual commands below are for maintainers, debugging, or Agent environments that cannot install the companion Skill.

Install from GitHub:

curl --proto '=https' --tlsv1.2 -fsSL https://github.com/JanYork/llm-wiki-cli/releases/latest/download/install.sh | sh

The installer supports x86_64/aarch64 macOS, glibc Linux, and Windows Git Bash, verifies the release checksum, and installs or updates lwc. It uses ~/.local/bin by default, or updates an existing copy in ~/.local/bin or ~/.cargo/bin. To choose another directory:

curl --proto '=https' --tlsv1.2 -fsSL https://github.com/JanYork/llm-wiki-cli/releases/latest/download/install.sh | LWC_INSTALL_DIR="$HOME/bin" sh

Alternatively, build and install from GitHub with Cargo:

cargo install --locked --git https://github.com/JanYork/llm-wiki-cli

Or install a local checkout:

git clone https://github.com/JanYork/llm-wiki-cli.git
cd llm-wiki-cli
cargo install --locked --path .

Companion Agent Skill

The repository includes skills/using-lwc, an Agent Skill that makes lwc a proactive memory layer for substantive sessions. Install it in the current Agent runtime's user-level Skills directory. For Codex, from a local checkout:

mkdir -p "${CODEX_HOME:-$HOME/.codex}/skills"
cp -R skills/using-lwc "${CODEX_HOME:-$HOME/.codex}/skills/"

The canonical invocation is $using-lwc. The setup prompt above also creates the optional $using-wiki alias without duplicating the Skill implementation.

When triggered, the Skill:

  • finds a compatible CLI or installs the official checksum-verified release;
  • initializes global memory in ~/.lwc/ once;
  • recalls bounded global and project context before repeated investigation;
  • initializes the active project on explicit invocation, otherwise asks first;
  • refuses project writes outside the current authorized workspace root;
  • separates project facts from reusable global knowledge;
  • integrates sources and writes durable answers back into the Wiki.

The Skill uses LWC_PROJECT_ROOT first as the canonical authorized workspace boundary, then narrows it to the selected active project; discovery and initialization cannot traverse above it.

Set LWC_AUTO_INSTALL=0 to disable automatic CLI installation. Automatic installation executes the reviewed installer bundled in the Skill, trusts this repository and its GitHub Release publishing boundary, and verifies the downloaded archive against SHA256SUMS; the checksum is integrity protection, not publisher code signing. Release binaries cover x86_64/aarch64 macOS, glibc Linux, and Windows through Git Bash. SKILL.md follows the Agent Skills resource layout, while agents/openai.yaml supplies OpenAI/Codex metadata. The CLI itself is runtime-neutral: any Agent that can execute it and load or adapt the Skill's instructions can use LWC. Skill commands, global instructions, and Hooks remain runtime-specific, so the setup prompt detects and configures the current host.

Quick Start

This section documents the CLI protocol that the Agent executes. Humans do not need to run these commands during normal use.

1. Initialize a project Wiki

cd your-project
lwc init
printf '# Schema\nEvery page declares provenance; source-grounded claims cite sources.\n' | lwc schema set -
printf '# Purpose\nBuild a durable project Wiki.\n' | lwc purpose set -

Project initialization adds the project-relative .lwc/ path to Git's local info/exclude file when needed, without changing the repository .gitignore. Use lwc init --no-git-exclude only when the Wiki is intentionally versioned.

2. Add source material

lwc source add-dir docs/

Files without an explicit title use their source origin as a stable, human-readable fallback. Identical bytes are deduplicated by SHA-256. Project sources that resolve outside the active Wiki root require --allow-external-source. High-confidence credential markers are rejected unless the reviewed source is explicitly acknowledged with --acknowledge-sensitive-source.

Each successful add also records the observed file path and its current immutable snapshot. Check only the sources relevant to the task before relying on file-backed evidence:

lwc source status 7 12

The command streams each live file through SHA-256 and reports path lineage (current or superseded) separately from filesystem state (current, modified, missing, unreadable, oversized, or unstable). It is read-only. Use source status --all only for explicit maintenance because its cost is proportional to the bytes in all tracked files. Inspect a modified path before updating knowledge:

lwc source diff 7
lwc source refs 7 --limit 1000

source diff compares the immutable source with its live file, or with another snapshot via --to-source. It returns a bounded unified diff: at most 8 MiB and 200,000 lines per side, 20,000 Unicode output characters by default, and 100,000 with --max-chars. If one source was observed at multiple paths, select an exact --path. A truncated diff is only a preview. source refs lists directly citing review candidates; it does not prove which pages are semantically affected. Re-run source add only after review when the same path contains a meaningful new revision. An A -> B -> A sequence remains three path observations even though content A reuses its original source ID. External live paths require --allow-external-source again; flagged live text also requires --acknowledge-sensitive-source after inspection.

Sources migrated from older stores remain explicitly untracked because LWC does not guess historical paths; re-add the intended file once to establish its first tracked revision. If a file or path head changes during the check, LWC returns source_status_unstable; retry instead of trusting a mixed-time result.

For a curated atomic import, paths in a JSON manifest resolve from the manifest's directory:

{
  "sources": [
    {"path": "ARCHITECTURE.md", "title": "Architecture contract"},
    {"path": "src/store.rs", "title": "SQLite store"}
  ]
}
lwc source add-manifest lwc-sources.json

3. Analyze and integrate one source

lwc ingest next --context-limit 50 --source-max-chars 100000
lwc ingest analyze 1 --file analysis.md

Use lwc ingest claim 7 when a manifest or scheduler already selected an exact pending source ID.

If source_window.has_more is true, continue reading from source_window.next_offset_chars:

lwc source show 1 --offset-chars 100000 --max-chars 100000

Create a cited source-summary page and integrate its contribution into at least one non-source page before completing the ingest task:

lwc page put source-1 \
  --title "Source 1 Summary" \
  --kind source \
  --summary "What this source contributes" \
  --file source-summary.md \
  --source 1

lwc page put durable-concept \
  --title "Durable Concept" \
  --kind concept \
  --summary "How this source changes shared knowledge" \
  --file concept.md \
  --source 1

lwc ingest complete 1

Both layers are required: the source page is a navigation and provenance aid; the non-source page makes knowledge compound. If a source genuinely changes no shared page, complete it with a specific audited explanation:

lwc ingest complete 1 \
  --no-derived-pages-reason "Duplicate evidence; existing synthesis already covers every supported claim"

Source citations automatically expose source-grounded provenance. For durable knowledge that comes from the user, an Agent observation, or an explicit hypothesis, repeat --provenance as needed instead of inventing a source:

lwc page put architecture-decision \
  --title "Architecture decision" \
  --kind query \
  --summary "Accepted constraint and remaining uncertainty" \
  --file decision.md \
  --provenance user-provided \
  --provenance hypothesis

page put replaces the complete citation and explicit-provenance sets. Read the existing page first, then repeat every still-valid --source and non-source --provenance value. Do not pass source-grounded explicitly; it is derived from citations. Provenance is returned by page reads, context, search, source references, and Markdown projection, but does not change search ranking.

4. Query the accumulated Wiki

lwc context --limit 50
lwc search "question keywords" --limit 20
lwc search "question keywords" --limit 20 --explain
lwc search "concept only" --type page --kind concept
lwc search "exact evidence" --type source
lwc page show source-1

Agent Workflow

The intended workflow is:

  1. Collect immutable sources.
  2. Claim one ingest task with bounded lwc ingest next, or ingest claim <ID> when the source was selected explicitly.
  3. Read every returned source window, plus the schema, purpose, and bounded context.
  4. Analyze before generating pages.
  5. Write or revise a source summary and shared durable pages with explicit --source citations.
  6. Complete only after both integration gates pass, or record why no shared page should change.
  7. Put a multi-command ingest or broad revision in one changeset, validate the draft, then publish it atomically.
  8. Use search, context, graph, and lint to keep the Wiki coherent over time.

See docs/agent-workflow.md for the full operating contract. Run lwc --help or lwc <command> --help for Agent-oriented preconditions, state transitions, side effects, and next actions.

Atomic Multi-command Changes

A single source or page command is transactional. Use a changeset when one logical update needs several commands and must not expose a partial Wiki:

lwc --scope project changeset begin architecture-refresh
lwc --scope project --changeset architecture-refresh source add-manifest sources.json
lwc --scope project --changeset architecture-refresh ingest claim 1
# Analyze, write cited pages, and complete ingest with the same selector.
lwc --scope project --changeset architecture-refresh lint
lwc --scope project --changeset architecture-refresh search "expected answer" --limit 5
lwc --scope project changeset show architecture-refresh
lwc --scope project changeset commit architecture-refresh

Draft reads see staged writes, while live SQLite and Markdown stay unchanged. changeset show reports staged operations, lint, revisions, conflicts, and readiness. Commit rejects empty drafts, lint issues, and any live/draft revision conflict; there is no force or automatic merge. Use --allow-lint-issues --reason "reviewed pre-existing debt" only for audited debt that the changeset did not introduce. After commit, rerun the same fixed retrieval checks against live state. Commit freezes the reviewed draft before publication; changeset_frozen blocks any later staged write. Retry the same commit for recovery, or discard after a reported conflict—never add more work to a frozen draft.

lwc --scope project changeset discard architecture-refresh
lwc --scope project changeset rollback <CHANGESET_ID>

Discard touches only an uncommitted draft. Commit creates a pre-commit checkpoint and returns the exact rollback ID; rollback is allowed only before the next live mutation and creates its own safety checkpoint. Project and global changesets are separate, --scope all is invalid, and init, maintenance, checkpoint, and nested changeset commands reject --changeset. Drafts never create a second Markdown projection. If a structured error reports committed=true with cleanup or materialization work remaining, do not repeat the knowledge changes; run the returned recovery action.

Scopes

lwc supports three scopes:

Scope Store Use
project Nearest ancestor .lwc/wiki.db Default, project-specific knowledge
global ~/.lwc/wiki.db Reusable cross-project knowledge
all Project and global stores Combined search and context only

Examples:

lwc --scope global init
lwc --scope global source add shared.md
lwc --scope all search "shared term"
lwc --scope all context

Knowledge writes are explicit. all does not create implicit cross-store citations or links; search --record only appends the query operation to each selected store.

Search and CJK

Search is lexical and deterministic.

  • Search terms are plain text, not raw FTS syntax.
  • --type auto is the default: compiled pages rank first, paired raw sources are hidden, and raw sources provide fallback recall.
  • Use --type page, --type source, or --type all to select a layer. Repeat --kind to restrict page results, such as --kind concept --kind synthesis.
  • Multi-character CJK query terms use adjacent bigrams; the index also retains non-stopword unigrams so one-character queries remain searchable.
  • Latin text is tokenized into lowercased alphanumeric terms.
  • Ranking keeps title, source filename, path/slug, summary, and body evidence distinct. Exact/partial title and path matches receive bounded boosts.
  • README/index/overview documents and explicit navigation hubs are query-conditionally downweighted in favor of specific feature documents; asking for the README or overview disables that penalty.
  • Page candidates may receive a bounded direct-link or shared-source graph boost. Common-neighbor-only relationships cannot change search order, and a broad navigation hub receives a bounded graph penalty.
  • --explain returns the exact score arithmetic, including lexical, generic, graph, manual-weight, and query-feedback signals. It does not record the query; --record remains the only search-history opt-in.
  • Fixed coefficients and lower-is-better ranks keep project and global results comparable under --scope all.

This is intentionally dictionary-free. The goal is stable behavior for product names, code names, mixed-language terms, and emerging vocabulary without depending on a word-segmentation dictionary.

Explicit retrieval weights and feedback

Use a document weight for a durable, query-independent judgment about a page or source. Use feedback for one exact ordered-token query fingerprint:

lwc weight set page payment-rules \
  --value 2 \
  --reason "Canonical payment rules specification" \
  --provenance agent-observed
lwc weight list page payment-rules

lwc weight feedback page payment-rules \
  --query "payment reconciliation rules" \
  --signal relevant \
  --reason "Verified against the expected answer" \
  --provenance agent-observed

lwc weight feedback-clear page payment-rules \
  --query "payment reconciliation rules" \
  --provenance agent-observed
lwc weight clear page payment-rules --provenance agent-observed

Document values are -2, -1, 1, or 2; use clear for zero. Both mechanisms only rerank lexical candidates and cannot make a nonmatching document appear. A user-provided row takes precedence over an agent-observed row while both remain auditable. Feedback stores the SHA-256 fingerprint, not the raw query, and does not transfer to paraphrases with different tokens. Reasons and operation records are durable, so never copy a sensitive query into --reason. Mutations require an explicit project or global scope; --scope all is rejected.

Maintenance and Projection

Useful maintenance commands:

lwc lint
lwc maintenance reindex
lwc maintenance materialize
lwc maintenance compact
lwc checkpoint create before-large-update
lwc checkpoint list
lwc log --limit 20

Notes:

  • lint is read-only by default. Add --record only when the lint pass belongs in durable operation history.
  • maintenance reindex rebuilds derived search artifacts from SQLite.
  • maintenance materialize rebuilds the projected Markdown tree from SQLite.
  • maintenance compact optimizes the contentless FTS5 index and attempts a WAL truncate checkpoint. Run it while the Wiki is idle and inspect busy plus after_bytes.
  • Search queries are private by default; add --record only when you want the query wording stored in the durable operation log.

lwc checkpoint create <NAME> uses SQLite's online backup API. Restore with lwc checkpoint restore <NAME>; LWC first creates a pre-restore-* safety checkpoint and then rebuilds the projection. Use source remove <ID> and page remove <SLUG> for guarded deletion: sources with citations and pages with inbound links are refused. Removing the current source for a tracked path stops tracking that path instead of silently exposing an older revision as current.

For a multi-source ingest or broad page replacement, prefer a changeset over a manual checkpoint: successful commit checkpoints automatically, publishes canonical SQLite in one transaction, and materializes live Markdown once. A returned wal_checkpointed=false means an active reader prevented immediate WAL truncation; the canonical commit still succeeded.

For an external filesystem backup, stop active lwc commands and copy the complete .lwc/ directory. Do not copy only wiki.db while a writer may still be using its WAL files.

Benchmark Suite

The opt-in benchmark imports a local UTF-8 corpus into a temporary Wiki and reports import time, search P50/P95, Recall@5/10, MRR, and storage before/after compaction. Ground truth is a JSONL file of queries and expected corpus-relative paths:

cargo build --release
LWC_BENCH_CORPUS=/path/to/sanitized-corpus \
LWC_BENCH_QUERY_SET=/path/to/query-set.jsonl \
LWC_BENCH_BINARY="$PWD/target/release/lwc" \
cargo test --test search_benchmark -- --ignored --nocapture

Normal cargo test --all-targets covers page-first search, type/kind filters, UTF-8 source windows, ingest completion gates, graph precision, migrations, lint, and WAL compaction. See benchmarks/README.md for the workload contract and fair before/after comparison rules.

Limits and Non-Goals

Current design constraints:

  • single-machine, single-user knowledge base;
  • UTF-8 text workflow;
  • bounded input size of 64 MiB per schema, purpose, source, or page body;
  • lexical search, not semantic vector retrieval.

Deliberate non-goals for this CLI:

  • no built-in LLM calls;
  • no vector database;
  • no daemon or background service;
  • no web UI or desktop UI;
  • no direct database editing contract.

If the projected Markdown drifts, rebuild it. If the SQLite schema is wrong, fix it through the CLI and migrations, not by hand.

Contributing

Issues and pull requests are welcome, especially around:

  • agent workflow ergonomics;
  • deterministic projection behavior;
  • durable citation and page maintenance contracts;
  • search quality for multilingual technical corpora.

Please read CONTRIBUTING.md before opening a pull request. Report security issues according to SECURITY.md.

License

Licensed under the Apache License 2.0.

About

An agent-first CLI for building persistent, source-grounded wikis from local documents, powered by Rust and SQLite.

Topics

Resources

Contributing

Security policy

Stars

15 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages