Skip to content
lx-0Public

About

Self-cartography engine: Turn your scattered personal data into a queryable Obsidian wiki. You and your agents read from the same surface.

Topics

Resources

Contributing

Security policy

Stars

4 stars

Watchers

0 watching

Forks

Latest commit

Β 

History

1,348 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

llm-wiki β€” self-cartography engine

Self-cartography engine.

Turn your scattered personal data into a queryable Obsidian wiki. You and your agents read from the same surface.

License Status Python Claude Agent SDK Ollama Obsidian

Warning

Heavy prototype development. This is a single-author scratch space, not a stable tool. Architecture, CLI surface, file layout, and config schema can break at any time without notice or migration path. No semver, no deprecation cycle, no compatibility promises. If you install it, expect to read commits before pulling.

14
substrate collectors
8
psychometric instruments
MIT
open source

Yesterday's Claude Code session is already a daily-log entry. This week's screenshots already have a summary article. Open a fresh agent session and the wiki index loads before you type. You read what the agent wrote. The agent reads what you wrote. Both refine the same surface.

Contents

What is this

An opinionated knowledge-compilation engine for personal substrates. Drop raw materials in β€” daily session logs, web clippings, screenshots, emails, calendars, browser history, HTML files β€” and a Claude Agent SDK loop compiles them into atomic Markdown articles with wikilinks. Renders as a navigable wiki inside Obsidian. The same wiki gets injected into every AI-agent session as context, so the loop closes: you read what you wrote, the agent reads what it wrote, both refine the same surface.

The vault holds data. The .wiki/ directory holds the engine. They never mix on disk.

A map, not an agent. The wiki is a map of the operator β€” a compiled, always-current, descriptive portrait of what you know, did, and committed to, read daily by you and your agents. It describes; it doesn't act. That's why it's a self-cartography engine, not a "second brain": "brain" implies the part that thinks and does, which is deliberately out of scope. Anything that acts on the map β€” automating a task, syncing a commitment to an external tool β€” is an operator-gated consumer downstream of the compiler, never the compiler itself.

What this isn't. Not a vector database, not a RAG service, not a Notion / Logseq / Mem replacement, not a docs-site generator, not a team wiki β€” and not a task manager, executive/agent layer, or "second brain." It is a compile loop for one operator's substrates, output as plain Markdown that any tool can read.

Why it exists

A solo knowledge worker's thinking lives in too many partial substrates: daily notes capture what happened, AI-agent memories capture working thought, clippings capture curiosity, screenshots and calendars capture everything else. Each is queryable in isolation, but not as a whole β€” and most are illegible even to the person who produced them.

Existing tools either solve a slice or solve a different problem:

  • Karpathy's LLM Wiki β€” the raw/ β†’ knowledge/ shape and the compile-don't-retrieve choice; a sketch, not a working pipeline.
  • claude-memory-compiler (Cole Medin) β€” the session-capture pattern; only handles AI-agent memories, doesn't ingest other substrates.
  • Obsidian, Notion, RAG systems β€” manual curation, team docs, or retrieval without compilation (weak cross-doc reasoning).

llm-wiki is the compilation layer between raw substrates and active consumption β€” by humans reading and by agents prompting. It is not an archive to be left behind; it is a working surface that gets refined by being used.

What you get

The map β€” substrates in, a compiled wiki out:

  • Two-path ingest + per-day rollup β€” automatic session capture (hooks β†’ daily/<date>/sessions.md) and substrate-source writers (Registry-discovered Collectors + clipper + manual drop β†’ raw/, plus per-day mirror summaries into daily/<date>/{health,meetings,voice,email,pictures,captures}.md β€” email carries top senders + recent subjects, not just a count) converge at one compiler. A daily-digest pass distills the per-source captures into a single daily/<date>.md (~500 words). All fourteen collectors ride the formal Collector Protocol (SPEC + @register + run()), registered lazily by name: email, calendar, jamie, gmeet, wispr, voice, pictures, capture, health, screenshots, browser, tabs, youtube, folder-index.
  • Multi-agent hooks β€” session-start / session-end / pre-compact wired into Claude Code, Codex, Gemini, and Cursor. Every session ends as a structured daily-log entry.
  • Curiosity loop β€” a small local Ollama model spots gaps after each compile and queues deep-scan requests for the next cycle.
  • Self-healing wiki β€” lint.py runs 18 structural checks (links, orphans, staleness, article shapes and schema, fact violations, …) plus an LLM contradiction scan, so the wiki stays consistent as it grows.

Consumers of the map β€” operator-gated layers that read the compiled wiki; where they act, only with explicit approval, downstream of the compiler:

  • Surfaced commitments β€” the compiler reads back the commitments your substrate reveals (jamie + gmeet transcripts; Task / Owner / Deadline / Context quartet) and surfaces them on the relevant knowledge/people/<slug>.md + knowledge/projects/<slug>.md entity pages as ## Action Items (Obsidian-Tasks-plugin syntax) and ## Open Threads. Resolved items demote to Timeline on the next compile; a dashboard pane + cross-entity Inbox MOC make them legible. This surfaces commitments as part of the portrait β€” it does not manage them: reminders, completion-as-workflow, and syncing to an external task tool are out of scope (a downstream consumer's job).
  • Optimization suggestions β€” the compiler proposes YAML automations (e.g. mail-filter rules) with per-action approval before execution.
  • Remote mirror (MCP) β€” wiki publish mirrors the vault's markdown one-way into a managed wiki on the operator's meinkontext server, so every agent anywhere reads the same corpus over MCP. Idempotent by content hash, wikilinks normalized to global slugs, deletions archive upstream; one browser OAuth consent (wiki publish --auth), then a compile piggyback keeps it fresh. See docs/setup-publish.md.
  • Operator self-reports β€” air-gapped analytical surface at <vault>/reports/. Validated clinical screens (PHQ-9, GAD-7, WHO-5, PSS-10, ISI, OLBI, K6, ASRS v1.1) scored by an informant agent reading the operator's own substrate. Deterministic Likert + cutoffs; the LLM only fills in raw answers. Two-pass analyst (per-study + cross-study). wiki study run <id> + wiki analyze. See docs/cli.md.

Overview

High-level dashboard. For the full cognitive-architecture diagram see docs/architecture.png. Both rendered from .excalidraw files in docs/.

What it looks like in Obsidian

Vault tour β€” Obsidian sidebar with vault folders open and a compiled article in the main pane

The sidebar is the data layout β€” raw/, daily/, knowledge/ β€” exactly the folders the compiler creates and reads. The open file is a compiled article: frontmatter, body, wikilinks, and sources, none of it written by hand.

Two ingest paths converge at compile

PATH A β€” Automatic capture                PATH B β€” Curated sources
─────────────────────────────             ──────────────────────────────────
session-start / session-end /             Collectors (Registry β€” fourteen):
pre-compact hooks attach to every         Β· mail, calendar   email, calendar
Claude Code / Codex / Gemini /            Β· meetings         jamie, gmeet, wispr
Cursor session. flush.py extracts         Β· mobile inboxes   voice, pictures, capture
the conversation transcript and           Β· body             health (Oura)
appends a structured entry to             Β· screen, web      screenshots, browser,
daily/<date>/sessions.md, then a                             tabs, youtube
daily-digest agent distils all            Β· local folders    folder-index (body-blind)
per-source captures into
daily/<date>.md (~500 words).             Other writers:
                                          Β· clippings-sweep  (Obsidian Web Clipper)
            β”‚                             Β· ingest-html      (file or URL)
            β”‚                             Β· process-inbox    (LLM-classified drop)
            β–Ό                                          β”‚
   daily/<date>/<source>.md +                          β–Ό
   daily/<date>.md (compile digest)       raw/{articles,papers,notes,transcripts,voice,
            β”‚                                  audio,captures,index,requests,suggestions}/
            └──────────────┐    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                           β–Ό    β–Ό
                  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                  β”‚     compile.py     β”‚   Claude Agent SDK loop.
                  β”‚                    β”‚   Reads index.md + AGENTS.md (vault),
                  β”‚                    β”‚   walks raw/ + daily/, distils into
                  β”‚                    β”‚   atomic articles + cross-links.
                  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                           β”‚
                           β–Ό
                  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                  β”‚     knowledge/     β”‚   LLM owns; you and your agents read.
                  β”‚  concepts/         β”‚
                  β”‚  connections/      β”‚   ← cross-source links
                  β”‚  people/ projects/ β”‚
                  β”‚  qa/               β”‚
                  β”‚  facts/            β”‚   ← human-owned hard facts (override sources)
                  β”‚  MOCs/             β”‚   ← curated topic hubs (Maps of Content)
                  β”‚  index.md          β”‚   ← master catalogue
                  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

After every compile, two side loops run on the new article:

  • Curiosity loop β€” a small local Ollama model spots gaps and writes JSON deep-scan requests to raw/requests/. The next compile cycle picks one up and fills the gap.
  • Optimization suggestions β€” the compiler emits YAML proposals to raw/suggestions/ for repeatable manual actions (e.g. mail filter rules). wiki suggestions applies them only after explicit per-action approval.

When raw sources contradict reality β€” a Slack thread that calls a project by its old name, an old memo that claims a never-won award β€” wiki correct lets you write a hard fact to knowledge/facts/<slug>.md. Hard facts inject into compile + query prompts at the highest authority, so future compilations honour them automatically. wiki correct apply <slug> then spawns an agent that walks the existing wiki, strikes contaminated claims, fixes wikilinks, and β€” for disambiguation facts β€” renames files. Raw sources stay immutable; only knowledge/ (and minimal correction notes in daily/) are touched.

More passes over the compiled wiki, each in docs/cli.md:

  • wiki reconcile β€” fixes concepts that contradict a hard fact, write-scoped to knowledge/concepts/; dry-run by default and off until you opt in.
  • wiki health-trends β€” a deterministic ## Trends block over the health corpus ($0, no LLM); off by default.
  • wiki dedup β€” finds the duplicate entity pages speech-to-text leaves behind; every merge is operator-confirmed.
  • wiki dream web-research β€” a labelled ## Public Profile for public people via Exa; opt-in per vault and per entity.
  • wiki usage β€” where the model budget goes, in tokens per provider/model, never dollars.

What a compiled article looks like

A real knowledge/concepts/agent-config-staleness.md from a working vault β€” illustrative excerpt:

---
title: "Agent Config Staleness Pitfall"
aliases: [stale-claude-md, stale-agent-config, wrong-base-pr-incident]
tags: [agents, configuration, claude-md, incident]
sources:
  - "daily/2026-04-16.md"
  - "daily/2026-04-24.md"
created: 2026-04-16
updated: 2026-05-02
---

# Agent Config Staleness Pitfall

Agent-facing configuration files (CLAUDE.md, AGENTS.md) are *executed* by
agents, not just *read*. When they contain stale facts about the repo
(default branch, layout, conventions), agents act on those stale facts and
produce broken artifacts …

## Key Points

- **CLAUDE.md is not documentation β€” it's instructions.** Agents follow it
  like a runbook. Outdated facts produce outdated actions. Freshness is the
  counter-pattern to [[concepts/documentation-redundancy-for-agents]].

## Sources

- [[daily/2026-04-16.md]] β€” Session `fd9195f9` (18:08): PR #121 rejected
- [[daily/2026-04-24.md]] β€” A2A skill hallucinated obsolete UI path

Three things to notice:

  • Frontmatter is structured (aliases, tags, sources, dates) so Dataview queries hit it cleanly.
  • [[wikilinks]] point both into the wiki (concepts/...) and back to durable sources (daily/..., raw/notes/..., raw/articles/..., raw/transcripts/...) β€” the audit trail is part of the article, not a metadata field.
  • The article is atomic. It argues one idea, cites two different days' sessions, and links to sibling concepts. The compiler chose this granularity from the raw substrates; nothing is hand-curated.

The defining choice: compile once, query fast

Knowledge is distilled into Markdown wikilinks at compile time β€” no embedding step, no retrieval at every query.

Approach Cost per query Latency Cross-doc reasoning
RAG re-embed + retrieve every time seconds weak β€” chunks are isolated
llm-wiki one-time compile per source ms β€” it's already markdown strong β€” LLM saw all sources during compile

Speed compounds. Every query is a Markdown read, so the wiki ends up read more often than it's written β€” by you, and by every agent you give vault access to. That inversion (output β†’ input ratio greater than 1) is the point.

The architecture wrapped around Karpathy's shape and Cole Medin's session capture β€” collectors, two-path ingest, curiosity loop, suggestions, lint, hooks across multiple agents, the engine/vault split β€” is the work of this project. Full design rationale in docs/concept.md; hard-won engine learnings (Ollama gotchas, rate-limit debugging, anti-patterns) in .ytstack/KNOWLEDGE.md.

Engine vs. vault split

<vault>/
β”œβ”€β”€ AGENTS.md              ← article-schema spec (seeded from templates/, then yours)
β”œβ”€β”€ dashboard.md           ← Dataview home page (seeded from templates/, then yours)
β”œβ”€β”€ raw/                   ← immutable curated sources (LLM reads, never writes)
β”œβ”€β”€ daily/                 ← per-day rollup: <date>/<source>.md per-source captures + <date>.md compile-stage digest
β”œβ”€β”€ knowledge/             ← LLM-compiled wiki (LLM owns, you and agents read)
β”œβ”€β”€ reports/               ← operator self-reports (air-gapped from compile): studies/<id>/runs/<ts>/instruments/*.md + _summary.md + _analysis.md, plus analyses/ for Pass-2
β”œβ”€β”€ inbox/                 ← transient β€” `wiki process-inbox` classifies + moves to raw/
β”œβ”€β”€ Clippings/             ← optional β€” Obsidian Web Clipper drop point
β”œβ”€β”€ .obsidian/             ← Obsidian config (community-plugins, core-plugins seeded)
β”œβ”€β”€ .claude/skills/        ← symlinks to .wiki/skills/<name> (auto-discovered)
└── .wiki/                 ← engine β€” hidden from Obsidian, never modified by hand

The vault holds the data. .wiki/ holds the engine. The two never mix on disk. The engine's internal layout is documented in docs/engine-layout.md.

Install

curl -fsSL https://raw.githubusercontent.com/lx-0/llm-wiki/main/install.sh | bash
# or with explicit target:
curl -fsSL https://raw.githubusercontent.com/lx-0/llm-wiki/main/install.sh | bash -s -- ~/path/to/vault

The installer clones into <target>/.wiki/, seeds config.yaml from config.example.yaml, and runs wiki seed. That first wiki call creates the Python environment at <target>/.wiki/.venv/ (or at ~/.venvs/<vault>-wiki when the vault sits in iCloud Drive / CloudStorage β€” cloud sync evicts files and kills running binaries; the engine addresses that environment via UV_PROJECT_ENVIRONMENT). The seed fills the vault root from .wiki/templates/ β€” but only when each target file is absent, never overwriting existing work β€” and wiki skills sync links the engine's skills.

Created path Source Purpose
<vault>/AGENTS.md templates/AGENTS.example.md Article-schema spec read by every compile prompt. Edit the Vault Owner + Language sections.
<vault>/dashboard.md templates/dashboard.md Obsidian Dataview home (recently-compiled / wiki stats / recent daily logs).
<vault>/.obsidian/{community,core}-plugins.json templates/.obsidian/ dataview + obsidian-excalidraw-plugin for first-launch approval; core defaults (daily-notes, properties, graph on; sync/publish off).
<vault>/.claude/skills/use-llm-wiki symlink to .wiki/skills/use-llm-wiki The one bundled engine-side skill β€” lets agents query, contribute, or diagnose this wiki via the wiki CLI from any project. Global-eligible: wiki skills install --global links it into ~/.claude/skills/ so agents anywhere can reach it. The other operator skills (engine-pr, excalidraw-diagram, ingest-audio, vault-health-check, vault-triage) ship via the Claude Code plugin marketplace yesterday-public-plugins, not as repo-bundled skills.

Prerequisites: bash, git, uv (Python package manager). Optional but recommended: a local Ollama β€” the curiosity loop, screenshot vision, inbox classification, HTML visual analysis, and per-article review all run on local models. The Claude paths (compile, query, lint contradiction check, flush) work without it.

After install:

cd ~/path/to/vault
./.wiki/wiki setup        # 6-question wizard (Ollama URL, compile model, compile-after hour, …)
./.wiki/wiki status       # config + hook install table + Ollama probe
./.wiki/wiki              # interactive home screen β€” what's pending + browse by group

Update

./.wiki/wiki update                # git pull --ff-only + sync skill symlinks into .claude/skills/
./.wiki/wiki skills status         # per-skill linked / collision / missing table + global state
./.wiki/wiki skills install        # ad-hoc resync without pulling
./.wiki/wiki skills install --global  # also link use-llm-wiki into ~/.claude/skills/ + register vault
./.wiki/wiki seed                  # additive: add missing vault templates (dashboard, plugin configs)
./.wiki/wiki seed --force          # overwrite existing templates with engine versions

wiki update pulls into .wiki/ (preserves config.yaml, re-syncs the environment where it lives), then runs wiki skills sync so newly-shipped engine skills land in <vault>/.claude/skills/ automatically. Foreign entries (your own skills, other tools' symlinks) are never touched. When skills.global_install is on, the sync also refreshes the global ~/.claude/skills/ symlink β€” the opt-in survives updates with no re-flagging.

If the .wiki/ checkout has uncommitted changes to tracked engine files (direct edits inside <vault>/.wiki/ β€” which a --ff-only pull would otherwise refuse), wiki update lists them and offers to git stash them so the pull can proceed. After pulling it re-applies the stash automatically, but only if it merges cleanly onto the new engine; if it would conflict, the stash is left unpopped (recover with git -C <.wiki> stash pop). Run it from a terminal β€” non-interactively (e.g. dashboard button) it refuses rather than touching your changes. (config.yaml, state/, logs/ are gitignored and never trigger this.)

wiki seed re-applies engine templates to the vault root after an update β€” adds missing files (dashboard.md, _dashboard-stats.md, .obsidian/plugins/<name>/data.json) and merges community-plugins.json (additive β€” never drops your own plugins). Default mode never overwrites your existing files; use --force to replace customisations of dashboard.md / AGENTS.md with the engine version.

Commands

Bare ./.wiki/wiki opens the home screen: a status line (384 articles Β· last compile 4h ago Β· ollama βœ“), what is pending in your vault (3 files in inbox/ β†’ process-inbox), a health banner when something is broken (missing hooks, unreachable Ollama, recent compile errors) with the fix, the shortcut keys, and the command groups to browse β€” the same groups wiki help prints, from one command table. / filters every command. Without a terminal (CI, hooks, pipes) it prints the help and exits. Agents read the same state as JSON: wiki menu --json (what's pending) and wiki doctor --json (config, connectivity and pipeline audit; --quick for hook use). Run engine code through wiki, not a bare uv run: on a cloud-synced vault that rebuilds the environment inside the vault. Full reference β€” every subcommand, every config key, every hook target β€” in docs/cli.md.

Documentation map

Doc What's inside
docs/PRINCIPLES.md The ten core rules β€” constitution of the wiki, with verbatim Karpathy / OpenClaw / own quotes + attribution
docs/concept.md Three-layer architecture, compile-vs-RAG, cognitive-function mapping, curiosity loop
docs/PROCESS.md (German) Live documentation of every data flow inside the engine β€” numbered processes (German prose, English diagrams; a PR that touches a flow keeps the prose language)
docs/cli.md Full CLI reference β€” every wiki <subcommand>, the home screen, every hook target
docs/config.md Every config key, generated from the schema
docs/FEATURES.md Implementation map of every engine feature β€” status, code location, trigger, known gaps. Maintained alongside code.
docs/engine-layout.md File-by-file tree of .wiki/ β€” the engine internals
docs/naming.md Naming conventions for raw sources and knowledge articles
docs/architecture.png Full Excalidraw render of the cognitive architecture
docs/meeting-collectors-architecture.md Jamie and Wispr direct HTTP collectors: authentication, source capture, daily provenance and compilation
AGENTS.md Conventions for AI agents working on this codebase (separate from the vault's own AGENTS.md)
.ytstack/PROJECT.md Project framing, success criteria, current status
.ytstack/DECISIONS.md Locked architectural choices
.ytstack/KNOWLEDGE.md Hard-won engine learnings (Ollama, rate limits, anti-patterns)

Security

Secrets never enter git: gitleaks runs in CI on every push and PR and nightly (secrets-scan.yml) and blocks the merge on a leak, and the pre-commit hooks run it before a commit (pip install pre-commit && pre-commit install). Rules and allowlist: .gitleaks.toml; a manual scan is gitleaks detect --no-banner -v. If you find a leak in history, rotate the secret immediately, then file an issue (don't post the leaked value). The code gates β€” ruff, ty, the size gates, the pytest suite β€” are listed in AGENTS.md.

For contributors

Engine internals (file-by-file tree, how a module runs) live in docs/engine-layout.md. Development conventions β€” how to add an agent target, a tunable, or a prompt; style, gates and side-effect rules β€” live in AGENTS.md.

About

Self-cartography engine: Turn your scattered personal data into a queryable Obsidian wiki. You and your agents read from the same surface.

Topics

Resources

Contributing

Security policy

Stars

4 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages