Lorekeeper

An LLM storytelling engine that never forgets what happened.

LLMs write great narrative text, but they have no memory. Run a multi-turn story session and sooner or later the model kills a character who's already dead, puts someone in a location they can't reach, or forgets who's allied with whom. The longer the story goes, the worse it gets.

Lorekeeper fixes this with a read-write knowledge graph loop. After each generated segment, an extraction pipeline pulls entities (characters, locations, events, objects) into a Neo4j property graph. Before the next generation, the system retrieves relevant facts from that graph and injects them as hard constraints into the prompt. A contradiction guard runs 5 Cypher-backed checks and warns the LLM before it writes, not after.

In a 20-segment paired evaluation against a rolling-text baseline, Lorekeeper cut contradiction scores by 61% (1.70 vs 4.35 mean weighted score). The baseline accumulated 3 Critical and 36 Major contradictions; NKGE held at 1 Critical and 14 Major. The longer the story runs, the wider the gap grows. The graph isn't just storage; it's actively steering the LLM away from inconsistent output.

A few of the bigger decisions behind this design:

• Dual RAG. Graph RAG (Cypher queries) pulls structured facts the LLM must respect. Vector RAG (ChromaDB) pulls tonal anchors for style matching. They go into different prompt sections with different instruction framing because they serve different purposes.
• Propose-Validate-Commit. The LLM proposes entity extractions with confidence scores. Deterministic Python code validates them (fuzzy name dedup, status consistency, confidence thresholds). Only approved proposals get MERGEd. The LLM proposes; it doesn't decide.
• Pre-generation guard. Five constraint checks run before generation and inject warnings into the prompt. In strict mode, the system retries up to twice, then branches the timeline to isolate the divergence rather than corrupt the main storyline.
• Persona-differentiated generation. Each character has a ChromaDB-stored voice profile (speech patterns, emotional baseline, knowledge boundaries) injected separately from facts and tonal context.

Same player actions, same seed story, 20 segments across a 4-act narrative arc. NKGE vs. a rolling-text baseline that keeps the last 3 segments as context. Many production systems for story generation these days utilize rolling text baselines with condensed summaries, so the baseline is representative of operational modes of many AI storytelling companies.

The baseline degraded steadily over 20 segments, averaging a contradiction score of 4.35 per segment with 3 Critical failures (dead characters acting, impossible object states). NKGE held at 1.70 mean, with 8 out of 20 segments fully clean. The gap between the two systems widened as the story grew more complex, which is exactly what graph memory is designed to address.

The coherence trade-off is small: -0.20 points. At 5 segments it was -0.60, meaning the system's coherence actually improves relative to baseline as it accumulates more graph context. That's a tunable dial that can be adjusted to adjust the system severity, by design.

Full methodology, severity weights, and reproduction steps are in the design document.

Per-segment contradiction scores across all 20 segments. The baseline (red) stays consistently high and gets worse in the later acts as the story accumulates more facts to potentially violate. NKGE (green) stays mostly below the baseline, with 8 fully clean segments:

The radar chart below shows the full trade-off profile across four dimensions. NKGE dominates on consistency and retrieval precision, matches on graph coverage, and trades a small amount of coherence for factual accuracy:

Five checks run against the live graph before every generation:

Here's what the hostile copresence guard looks like in practice:

Guard injected: "Kael and Maren have a hostile relationship. Their interaction should reflect this tension — do not write them as friendly or cooperative without justification."

NKGE output: Kael's chair scraped against the wooden floor as he stood. "Thornwood Bridge," he said, his voice cutting through the smoky air. "Wounded traveler came through here bleeding. Said someone with your... particular skills hit his group hard." Maren didn't flinch. "Bridges see a lot of traffic. Dangerous places... accidents happen."

The guard didn't block the interaction. It made sure the dialogue carried the right tension. The LLM adjusted its tone on its own once the constraint was in the prompt.

The graph grows with every generation turn. Node colors:

• Blue = Character | Green = Location | Orange = Event | Grey = Object | Purple = Segment | Red = Faction

The structural relationships that tie things together:

• PARTICIPATED_IN links characters to the events they appeared in
• CAUSED_BY chains events into a causal timeline
• LOCATED_AT tracks where characters currently are
• OWNS tracks object ownership as it changes hands
• KNOWS captures character relationships, including sentiment (allied, hostile)

These edges are all created deterministically during extraction. The system doesn't rely on the LLM to propose them.

• Python 3.12+
• Neo4j Desktop (Community or Enterprise) running locally on bolt://localhost:7687
• Anthropic API key for Claude Sonnet

git clone https://github.com/your-username/lorekeeper.git
cd lorekeeper
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt

cp .env.example .env
# Edit .env with your Anthropic API key and Neo4j credentials

Run these in order. Each one is self-contained:

notebooks/01_schema_and_ingest.ipynb   # Create schema, ingest seed story
notebooks/02_extraction_pipeline.ipynb  # Demo extraction with HITL review
notebooks/03_dual_retrieval_pipeline.ipynb  # Full pipeline end-to-end
notebooks/04_evaluation_harness.ipynb   # Paired NKGE vs. baseline eval

streamlit run app.py

uvicorn api:app --host 0.0.0.0 --port 8000 --reload
# API docs at http://localhost:8000/docs

pytest tests/ -v

lorekeeper/
├── notebooks/
│   ├── 01_schema_and_ingest.ipynb      # Schema setup + seed story ingestion
│   ├── 02_extraction_pipeline.ipynb    # Two-stage extraction demo
│   ├── 03_dual_retrieval_pipeline.ipynb # Full pipeline with dual RAG
│   └── 04_evaluation_harness.ipynb     # Paired evaluation runner
├── src/
│   ├── schema.py          # Pydantic v2 models for the ontology and evaluation types
│   ├── graph_client.py    # Neo4j wrapper with typed queries, MERGE helpers, enrichment
│   ├── extraction.py      # Propose, validate, commit pipeline with auto-linking
│   ├── retrieval.py       # Tiered Cypher (T1-T4) + ChromaDB vector retrieval
│   ├── guard.py           # 5-check contradiction guard + branch manager
│   ├── persona.py         # Character voice profiles via PersonaStore + PersonaGenerator
│   ├── pipeline.py        # LangGraph StateGraph orchestration (read-write-verify loop)
│   ├── prompts.py         # All prompt templates with version tracking
│   ├── eval.py            # LLM judge, paired runner, metric computation
│   └── tracing.py         # OpenTelemetry instrumentation
├── tests/                 # 137 unit tests across 8 test files
├── eval_runs/             # Persisted evaluation artifacts (JSON)
├── app.py                 # Streamlit interactive frontend
├── api.py                 # FastAPI REST API (6 endpoints, OpenAPI docs)
├── prompts_registry.json  # Prompt version governance registry
├── NKGE_Project_Design_Document.md  # Full system design document
├── requirements.txt
└── .env.example

(:Character)-[:LOCATED_AT]->(:Location)
(:Character)-[:KNOWS {sentiment}]->(:Character)
(:Character)-[:PARTICIPATED_IN]->(:Event)
(:Character)-[:OWNS]->(:Object)
(:Character)-[:MEMBER_OF]->(:Faction)
(:Event)-[:CAUSED_BY]->(:Event)
(:Segment)-[:REFERENCES_GRAPH_STATE]->(*)

All writes go through MERGE (never CREATE) so operations are idempotent. Every edge carries a branch_id to keep timelines isolated from each other.

Token counting uses tiktoken's cl100k_base encoding as a proxy for Claude's tokenizer. The variance is roughly 5%, which is close enough for budget enforcement.

Every prompt lives in src/prompts.py and is version-tracked in prompts_registry.json. Changing a prompt means documenting the rationale and recording eval score deltas before the new version gets promoted. No inline prompt strings exist anywhere else in the codebase.

OpenTelemetry traces wrap each generation cycle. The spans carry structured attributes that make it straightforward to debug slow or misbehaving runs:

• retrieval.graph_tokens, retrieval.vector_tokens for context budget usage
• guard.violation_count, guard.severity.* for constraint check outcomes
• generation.output_tokens, generation.retry_count for LLM call stats
• extraction.proposed/approved/flagged/committed for pipeline throughput

Console exporter during development; swap in an OTLP exporter to ship traces to a production collector.

Interactive docs at http://localhost:8000/docs via FastAPI's Swagger UI.

Every design choice in this system is aimed at one thing: showing that structured memory measurably improves LLM output quality.

Some of the opinions baked in:

• The baseline is realistic (rolling text summary of recent segments), not a trivially weak comparison
• Contradictions carry severity weights tied to narrative impact, so a dead-character error counts more than a tone mismatch
• The coherence trade-off is tracked openly, not swept under the rug
• Guard constraints fire before generation, not after, because it's cheaper and more effective to prevent bad output than to detect it
• The extraction pipeline treats LLM output as untrusted input with multi-stage JSON recovery and deterministic validation
• Environmental detail objects (things like "morning mist" or "hoofprints") get extracted and stored but intentionally remain structurally unlinked; that's correct behavior, not a gap

The full design rationale is in the design document.

MIT