Skip to content

Repository files navigation

gracedb

Go Embedded AI Memory + Knowledge Graph Database

Overview

gracedb is a Go embedded AI memory and knowledge graph database built on a pluggable KV backend (Badger by default). It provides vector search, full-text search, knowledge management, session management, property graphs, RDF/SPARQL, and MCP services — all in a single, zero-external-dependency library.

Maturity

gracedb's core engine is mature: the storage, vector search, FTS, semantic memory, graph, RDF, and MCP surfaces are implemented and covered by nearly ten thousand lines of Go tests and benchmarks. The advanced AI workflows built on top are still simplified or caller-extended, and the whole system has not yet been validated under production load. Treat the project as suitable for local applications, experiments, and controlled internal use before hardening it for production.

Stable Core

Embedder interface, text auto-vectorization, Quick API, vector CRUD, HNSW/IVF/Flat/LSH indexes, full-text search, metadata filtering, semantic Agent Memory, knowledge storage, session/document management, backup/restore, and OpenTelemetry hooks.

Experimental Or Simplified

Area Current state
LLM-driven entity extraction Built-in extraction is heuristic; callers can inject LLM extractors.
LLM Reflect Built-in reflection is rule-based; callers can inject LLM reflectors.
SPARQL SELECT/ASK with FILTER (comparison), ORDER BY, OFFSET, LIMIT, {...} UNION {...} alternation, OPTIONAL { } left-outer-join groups, GROUP BY with seven aggregate functions (COUNT/SUM/MIN/MAX/AVG/GROUP_CONCAT/SAMPLE), AS alias projection and HAVING group filtering. Full SPARQL 1.1 (aggregates over subqueries, federated queries, etc.) is not yet supported.
GraphRAG workflows Useful pipeline primitives, not a full managed RAG product.
Index recovery Stored vectors are durable and searchable after reopen; indexes load lazily on first search (from snapshot or rebuilt from vectors), auto-rebuilding from vectors if a snapshot is corrupt; production deployments should still add crash/recovery validation for their workload.

Documentation

Document Content
Getting Started Installation, configuration, quick start, FAQ
API Reference Complete API, data types, storage key formats
Architecture Layered architecture, data flow, index system, extension points
Module Guide Package responsibilities, key types, dependency graph
Implementation Plan Phased development roadmap
Advanced Memory Memory strength model, consolidation, layered wake-up, FTS rebuild
中文版 中文文档

Features

  • Vector Search — HNSW / IVF / Flat / LSH indexes with cosine similarity; HNSW falls back to an exact scan on small collections for deterministic results
  • Full-Text Search — Chinese segmentation + stop words + Levenshtein fuzzy + RRF hybrid fusion
  • Knowledge Storage — auto-chunking + FTS indexing + document-level aggregation
  • Agent Memory — scope/namespace/TTL isolation with semantic + lexical hybrid retrieval
  • Property Graph — node/edge CRUD, BFS/DFS/shortest path traversal
  • RDF/SPARQL — N-Triples import/export, SPARQL SELECT/ASK, GROUP BY aggregation, RDFS inference, SHACL validation
  • GraphRAG Toolbox — 9 built-in tools for LLM orchestration
  • MCP Service — full Model Context Protocol server (tools + resources + prompts + capability negotiation), stdio transport
  • Backup/Restore — protocol-agnostic snapshot, restorable across storage engines
  • OpenTelemetry — automatic spans and metrics on core operations
  • KnowledgeMemory — fused memory + knowledge recall with reflection and consolidation
  • Auto-Retain — automatic fact extraction during conversation

Quick Start

go get github.com/dshmyz/gracedb
package main

import (
    "fmt"
    "log"

    "github.com/dshmyz/gracedb/pkg/gracedb"
)

func main() {
    // Open database
    db, err := gracedb.Open("/tmp/gracedb-data")
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    // Create collection
    coll, _ := db.CreateCollection("my_docs")
    fmt.Println("created:", coll.Name)

    // Insert vector
    vec := []float32{0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8}
    embID, _ := db.Upsert("my_docs", "doc-1", vec, "Hello world", nil, nil)
    fmt.Println("embedded:", embID)

    // Search
    results, _ := db.Search("my_docs", vec, gracedb.SearchOptions{
        TopK:            5,
        UseVectorSearch: true,
    })
    fmt.Println("found:", len(results), "results")

    // Backup
    db.Backup("/tmp/gracedb-backup.db")

    // Quick API
    q := db.Quick()
    id, _ := q.AddToCollection(ctx, "my_docs", vec, "Quick add")
    fmt.Println("quick add:", id)
}

Configuration

db, _ := gracedb.Open("/tmp/data",
    gracedb.WithIndexType("hnsw"),         // hnsw / ivf / flat / lsh
    gracedb.WithIndexTypes([]string{"hnsw", "lsh"}), // multi-index hybrid
    gracedb.WithSimilarity("cosine"),       // cosine / euclidean
    gracedb.WithEmbedder(myEmbedder),       // types.Embedder interface
)

Architecture

┌─────────────────────────────────────────────┐
│                gracedb.DB                    │  ← Facade
│  Quick / Toolbox / Backup / Trace / Ontology │
├─────────────────────────────────────────────┤
│              KnowledgeMemory                 │  ← Recall/Reflect/Consolidate
│  AutoRetain / GroupAggregate                 │
├─────────────────────────────────────────────┤
│         BadgerStore                          │  ← Persistence (with in-memory index)
│  CRUD / Search / FTS / Index / Aggregation   │
├─────────────────────────────────────────────┤
│         GraphStore / RDF                     │  ← Graph engine
│  Nodes/Edges/Traversal/SPARQL/RDFS/SHACL     │
├─────────────────────────────────────────────┤
│          KVBackend interface                 │  ← Pluggable storage seam
│  Update / View / Backup / Restore / Sync     │
├─────────────────────────────────────────────┤
│         badgerKV  |  memkv (in-memory)       │  ← Storage engines
│  LSM-tree / MVCC / ACID  |  zero-dep         │
└─────────────────────────────────────────────┘

The storage engine is fully pluggable through the KVBackend interface (pkg/store/kv.go). By default gracedb uses badgerKV (Badger v4); pass store.NewMemKV() via WithKVBackend (pkg/gracedb/db.go) to run on a pure in-memory engine:

Usage Examples

Text Auto-Vectorization

db, _ := gracedb.Open("/tmp/data", gracedb.WithEmbedder(myEmbedder))

// Insert text (auto-vectorized)
id, _ := db.InsertText("docs", "text-1", "This is Chinese text", nil)

// Text search (vectorized or FTS fallback)
results, _ := db.SearchText("docs", "Chinese", 10)

Knowledge Management

// Save knowledge (auto-chunked)
record, _ := db.SaveKnowledge("docs", "wiki-1", "Go Language",
    "Go is a statically typed, compiled language...",
    types.KnowledgeSaveRequest{ChunkSize: 500, ChunkOverlap: 50})

// Search knowledge
resp, _ := db.SearchKnowledge("docs", "programming language", 5)

Agent Memory

// Save memory with scope/namespace
db.SaveMemory(types.MemorySaveRequest{
    MemoryID:  "mem-1",
    Content:   "User prefers Go over Python",
    Scope:     "user",
    UserID:    "user-123",
    Namespace: "preferences",
    TTLSeconds: 3600,
})

// Search memory
resp, _ := db.SearchMemory(types.MemorySearchRequest{
    Query:  "prefers",
    UserID: "user-123",
    Scope:  "user",
    TopK:   5,
})

Property Graph

g := db.Graph()

g.UpsertNode(&graph.GraphNode{
    ID: "person-1", Type: "person",
    Properties: map[string]string{"name": "Alice"},
})
g.UpsertNode(&graph.GraphNode{
    ID: "lang-1", Type: "language",
    Properties: map[string]string{"name": "Go"},
})
g.UpsertEdge(&graph.GraphEdge{
    FromNodeID: "person-1", ToNodeID: "lang-1",
    Type: "likes", Weight: 1.0,
})

// Query neighbors
nodes, edges, _ := g.GetNeighbors("person-1", graph.NeighborOptions{
    Direction: "out", Limit: 10,
})

// Traversal
result, _ := g.BFS("person-1", graph.NeighborOptions{MaxDepth: 2})

RDF/SPARQL

rdf := db.RDF()

// Insert triple
rdf.UpsertTriple(&rdf.Triple{
    Subject:   rdf.NewIRI("http://example.org/person/1"),
    Predicate: rdf.NewIRI("http://example.org/likes"),
    Object:    rdf.NewIRI("http://example.org/lang/go"),
})

// SPARQL query
results, _ := rdf.SPARQLSelect(`SELECT ?s ?p ?o WHERE { ?s ?p ?o . }`)

// Solution modifiers: FILTER, ORDER BY, LIMIT
top, _ := rdf.SPARQLSelect(
    `SELECT ?person ?name WHERE { ?person <http://schema.org/name> ?name FILTER(?name >= "A") } ORDER BY DESC(?name) LIMIT 3`)

// ASK query
exists, _ := rdf.SPARQLAsk(`ASK WHERE { <http://example.org/person/1> ?p ?o . }`)

// GROUP BY aggregation: count items per type, keep groups with count > 1
counts, _ := rdf.SPARQLSelect(`
    SELECT ?type (COUNT(?item) AS ?c) WHERE {
      ?item <http://ex.com/type> ?type
    } GROUP BY ?type HAVING (COUNT(?item) > 1)`)

Ontology Management

o := db.Ontology()

// Define ontology
o.DefineClass("http://example.org/Person", "")
o.DefineClass("http://example.org/Developer", "http://example.org/Person")
o.DefineProperty("http://example.org/knows", "http://example.org/Person", "http://example.org/Person")

// Add facts
o.AddFact("http://example.org/person/alice", "http://example.org/knows", "Bob")

// RDFS inference (materialize new triples)
count, _ := o.Infer()

// SHACL validation
report, _ := o.Validate()

KnowledgeMemory (Recall/Reflect/Consolidate)

km := db.KnowledgeMemory(nil) // nil = use rule-based reflector

// Recall: fused memory + knowledge + graph expansion
resp, _ := km.Recall(ctx, knowledge.KnowledgeMemoryRecallRequest{
    Query:         "what does the user like?",
    TopKMemories:  5,
    TopKKnowledge: 4,
    MaxHops:       2, // graph expansion depth
})

// Reflect: synthesize structured summary
reflection, _ := km.Reflect(ctx, knowledge.KnowledgeMemoryReflectRequest{
    Query: "user preferences",
})
fmt.Println("Summary:", reflection.Summary)
fmt.Println("Themes:", reflection.Themes)

// Consolidate: store summary + optionally promote to knowledge
consolidated, _ := km.Consolidate(ctx, knowledge.KnowledgeMemoryConsolidateRequest{
    Reflect: knowledge.KnowledgeMemoryReflectRequest{
        Query: "user preferences",
    },
    PromoteToKnowledge: true,
})

Auto-Retain (Automatic Fact Extraction)

db.SetFactExtractor(func(ctx context.Context, msgs []*types.Message) ([]gracedb.ExtractedFact, error) {
    // Extract facts from conversation (can use LLM here)
    return []gracedb.ExtractedFact{
        {ID: "fact-1", Content: "User likes Go", Type: "preference"},
    }, nil
})

db.SetAutoRetain(gracedb.AutoRetainConfig{
    Enabled:      true,
    WindowSize:   6,
    TriggerEvery: 2, // extract every 2 messages
})

// AddMessage now triggers extraction automatically
db.AddMessage(&types.Message{SessionID: "sess-1", Role: "user", Content: "I like Go"})
db.AddMessage(&types.Message{SessionID: "sess-1", Role: "assistant", Content: "Go is great!"})
// → AutoRetain fires, extracts facts, stores as memory

Aggregation

// Simple aggregation
result, _ := db.Aggregate("docs", "score", store.AggAvg)
fmt.Printf("Average score: %.2f\n", result.Avg)

// GROUP BY aggregation
groups, _ := db.GroupAggregate("docs", "category", "price", store.AggAvg)
for category, r := range groups {
    fmt.Printf("%s: avg=%.2f, count=%d\n", category, r.Avg, r.Count)
}

MCP Server

db.MCPServer() composes the tools, resources and prompts clusters into one complete Model Context Protocol server — the drop-in entry point for standard MCP clients:

server := db.MCPServer()
server.RunStdio(context.Background())

A full server exposes:

  • Tools — the 15-tool GraphRAG toolbox (search_knowledge, save_memory, vector_search, sparql_query, …), callable from any MCP client.
  • Resources — addressable gracedb:// URIs read live on demand, e.g. gracedb://memory/{id}, gracedb://knowledge/{collection}/{id}, gracedb://graph/node/{id}, gracedb://rdf/triple/{id} and gracedb://collections.
  • Prompts — four templates (memory_augmented_reply, knowledge_qa, consolidation, graph_rag_expand) that render to message sequences with live database context.
  • Capabilities — an initialize handshake advertises the enabled clusters; pass gracedb.WithCapabilities(tools, resources, prompts bool) when opening the DB to trim them.

For a tools-only server (backward compatible), keep using db.NewMCPServer(name, version).

Backup & Restore

db.Backup("/tmp/backup.db")

db2, _ := gracedb.Open("/tmp/restored")
db2.Restore("/tmp/backup.db")

Index Management

// Preload index (from snapshot or rebuild) — optional
db.LoadIndex("docs")

// Save index snapshot
db.SaveIndex("docs")

// Rebuild FTS index
db.RebuildIndex("docs")

Vector indexes are loaded lazily: the first search on a collection auto-builds the in-memory index from its persisted snapshot (or rebuilds from stored vectors), so forgetting to call LoadIndex no longer silently degrades to a slow full scan. LoadIndex is only needed if you want the index warm before the first query.

Storage Key Format

Key Pattern Purpose
coll:<name> Collection metadata
emb:<cid>:<eid> Embedding metadata
emb:vec:<cid>:<eid> Vector data
fts:<token>:<cid>:<eid> FTS inverted index (value = TF count)
mem:<bucket>:<id> Memory metadata
mem:content:<bucket>:<id> Memory content
mem:vec:<bucket>:<id> Memory semantic vector
mem:fts:<token>:<bucket>:<id> Memory lexical inverted index
mem:idx:<id> Memory ID to bucket index
graph:node:<id> Graph node
graph:edge:<id> Graph edge
rdf:t:<id> RDF triple
idx:snapshot:<cid> Vector index snapshot
sess:<id> Session

Examples

go run examples/main.go              # full API tour
go run examples/llm_reflector        # wire your own LLM KnowledgeMemoryReflector
go run examples/llm_extractor        # wire your own LLM entity/relation extractor
go run examples/sparql_modifiers     # SPARQL FILTER / ORDER BY / LIMIT via embedded Go API

Testing

go test ./...              # All tests
go test -v ./pkg/index/    # Verbose output
go test -bench=. ./pkg/store/  # Benchmarks

Acknowledgments

gracedb is inspired by and built upon the work of these excellent projects:

  • CortexDB — gracedb targets CortexDB feature parity as its design reference. The KnowledgeMemory, AutoRetain, GraphRAG toolbox, and MemoryFlow workflows are all modeled after CortexDB's architecture and APIs.
  • Badger — gracedb's default storage engine via the badgerKV backend. Badger's high-performance LSM-tree, MVCC, and ACID transaction support make gracedb's embedded design possible. The KVBackend interface allows swapping it for other engines (e.g. the in-memory memkv).
  • 智算多多 (zsdodo) — 致谢智算多多(zsdodo)提供算力支持。

License

MIT

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages