A single-binary data layer for AI agents. Structured tables, full-text search, and vector search — over your own storage, with schema management and migration built in.
A dolmen is an ancient stone table: durable, zero-maintenance, still standing after everyone who built it moved on. That's the operating model — one static Go binary, SQLite files on a volume, no database server to babysit.
Builders of agents and AI skills whose data has outgrown one process — shared memory, telemetry, user metrics, files — and who want it searchable and governed without taking on database operations. One server, three ways in:
- MCP clients — Claude, Cursor, anything that speaks MCP, over HTTP or stdio.
- HTTP clients — a plain JSON API, one operation per
POST /v1/{op}. - Go programs — in-process, as a library; no server at all.
With Go 1.26.11+ (Windows PowerShell variant in the Quickstart; a released container below needs only Docker):
git clone https://github.com/lsm/dolmen.git && cd dolmen
CGO_ENABLED=0 go build -o dolmen ./cmd/dolmen
./dolmen -addr 127.0.0.1:8790 -data ./dataOr run the released container, which needs nothing but Docker:
docker run --rm -it -p 127.0.0.1:8790:8790 -v dolmen-data:/data ghcr.io/lsm/dolmen:v0.7.0It is up when the health check answers:
curl -s http://127.0.0.1:8790/healthz
# {"status":"ok"}From there, First API calls create a table and search it, and MCP (agents) connects an agent host.
Building an AI skill or agent is fast now; holding its data is not. Any agent can spin up a local SQLite in a folder, but the moment you need centralized, multi-user storage — telemetry, memory, user metrics, blobs — you cross into a different profession: deploying, operating, backing up, and migrating a database service. Dolmen removes that burden: point it at a directory, get governed tables plus search plus an agent-native interface.
- Go 1.26.11+ (only to build; the binary is otherwise standalone).
- Git.
- For
vectorizefields and text queries insearch_vector: nothing extra by default. The built-inlocalprovider is enabled automatically and embeds in-process (it downloads the model on first use). Use an OpenAI-compatible endpoint instead by settingDOLMEN_EMBED_PROVIDER=openai.
git clone https://github.com/lsm/dolmen.git
cd dolmen
CGO_ENABLED=0 go build -o dolmen ./cmd/dolmen
./dolmen -addr 127.0.0.1:8790 -data ./dataOn Windows (PowerShell):
git clone https://github.com/lsm/dolmen.git
cd dolmen
$env:CGO_ENABLED = 0
go build -o dolmen.exe ./cmd/dolmen
.\dolmen.exe -addr 127.0.0.1:8790 -data ./dataThe first run creates the data directory (./data by default) and, with the default local
embedding provider, the model cache (./data/models). On Unix these are opened with owner-only
permissions (0700 for the directory, 0600 for files). On Windows the data directory gets a
protected, inheritable ACL granting full control to the running user, SYSTEM and
BUILTIN\Administrators and nothing inherited from its parent, reapplied on every open; namespace
files, nested namespace directories, the grant registry and the model cache inherit it. By default the server binds to
127.0.0.1:8790 and does not authenticate, so keep it on a private interface.
Embeddings are on by default. The built-in
localprovider downloadssentence-transformers/all-MiniLM-L6-v2(~90 MB) from the Hugging Face Hub the first time avectorizefield is written. Start the server and watch the log forlocal embedding model is not cachedto confirm the state. Pre-seed the cache, or download the model tarball from the releases page, for offline or HF-blocked installs (see Offline install).
Bash:
curl -s http://127.0.0.1:8790/healthzWindows PowerShell:
curl.exe -s http://127.0.0.1:8790/healthzExpected output:
{"status":"ok"}All successful /v1/{operation} POSTs return {"ok":true,"data":{...}}; errors return
{"ok":false,"error":{"code":"...","message":"...","request_id":"..."}} — the request id is the
request's X-Request-Id header when one was sent, otherwise a server-generated id, echoed back as
the X-Request-Id response header so any error can be correlated with the server log — with a
matching 4xx/5xx status. Error codes are stable for branching: invalid_request, not_found,
query_error, conflict, unauthorized, forbidden, embedder_unavailable, sql_engine_unavailable,
storage_unavailable, canceled, timeout, internal_error.
canceled means the request was cancelled before it completed (over the stdio transport, by a
client cancellation notification or the shutdown drain); the operation may or may not have
finished server-side — check with a query before retrying a write.
timeout means the server stopped the request at one of its time limits: 504 when an operation
ran past -op-timeout (or -migrate-timeout), 408 when the request body arrived too slowly. A
timed-out read can be narrowed and retried; a timed-out write may or may not have committed, so
check before retrying it.
On PostgreSQL, timeout (504) also answers a request that found the database at its connection
limit (SQLSTATE 53300); nothing was started, so it is safe to retry shortly, and its message says so.
embedder_unavailable (503) means the server's embedding provider could not load its model — with
the local provider, typically the first-use Hugging Face download failing — and its message names
the offline remediations (pre-seed the model cache, or point DOLMEN_EMBED_MODEL at a local model
directory); retrying the same request makes no sense until the model can load.
sql_engine_unavailable (503) is the lakehouse engine's counterpart: its DuckDB sidecar is missing
or failed to start, and the message says which (check -duckdb-sidecar and -duckdb-extensions).
A failed start is retried with a backoff of up to thirty seconds.
storage_unavailable (503) is reserved for the file store of file fields
(design): the store cannot take the write right now, and it may
be retried shortly.
The one exception is GET /v1/openapi.json, which serves the raw OpenAPI document.
/livez (and its alias /healthz) returns {"status":"ok"}, /readyz returns {"status":"ready", ...}, and /mcp returns JSON-RPC responses.
GET /metrics serves Prometheus text format: dolmen_operations_total{op,outcome} (outcome is ok or an error code such as forbidden or timeout), the dolmen_operation_duration_seconds{op} histogram, dolmen_operations_in_flight, dolmen_subscriptions_active, dolmen_uptime_seconds and dolmen_build_info{version}. Labels are bounded: only operation names and error codes, never namespaces, tables, principals or request ids, so the series count stays under a few hundred. A useful baseline alert is a rising rate of outcome="internal_error" or outcome="timeout", or p99 latency of a hot operation. Correlate a single request through the access log (-log-level debug) by its X-Request-Id.
These examples are shown with Bash curl. Windows PowerShell variants follow each command; use
curl.exe (PowerShell's curl is an alias for Invoke-WebRequest), double quotes around the -H
value, and single quotes around the one-line JSON payload with each inner " escaped as \" —
Windows PowerShell 5.1 strips unescaped inner quotes when invoking native commands, which corrupts
the JSON. Expected outputs are identical.
Create a table:
curl -s localhost:8790/v1/create_table -H 'Content-Type: application/json' -d '{
"namespace": "myapp", "table": "events",
"fields": [
{"name": "title", "type": "string", "fulltext": true},
{"name": "detail", "type": "text"},
{"name": "score", "type": "number"},
{"name": "embedding", "type": "vector", "dim": 4}
]
}'Windows PowerShell:
curl.exe -s http://127.0.0.1:8790/v1/create_table -H "Content-Type: application/json" -d '{\"namespace\":\"myapp\",\"table\":\"events\",\"fields\":[{\"name\":\"title\",\"type\":\"string\",\"fulltext\":true},{\"name\":\"detail\",\"type\":\"text\"},{\"name\":\"score\",\"type\":\"number\"},{\"name\":\"embedding\",\"type\":\"vector\",\"dim\":4}]}'Expected output (schema summary, version starts at 1):
{"ok":true,"data":{"table":{"namespace":"myapp","name":"events","version":1,"fields":[{"name":"title","type":"string","fulltext":true},{"name":"detail","type":"text"},{"name":"score","type":"number"},{"name":"embedding","type":"vector","dim":4}]}}}Insert a record:
curl -s localhost:8790/v1/insert -H 'Content-Type: application/json' -d '{
"namespace": "myapp", "table": "events",
"records": [{"title": "first bug", "detail": "token expiry not checked", "score": 0.75, "embedding": [0.5, 0.25, -0.5, 0.0]}]
}'Windows PowerShell:
curl.exe -s http://127.0.0.1:8790/v1/insert -H "Content-Type: application/json" -d '{\"namespace\":\"myapp\",\"table\":\"events\",\"records\":[{\"title\":\"first bug\",\"detail\":\"token expiry not checked\",\"score\":0.75,\"embedding\":[0.5,0.25,-0.5,0.0]}]}'Expected output:
{"ok":true,"data":{"ids":[1],"inserted":1}}Search full text:
curl -s localhost:8790/v1/search_fulltext -H 'Content-Type: application/json' -d '{
"namespace": "myapp", "table": "events", "query": "bug"
}'Windows PowerShell:
curl.exe -s http://127.0.0.1:8790/v1/search_fulltext -H "Content-Type: application/json" -d '{\"namespace\":\"myapp\",\"table\":\"events\",\"query\":\"bug\"}'Expected output:
{"ok":true,"data":{"results":[{"id":1,"created_at":"...","title":"first bug","detail":"token expiry not checked","score":0.75,"embedding":[0.5,0.25,-0.5,0.0]}],"truncated":false,"limit":10}}Raw vector search on a caller-supplied embedding column (no provider needed; text queries instead
require a vectorize field plus a provider):
curl -s localhost:8790/v1/search_vector -H 'Content-Type: application/json' -d '{
"namespace": "myapp", "table": "events", "vector": [0.5, 0.25, -0.5, 0.0]
}'Windows PowerShell:
curl.exe -s http://127.0.0.1:8790/v1/search_vector -H "Content-Type: application/json" -d '{\"namespace\":\"myapp\",\"table\":\"events\",\"vector\":[0.5,0.25,-0.5,0.0]}'Run read-only SQL:
curl -s localhost:8790/v1/query -H 'Content-Type: application/json' -d '{
"namespace": "myapp", "sql": "SELECT title, score FROM events WHERE score > ?", "args": [0.5]
}'Windows PowerShell:
curl.exe -s http://127.0.0.1:8790/v1/query -H "Content-Type: application/json" -d '{\"namespace\":\"myapp\",\"sql\":\"SELECT title, score FROM events WHERE score > ?\",\"args\":[0.5]}'Update rows matching a WHERE filter (and upsert inserts when nothing matches):
curl -s localhost:8790/v1/update -H 'Content-Type: application/json' -d '{
"namespace": "myapp", "table": "events", "filter": "score > ?", "args": [0.5],
"set": {"score": 0.5, "title": "triaged bug"}
}'Windows PowerShell:
curl.exe -s http://127.0.0.1:8790/v1/update -H "Content-Type: application/json" -d '{\"namespace\":\"myapp\",\"table\":\"events\",\"filter\":\"score > ?\",\"args\":[0.5],\"set\":{\"score\":0.5,\"title\":\"triaged bug\"}}'The local provider is enabled by default. To change or disable it, set DOLMEN_EMBED_PROVIDER.
Bash:
# Built-in local embeddings are the default — in-process inference, zero external services.
# It downloads sentence-transformers/all-MiniLM-L6-v2 (~90 MB) from the Hugging
# Face Hub on first use and caches it under the data dir; every later start reuses the cache.
# (To use a different local model, set DOLMEN_EMBED_MODEL.)
# Mixed-language/CJK data — the multilingual model (see "Choosing an
# embedding model" below; dolmen adds the e5 query:/passage: prefixes
# itself, callers do not):
DOLMEN_EMBED_PROVIDER=local \
DOLMEN_EMBED_MODEL=intfloat/multilingual-e5-small \
./dolmen
# Use an OpenAI-compatible endpoint instead (OpenAI, Ollama, vLLM):
DOLMEN_EMBED_PROVIDER=openai \
DOLMEN_EMBED_API_KEY=sk-... \
DOLMEN_EMBED_MODEL=text-embedding-3-small \
./dolmen
# OpenAI-compatible local endpoints (Ollama, vLLM) need the base URL:
DOLMEN_EMBED_PROVIDER=openai \
DOLMEN_EMBED_BASE_URL=http://localhost:11434/v1 \
DOLMEN_EMBED_MODEL=nomic-embed-text \
./dolmen
# Disable server-side embeddings entirely (caller must supply vectors):
DOLMEN_EMBED_PROVIDER=none ./dolmen
# Another local model (e.g. multilingual/CJK — see the full-text CJK caveat):
DOLMEN_EMBED_MODEL=sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 ./dolmenWindows PowerShell:
# OpenAI-compatible endpoint instead of the default local provider:
$env:DOLMEN_EMBED_PROVIDER = "openai"
$env:DOLMEN_EMBED_API_KEY = "sk-..."
$env:DOLMEN_EMBED_MODEL = "text-embedding-3-small"
.\dolmen.exe
# OpenAI-compatible local endpoints (Ollama, vLLM):
$env:DOLMEN_EMBED_PROVIDER = "openai"
$env:DOLMEN_EMBED_BASE_URL = "http://localhost:11434/v1"
$env:DOLMEN_EMBED_MODEL = "nomic-embed-text"
.\dolmen.exe
# Disable server-side embeddings:
$env:DOLMEN_EMBED_PROVIDER = "none"
.\dolmen.exe
# Another local model:
$env:DOLMEN_EMBED_MODEL = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
.\dolmen.exeWhichever provider is configured, describe_server reports its status read-only over both /v1
and MCP — provider, model, the identity that pins vectorized tables, whether server-side embedding
is usable, and (local only) what the next vectorized write will meet (model_state) — so the
active provider is visible without attempting a write. usable is
configuration status only: the provider is not called, so an endpoint that is down or rejects its
credentials still fails at first use, not here.
DOLMEN_EMBED_MODEL accepts any Hugging Face model id the rembed
runtime supports (or an absolute model-directory path). Two are packaged as release assets and
tested:
sentence-transformers/all-MiniLM-L6-v2(default) — English-only, 384-dim, fast, ~90 MB. The right choice for English-only data.intfloat/multilingual-e5-small— 100+ languages including Japanese/Chinese/Korean, also 384-dim, ~450 MB from the Hub (~270 MB as the release tarball). Use it when rows or queries mix languages: with the English-centric default, semantic recall is per-language only — an English "connection pool exhausted" query will not surface a Japanese 接続プール枯渏 incident.
The e5 prefix caveat is handled for you. e5-family models were trained with an asymmetric
retrieval contract — search text prefixed "query: " and stored text prefixed "passage: " —
and embed exactly the text they are given, so missing prefixes silently degrade ranking. Dolmen
adds the prefixes server-side: stored rows are embedded as passage: <field value>, search_vector
text queries as query: <text>, for any model whose own name segment carries e5
(intfloat/e5-*, intfloat/multilingual-e5-*, and their offline directory forms; instruct-tuned
variants are excluded — they need different prompts). Callers never add or see the prefixes, and
symmetric models — the MiniLM default, the sentence-transformers/paraphrase-multilingual-*
family — get nothing prepended. Other asymmetric families (bge, arctic) use longer
model-specific instructions dolmen does not know; avoid them.
Switching models re-embeds through migrate. Every vectorized table records the identity of
the space it was embedded in — local/<model> for symmetric models (e.g.
local/sentence-transformers/all-MiniLM-L6-v2), and a versioned marked form when the e5 prefix
contract is active (e.g. local/v2:intfloat/multilingual-e5-small#e5) — visible as identity in
describe_server and embed_space in describe_table. A server whose identity differs — a
different model, or a model whose prefix contract changed — rejects inserts and text searches on
those tables until each is re-embedded: migrate with set_vectorize off, then on (the backfill
re-embeds every row).
A vectorizing migration is resumable. The embedding backfill runs outside the write transaction
and each batch of vectors is kept as it is produced, so a provider that dies, a migrate that runs
out of time, or a process that is killed part way through does not throw the work away: re-issue the
same migrate and it embeds only the rows it does not already hold. A dry_run says so before you
commit — embed_rows is what the call still needs from the provider and staged_rows is what an
earlier attempt already finished, and the two add up to the rows the change will vectorize. A
migration that keeps losing to concurrent writes answers 409 and says the same thing. The staged
vectors are invisible to every other operation, are dropped when the migration lands (or when the
table is dropped or vacuumed), and are never reused by a different provider identity, a different
model of the same identity, or a table that was dropped and recreated. See
docs/design/embedding-backfill.md for why it is shaped this
way.
In networks that block huggingface.co (or on air-gapped machines), download a packaged model
instead of relying on the Hub — the English default and the multilingual model are both available
(see "Choosing an embedding model").
Models are not attached to a dolmen release. They change far less often than the binary does, so they are published once under their own tag and shared by every version; the tag each release uses is linked from its release notes.
models="models-v1"
base="https://github.com/lsm/dolmen/releases/download/${models}"
curl -LO "${base}/dolmen-model-all-MiniLM-L6-v2.tar.gz"
# and/or, for mixed-language/CJK data (~270 MB):
curl -LO "${base}/dolmen-model-multilingual-e5-small.tar.gz"
# Option A: extract into the data directory's model cache, then run normally.
# (The multilingual model needs DOLMEN_EMBED_MODEL; the default does not.)
mkdir -p data/models
tar -xzf dolmen-model-multilingual-e5-small.tar.gz -C data/models
DOLMEN_EMBED_PROVIDER=local \
DOLMEN_EMBED_MODEL=intfloat/multilingual-e5-small \
./dolmen
# Option B: extract anywhere and point DOLMEN_EMBED_MODEL at the directory.
mkdir -p /opt/dolmen/models
tar -xzf dolmen-model-multilingual-e5-small.tar.gz -C /opt/dolmen/models
DOLMEN_EMBED_PROVIDER=local \
DOLMEN_EMBED_MODEL=/opt/dolmen/models/intfloat--multilingual-e5-small \
./dolmenEach tarball contains the org--name model directory (the same layout the local provider uses
under <data>/models): ~80 MB compressed for the MiniLM default, ~270 MB for
multilingual-e5-small. Each archive opens with a size manifest (.dolmen-sizes.json) that lets
the server reject a partially extracted cache instead of loading a truncated model.
Verify the server embeds without reaching huggingface.co by creating a vectorized table, inserting
a row, and running a text vector search:
curl -s localhost:8790/v1/create_table -H 'Content-Type: application/json' -d '{
"namespace": "demo", "table": "notes",
"fields": [{"name": "body", "type": "text", "vectorize": true}]
}'
curl -s localhost:8790/v1/insert -H 'Content-Type: application/json' -d '{
"namespace": "demo", "table": "notes",
"records": [{"body": "A cat sat on the mat."}]
}'
curl -s localhost:8790/v1/search_vector -H 'Content-Type: application/json' -d '{
"namespace": "demo", "table": "notes", "text": "feline"
}'Note that describe_server's usable is configuration-only — it is true whether or not the cache
actually holds the model — while model_state (local provider) does check the cache on disk, and
says which of three things the next vectorized write will meet: cached means the weights are
complete and nothing is downloaded; download_on_first_use is an ordinary first run, where that
operation downloads the model from the Hugging Face Hub and can take ten seconds or more; and
incomplete means DOLMEN_EMBED_MODEL names a model directory that is not complete, which no
download repairs and an operator has to fix. The round-trip above
remains the end-to-end verification: with huggingface.co unreachable and the cache correctly
pre-seeded, the insert and the search_vector(text=...) load the model from disk; a missing or
incomplete cache fails at that first operation with the offline remediations in the error message.
Local provider notes:
- Model cache lives at
<data>/models/(oneorg--namedirectory per model). SetREMBED_CACHEto override the location, andHF_TOKENfor gated Hugging Face repos. - e5 role prefixes are automatic. The e5 family's
"query: "/"passage: "prefixes are prepended server-side (see "Choosing an embedding model"); symmetric models, including the default MiniLM, are embedded as-is. Models with other instruction contracts (bge, arctic) still rank worse than they should — prefer the e5 family or a symmetric model. - Offline installs: pre-seed the model cache under
<data>/models/(the tarball extracts to the rightorg--namelayout), or setDOLMEN_EMBED_MODELto an absolute model-directory path. Both forms skip the Hub entirely whenhuggingface.cois unreachable. See the Offline install section below. - Identity pinning works as with the OpenAI provider: tables record
local/<model>(the versionedlocal/v2:<model>#e5form when the prefix contract is active) as their embedding space, and a model change is rejected until the table is re-embedded (migratewithset_vectorizeoff, then on).
Dolmen reads its startup configuration from command-line flags and environment
variables. Unknown flags and positional arguments are rejected with an error.
dolmen mcp takes the same flags and environment and serves the MCP surface
over stdio instead of HTTP (see MCP (agents)).
| Flag | Environment variable | Default | Description |
|---|---|---|---|
-addr |
DOLMEN_ADDR |
127.0.0.1:8790 |
HTTP listen address (dolmen mcp does not listen) |
-data |
DOLMEN_DATA |
data |
Data directory (one SQLite file per namespace) |
-engine |
DOLMEN_ENGINE |
sqlite |
Storage engine: sqlite (default), postgres or lakehouse. postgres needs -pg-dsn; unknown values are rejected with an error. lakehouse stores tables as Apache Iceberg (Parquet) under the data directory and runs query and filters in the dolmen-duckdb sidecar (see docs/lakehouse-operations.md) |
-duckdb-sidecar |
DOLMEN_DUCKDB_SIDECAR |
dolmen-duckdb beside the binary |
Path to the dolmen-duckdb sidecar, with -engine lakehouse only. Without a runnable sidecar the server still starts, and query and filtered operations are refused |
-duckdb-extensions |
DOLMEN_DUCKDB_EXTENSIONS |
duckdb-extensions beside the binary |
Directory holding the DuckDB iceberg extension the sidecar loads, with -engine lakehouse only |
-pg-dsn |
DOLMEN_PG_DSN |
— | PostgreSQL connection string; required with -engine postgres and rejected without it. The connection pool holds at most 20 connections unless the string sets pool_max_conns=N; keep the total across every server sharing the database below its max_connections |
-pg-catalog |
DOLMEN_PG_CATALOG |
dolmen_catalog |
PostgreSQL catalog schema |
-pg-query-role |
DOLMEN_PG_QUERY_ROLE |
— | Pre-provisioned restricted role that caller SQL runs as; required for the query op |
-auth |
DOLMEN_AUTH |
off |
Authentication. off is the v0.2.0 behavior: no identity, no credential, bind to loopback. on is deny-by-default and refuses to start without an identity source and a reachable root administrator, which on first start means DOLMEN_ADMIN_KEY (see Authentication) |
| — | DOLMEN_ADMIN_KEY |
— | Bootstrap admin credential. Needed with -auth on until another source yields a root administrator, and removable after the hand-over (see Permissions). 32–256 characters of [A-Za-z0-9_-], presented as Authorization: Bearer <key>. Environment only — flags are visible in process listings |
| — | DOLMEN_SECRET_KEY |
— | Key that encrypts secret fields at rest (AES-256-GCM) on either engine: 32 bytes, base64-encoded (openssl rand -base64 32). Without it, creating a secret field or writing a secret value is refused. Environment only; never logged |
| — | DOLMEN_SECRET_KEY_FILE |
— | Path to a file holding that key instead. Set at most one of the two |
| — | DOLMEN_SECRET_KEYS_OLD |
— | Retired secret keys, comma-separated base64, used only to decrypt values written before a key rotation (see "Rotating the secret key"). Needs DOLMEN_SECRET_KEY. Environment only |
| — | DOLMEN_SECRET_KEYS_OLD_FILE |
— | Path to a file holding those retired keys, one per line, instead. Set at most one of the two |
-trusted-proxies |
DOLMEN_TRUSTED_PROXIES |
— | Comma-separated CIDRs (bare IPs allowed) whose peers may assert X-Dolmen-Principal / X-Dolmen-Groups. The same peers are the only ones whose forwarding headers (X-Forwarded-Host/-Proto/-Prefix, Forwarded, original-URI headers) shape the public links dolmen advertises; from other peers they are dropped. Trust is decided from the immediate TCP peer, never from X-Forwarded-For |
-max-groups |
DOLMEN_MAX_GROUPS |
128 |
Maximum group entries accepted per request, 1 to 1024. An over-limit list fails the identity rather than dropping a group |
| — | DOLMEN_AUTH_OIDC_ISSUER |
— | Identity provider issuer URL. Enables sign-in at /v1/auth/begin (see Signing in through an identity provider) |
| — | DOLMEN_AUTH_OIDC_CLIENT_ID |
— | OAuth client id registered with the provider |
| — | DOLMEN_AUTH_OIDC_CLIENT_SECRET |
— | OAuth client secret. Environment only |
| — | DOLMEN_AUTH_OIDC_PRESET |
— | github signs in with GitHub instead of a generic OIDC provider. Refused alongside DOLMEN_AUTH_OIDC_ISSUER |
| — | DOLMEN_AUTH_OIDC_SCOPES |
— | Comma-separated extra scopes to request |
| — | DOLMEN_AUTH_OIDC_GROUPS_CLAIM |
groups |
Claim carrying the caller's groups |
| — | DOLMEN_AUTH_OIDC_TOKEN_TTL |
168h |
Lifetime of an issued token, 1h to 720h |
| — | DOLMEN_AUTH_OIDC_DEPLOYMENT_ID |
minted on first start | Pins this deployment's token issuer id. A mismatch against the stored value is refused at startup |
-version |
— | — | Print version and exit |
-prefix |
DOLMEN_PREFIX |
— | Mount all endpoints (/livez, /readyz, /healthz, /metrics, /version, /skills*, /v1/*, /mcp) under this URL prefix. Use with a pass-through proxy that forwards the full path |
-base-url |
DOLMEN_BASE_URL |
— | Public base URL for the links rendered into the skills manifest, the skill markdown, and the MCP initialize instructions. Default: derive from the request Host and forwarded headers. Refused when it ends with -prefix |
-vector-cache-size |
DOLMEN_VECTOR_CACHE_SIZE |
512MiB |
Memory for decoded vectors kept between vector searches, shared by all tables; about 4 bytes per dimension plus about 125 bytes per row. 0 disables it and every search reads vectors from disk. Results are identical either way. SQLite engine only |
-max-file-size |
DOLMEN_MAX_FILE_SIZE |
1GiB |
Largest single file a write may attach, as bytes or with KiB/MiB/GiB/TiB. It applies on top of a file field's max_bytes, so the lower of the two wins. A larger file is refused with 413 |
-max-namespace-files-size |
DOLMEN_MAX_NAMESPACE_FILES_SIZE |
0 |
Largest total of one namespace's stored files, each file counted once however many rows reference it. A write that would pass it is refused with 507 and writes nothing. Files do not count toward -max-namespace-size. 0 is unbounded |
-max-request-files-size |
DOLMEN_MAX_REQUEST_FILES_SIZE |
2GiB |
Most file bytes one multipart write may carry across its parts; past it the write is refused with 413 |
-files-fetch-hosts |
DOLMEN_FILES_FETCH_HOSTS |
— | Comma-separated hosts a {"url": …} file value may be fetched from: exact names, *.example.com for subdomains, or * for any public host. Empty disables fetching. Every connection is refused to loopback, private, link-local and other non-public addresses after DNS resolution, redirects are re-checked, and no credentials are sent. Fetches send the User-Agent dolmen (+https://github.com/lsm/dolmen) |
-files-store |
DOLMEN_FILES_STORE |
local |
Where file fields keep their bytes: local, beside each namespace, or s3, an S3-compatible bucket. dolmen mcp over stdio serves no files, so it ignores this and the -files-s3-* flags. With s3, -max-namespace-files-size must be 0 and -max-file-size at most 5TiB, and the server marks its data directory (files.store) so dolmen backup and dolmen restore refuse it, however the store was selected: back up the bucket with S3 tooling |
-files-s3-bucket |
DOLMEN_FILES_S3_BUCKET |
— | Bucket for file bytes, required with -files-store s3; startup refuses if it is unreachable or missing. A multipart write's uploads to the bucket get -files-fetch-timeout on top of -op-timeout, so raise it for very large files |
-files-s3-endpoint |
DOLMEN_FILES_S3_ENDPOINT |
https://s3.amazonaws.com |
S3 endpoint URL; http:// is allowed, for a local MinIO |
-files-s3-public-endpoint |
DOLMEN_FILES_S3_PUBLIC_ENDPOINT |
— | Endpoint URL file_url links are presigned for, when clients reach the bucket at another address than the server does (a MinIO on loopback or a private VPC endpoint behind a public name). Uploads and reads keep using -files-s3-endpoint. Empty presigns for -files-s3-endpoint |
-files-s3-region |
DOLMEN_FILES_S3_REGION |
— | S3 region; empty lets the client discover it |
-files-s3-prefix |
DOLMEN_FILES_S3_PREFIX |
— | Key prefix, so one bucket can serve several deployments |
-files-s3-path-style |
DOLMEN_FILES_S3_PATH_STYLE |
false |
Path-style addressing, which MinIO and some S3-compatible stores need. Credentials never come from a flag: the standard chain reads AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN, then ~/.aws/credentials, then an IAM or IRSA role |
-files-fetch-timeout |
DOLMEN_FILES_FETCH_TIMEOUT |
5m |
Longest one server-side file fetch may take, 1s to 24h; a write that fetches gets it on top of -op-timeout |
-max-pending-files-size |
DOLMEN_MAX_PENDING_FILES_SIZE |
8GiB |
Most file bytes all in-flight writes may hold in temporary space at once, so concurrent uploads cannot fill the disk; past it a write is refused with 503 storage_unavailable and may be retried |
-max-namespace-size |
DOLMEN_MAX_NAMESPACE_SIZE |
0 |
Largest a namespace file may grow, as bytes or with KiB/MiB/GiB/TiB. A write that would pass it is refused with 507 and writes nothing; reads keep working. 0 is unbounded. SQLite engine only (see Disk use) |
-log-level |
DOLMEN_LOG_LEVEL |
info |
Log verbosity: debug, info, warn, or error. debug adds one line per operation over HTTP, MCP or stdio with the operation, outcome code, status, duration, request size, and request id, and never the payload, SQL, arguments, or credentials |
-sync |
DOLMEN_SYNC |
full |
Commit durability. full: an acknowledged commit survives power loss. normal: it survives a process crash, but the last commits before a power or OS failure may be lost, for faster writes. Each namespace's writer is checked at open, and a mismatch is refused. SQLite engine only (see Durability) |
-shutdown-grace |
DOLMEN_SHUTDOWN_GRACE |
60s |
On SIGTERM, how long running requests may finish before cancellation. 0 cancels immediately; otherwise 1s to 24h. Cleanup then has a separate fixed 5s hard cap; the process exits even if store close stalls, and remaining database connections close and roll back their transactions. See Shutting down |
-change-retention |
DOLMEN_CHANGE_RETENTION |
168h |
Change-log retention for changes_since / wait_for / subscribe. 0 disables pruning (records and cursors never expire); otherwise 1h to 2160h |
-max-subscription-age |
DOLMEN_MAX_SUBSCRIPTION_AGE |
30m |
subscribe connection age bound: the stream teaching-closes at the bound and the client reconnects from its cursor. 0 disables the bound; otherwise 1s to 24h |
-read-timeout |
DOLMEN_READ_TIMEOUT |
2m |
Time to read one request, headers and body; a body that arrives too slowly is answered 408 with code timeout. 0 disables the bound; otherwise 1s to 24h |
-write-timeout |
DOLMEN_WRITE_TIMEOUT |
2m |
Time to write one response once it starts, so a client that stops reading cannot hold the connection. subscribe streams bound each frame instead. 0 disables the bound; otherwise 1s to 24h |
-idle-timeout |
DOLMEN_IDLE_TIMEOUT |
2m |
Time a keep-alive connection may wait for its next request. 0 disables the bound; otherwise 1s to 24h |
-max-header-bytes |
DOLMEN_MAX_HEADER_BYTES |
1048576 |
Largest request header block accepted; larger ones are answered 431. 4096 to 16777216 |
-op-timeout |
DOLMEN_OP_TIMEOUT |
2m |
Time for one operation's work over HTTP, MCP or stdio; wait_for gets its timeout_ms on top, and migrate answers to -migrate-timeout instead. Dolmen checks cancellation during compilation; PostgreSQL parser workers have bounded admission and waiting so canceled compilation can release its transaction. An operation past the limit is answered 504 with code timeout. 0 disables the bound; otherwise 1s to 24h |
-migrate-timeout |
DOLMEN_MIGRATE_TIMEOUT |
0 |
Time for one migrate call. A vectorizing migration embeds its rows outside the write transaction and keeps each batch, so a call that runs out of time can be re-issued to finish the rest rather than start over. 0 leaves it unbounded; otherwise 1s to 24h |
-max-open-namespaces |
DOLMEN_MAX_OPEN_NAMESPACES |
128 |
Namespaces held open at once. Past it, the least recently used idle namespace closes and reopens on its next request; one in use is never closed. At least 1; SQLite engine only (see Open namespaces and file descriptors) |
| — | DOLMEN_SKILL_NAMESPACE_HINT |
built-in default | Hint text rendered into the served skill markdown |
| — | DOLMEN_ALLOWED_ORIGINS |
— | Comma-separated allowed HTTP origins for CORS; localhost, 127.0.0.1, and ::1 are always allowed |
| — | DOLMEN_EMBED_PROVIDER |
local |
Embedding provider: local (built-in in-process embeddings via rembed, default), openai (any OpenAI-compatible endpoint), or none (caller supplies vectors). Unknown values produce an error |
| — | DOLMEN_EMBED_BASE_URL |
https://api.openai.com/v1 |
Base URL for an OpenAI-compatible provider |
| — | DOLMEN_EMBED_MODEL |
provider default | Model: sentence-transformers/all-MiniLM-L6-v2 for local (or an absolute model-directory path; intfloat/multilingual-e5-small for mixed-language/CJK — e5-family ids get query:/passage: role prefixes automatically), text-embedding-3-small for openai |
| — | DOLMEN_EMBED_API_KEY |
— | API key for an OpenAI-compatible provider. If set (even to ""), it takes precedence over OPENAI_API_KEY |
| — | OPENAI_API_KEY |
— | Fallback API key when DOLMEN_EMBED_API_KEY is unset |
| — | REMBED_CACHE |
<data>/models |
Model cache directory for the local provider (overrides the data-dir location) |
| — | HF_TOKEN |
— | Hugging Face token for gated repos downloaded by the local provider |
Dolmen defaults to -auth off: no credential is required, no identity exists, and
the server binds to loopback. That is the right mode for a local agent and it is
not going away.
-auth on turns on deny-by-default. Identity can come from four sources: the
bootstrap admin key, a gateway's headers, API keys, and sign-in through an
identity provider, each covered below. The admin key comes first, because it is
how the first grants get made:
DOLMEN_AUTH=on \
DOLMEN_ADMIN_KEY="$(openssl rand -base64 32 | tr '+/' '-_' | tr -d '=')" \
./dolmen -addr 0.0.0.0:8790Callers then present the key as a bearer token:
curl -sS http://localhost:8790/v1/list_namespaces \
-H "Authorization: Bearer $DOLMEN_ADMIN_KEY" \
-H 'Content-Type: application/json' -d '{}'Every /v1/{op}, /mcp, and /v1/subscribe request without an accepted
credential answers 401 with error code unauthorized. Rejections are
deliberately uniform — a wrong key, a malformed one, and a missing one produce
the same message, so the response never says which part failed.
/livez, /readyz, /healthz, /metrics, /version, /skills*, and /v1/openapi.json stay unauthenticated
in both modes: they are probes, counters and client-side schema discovery, and
expose no row data. /metrics reveals request volume per operation; if that matters, block it at your proxy.
If dolmen sits behind an authenticating proxy, the proxy can assert the caller's
identity with headers, and dolmen will trust them only from peers inside
-trusted-proxies:
DOLMEN_AUTH=on \
DOLMEN_ADMIN_KEY="$key" \
DOLMEN_TRUSTED_PROXIES=10.0.0.0/8 \
./dolmen -addr 0.0.0.0:8790The proxy then sends X-Dolmen-Principal: alice and, optionally,
X-Dolmen-Groups: team-a,readers. Trust comes from the TCP peer address, never
from X-Forwarded-For, which any client can spoof; headers from a peer outside
the configured ranges are ignored entirely. A malformed principal or group, an
over-limit group list, or an attempt to assert the reserved dolmen-admin
principal is a 401, never a silently weakened identity. A bearer credential
outranks an asserted header, and an invalid bearer is refused rather than
falling back to the header identity.
An authenticated caller starts with nothing. Access comes from grants: a
subject (a principal or a group) gets verbs on an object (a namespace, a
table, or * for the whole server).
The seven verbs are create, read, update, delete, schema, admin,
and reveal. The first six are CRUD-shaped on purpose — an append-only table is
create without update or delete, which a bundled "write" permission could
not express. reveal returns secret fields in plaintext; admin does not
imply it, and every reveal is written to the audit log without the value.
# The bootstrap admin key can grant. Give a group read access to a namespace:
curl -sS http://localhost:8790/v1/grant \
-H "Authorization: Bearer $DOLMEN_ADMIN_KEY" \
-H 'Content-Type: application/json' \
-d '{"subject":{"type":"group","id":"team-a"},
"object":{"namespace":"acme"},
"verbs":["read"]}'Grants inherit downward: a grant on acme covers every table in it and
every namespace under it; * covers everything. They never inherit upward — a
grant on one table says nothing about its namespace or its siblings. A caller's
effective verbs are the union of every grant matching their principal and every
grant matching any of their groups, over the object and everything covering it.
There are no deny grants and no precedence — only union.
Grants are idempotent by (subject, object): re-granting merges new verbs into
the existing grant and keeps its created_at. revoke takes explicit verbs,
never an implicit "all", and the grant disappears when its last verb goes.
Dropping a namespace or table removes the grants targeting it.
A few rules worth knowing:
- Authorization precedes existence. An ungranted caller gets
403whether or not the object exists, andlist_tableson a namespace they hold nothing under answers404— so neither can be used to enumerate names. queryneedsreadon the whole namespace, not one table: raw SQL can reference any table in it.drop_tableneedsadminas well asschema, because dropping a table deletes the grants targeting it, and changing what others may do isadmin.- Implicit namespace creation is off when auth is on. A write to a namespace
that does not exist answers
404instead of creating it, which would bypass theadmingrant on the parent. - Grant-free ops still work:
describe_server,capabilities,infer_schema,whoami, andlist_namespaces(which lists only what the caller can reach).
After a 403, whoami reports the principal, groups, and identity source the
request authenticated as — enough to ask an administrator for the right grant.
Under auth describe_server also reports auth — the mode and the identity
sources the server accepts (admin-key, api-keys, trusted-proxy, oidc) —
and inlines the engine's capabilities. With auth off both are absent.
Bootstrapping and handing over. The admin key exists to mint the first real
administrators, not to reign. Grant admin on * to a principal, then remove
DOLMEN_ADMIN_KEY from the environment and restart: the bootstrap identity
disappears while the grants persist. Revoking the last root administrator is
refused while no replacement exists, and setting DOLMEN_ADMIN_KEY again always
recovers a locked-out deployment.
A table can restrict rows to whoever wrote them:
curl -sS http://localhost:8790/v1/create_table \
-H "Authorization: Bearer $DOLMEN_ADMIN_KEY" \
-H 'Content-Type: application/json' \
-d '{"namespace":"acme","table":"notes",
"fields":[{"name":"body","type":"text"}],
"row_access":"own"}'The server adds an implicit owner column and stamps it on every insert.
Callers never supply it — it is refused like any unknown field. What a caller
sees then depends on the verbs they hold on the table:
| Caller holds | Sees |
|---|---|
read |
every row (the read verb is table-wide by definition) |
any of create, update, delete |
only the rows they wrote |
only schema or admin |
no rows; describe_table reports row_count 0 |
Never grant
readto the people arow_accesstable keeps apart.readshows every owner's rows, even alongside other verbs. For "each person sees only their own rows", grantcreate(plusupdate/deleteif they edit), and keepreadfor whoever should see everyone's rows, such as an auditor.
Own-row visibility rides with any data verb, so a create-only appender can
search back what it appended without being able to read anyone else's rows.
When teams must not see each other's schema at all, a sub-namespace per team is
the other tool; Separating tenants
compares the two.
row_access can be turned on later with migrate, but only while the table is
empty — there is no honest way to assign owners to rows that already exist,
since no operation can write another principal's rows as that principal. To
adopt it for existing data, create a new table and replay each owner's rows
under their own identity. Turning it off keeps the column and its values and
needs admin as well as schema and read, because it changes what every
other data-verb holder may reach.
Filters work under a scope, over your own rows only. update, delete,
upsert and filtered searches take a SQL WHERE fragment, and under auth that
fragment is checked against a fixed allowlist and then evaluated behind a
barrier: the scope is applied first, so a filter never runs against a row you
cannot see. That matters beyond the rows it returns — an expression that merely
errors on a foreign row would report what that row holds.
Idempotency keys belong to whoever used them. A key is unique per table
and owner, so two principals using the same string are using two different
keys: neither conflicts with the other, neither reveals the other, and the
first to use a key cannot squat it against everyone else. A retry consults only
your own domain, so it finds your record through grant changes and even after
row_access is turned off — the place it looks can never move under you.
Records written before authentication was turned on are kept, and replay to
callers holding table-wide read, whose ids those already are.
upsert_by_key matches on the natural key within your visible set, not across
the table. A key held by a row you cannot see counts as no match at all, so the
insert branch runs and you get a fresh row of your own; the two rows then
coexist under the same key, which is what natural keys do when several people
use them. Nothing in the response — ids, counts, or errors — distinguishes a key
someone else is using from one nobody is using.
A machine cannot do an interactive sign-in, and a shared long-lived token is the wrong shape for a CI job. Mint a key instead:
curl -sS http://localhost:8790/v1/create_key \
-H "Authorization: Bearer $DOLMEN_ADMIN_KEY" \
-H 'Content-Type: application/json' \
-d '{"name":"ci runner","principal":"ci-bot","groups":["builders"]}'The response carries the credential once — it is stored hashed and can never be shown again. Mint a new key if it is lost. The caller presents it exactly like any other bearer token:
curl -sS http://localhost:8790/v1/query \
-H "Authorization: Bearer dlm_..." \
-H 'Content-Type: application/json' -d '{"namespace":"acme","sql":"SELECT 1"}'A key grants nothing by itself. It authenticates as its principal, and that principal needs grants like anyone else — which is what makes it safe to mint one before deciding what it may do. Group grants reach key identities too, so a fleet of machines can share one grant through a group.
Every key carries a server-generated id. revoke_key selects by that id, so two
keys sharing a name and principal stay individually revocable — you can drop
exactly the compromised credential. list_keys reports ids, names, principals,
groups and revocation state, never credentials. A revoked key is refused with
the same 401 as an unknown one.
Revoking a key is refused when it would leave the deployment with no usable root
administrator; setting DOLMEN_ADMIN_KEY and restarting always recovers.
For humans, dolmen can run the sign-in itself — no gateway, no proxy:
DOLMEN_AUTH=on \
DOLMEN_ADMIN_KEY="$key" \
DOLMEN_AUTH_OIDC_ISSUER=https://login.microsoftonline.com/<tenant>/v2.0 \
DOLMEN_AUTH_OIDC_CLIENT_ID=... \
DOLMEN_AUTH_OIDC_CLIENT_SECRET=... \
./dolmen -addr 0.0.0.0:8790Register <base-url>/v1/auth/callback as the redirect URI with your provider,
then send people to /v1/auth/begin. They sign in with the provider, land back
on a small page carrying a token, and present it as a bearer credential like any
other. DOLMEN_AUTH_OIDC_PRESET=github uses GitHub instead of a generic OIDC
provider; it brings its own issuer, so setting it alongside
DOLMEN_AUTH_OIDC_ISSUER is refused at startup rather than silently preferring
one. Those two endpoints are the only non-JSON surface dolmen serves, and they
appear in /v1/openapi.json only on a deployment that serves them.
dolmen stores no users and no sessions. The token is Ed25519-signed and
stateless, valid for DOLMEN_AUTH_OIDC_TOKEN_TTL (default 7 days, range 1h to
720h). When it expires the caller signs in again. There is no per-device
revocation and no sliding session. To sign everyone out at once, rotate the
signing key and retire its predecessor:
curl -sS http://localhost:8790/v1/rotate_signing_key \
-H "Authorization: Bearer $DOLMEN_ADMIN_KEY" \
-H 'Content-Type: application/json' -d '{"retire_previous":true}'Without retire_previous the successor signs new tokens while the predecessor
keeps verifying, so tokens already in people's hands live out their TTL — the
overlap you want for a routine rotation. With it, every token signed by an
earlier key stops working.
On the replica that served the rotation the change is immediate. Others pick it up within their keyring refresh interval (30 seconds), since the keyring lives in the shared registry rather than in any one process.
Identities are qualified by issuer. A subject is only unique within its
provider, so the principal is oidc:v1:<issuer-digest>:<sub> — never the email,
which changes. Groups are qualified the same way. Two consequences worth
knowing: changing DOLMEN_AUTH_OIDC_ISSUER mints a completely separate set of
principals and inherits no grants, and Entra emits group claims as object GUIDs
rather than names, so Entra deployments grant on the GUIDs or sync names
themselves.
The signing key and the deployment's own issuer id are minted on first start and
persist beside the grants, so tokens survive restarts and verify across
replicas. Pin the id with DOLMEN_AUTH_OIDC_DEPLOYMENT_ID if you need it
stable; a mismatch against the stored value is refused at startup, naming both.
Two deployments that accidentally share a keyring still will not accept each
other's tokens, because the id is checked as well as the signature.
dolmen mcp (the stdio transport) refuses to start with -auth on: a pipe
carries no per-request credential, and treating whoever launched the subprocess
as an administrator would be a silent bypass. Stdio is reachable only by its
parent process, so run it with auth off.
Running authentication for a team — which identity sources to enable, TLS, the first start and hand-over, what a gateway must do, and how sign-in behaves across replicas — is covered in the deployment guide.
Dolmen can be exposed at a sub-path behind a reverse proxy in two ways. In both recipes the public links rendered by the skills manifest, the skill markdown, and the MCP initialize instructions must match the URL the client uses.
The proxy removes the sub-path before forwarding to dolmen. Set the public base
URL explicitly or rely on forwarded headers (X-Forwarded-Proto,
X-Forwarded-Host, X-Forwarded-Prefix).
Forwarding headers count only from a peer listed in -trusted-proxies /
DOLMEN_TRUSTED_PROXIES; from any other peer they are dropped, so a client
cannot make dolmen advertise links to another host. For a proxy on the same
machine, that means DOLMEN_TRUSTED_PROXIES=127.0.0.1 (add ::1 if it connects
over IPv6). With -auth on, a listed peer may also assert identity through
X-Dolmen-Principal / X-Dolmen-Groups, so a proxy you trust only for its
URL must remove those headers from what clients send
(proxy_set_header X-Dolmen-Principal ""; and the same for X-Dolmen-Groups
in nginx).
nginx:
location /dolmen/ {
proxy_pass http://127.0.0.1:8790/;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Prefix /dolmen;
}Caddy (strips the matched prefix automatically):
handle_path /dolmen/* {
reverse_proxy 127.0.0.1:8790 {
header_up X-Forwarded-Prefix /dolmen
}
}With the forwarded header and the proxy listed in DOLMEN_TRUSTED_PROXIES, no
other dolmen configuration is needed. If your proxy does not add X-Forwarded-*
headers, or you would rather not list it, set DOLMEN_BASE_URL to the full
public URL instead:
DOLMEN_BASE_URL=https://example.com/dolmen ./dolmenA rewrite that strips the sub-path is a stripping proxy, and it is the easiest
one to get wrong, because nginx supplies none of the context dolmen needs:
Host defaults to the upstream address (127.0.0.1:8790), and nginx never
sends X-Forwarded-Prefix on its own. Left alone, dolmen advertises links to
its own loopback address with no sub-path.
Dolmen recovers the sub-path by itself when a trusted proxy forwards the
original request URI — nginx's $request_uri — so this config, with
DOLMEN_TRUSTED_PROXIES=127.0.0.1, works without naming the prefix twice:
location /dolmen/ {
rewrite ^/dolmen/(.*)$ /$1 break;
proxy_pass http://127.0.0.1:8790;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Original-URI $request_uri;
}X-Forwarded-Uri, X-Original-URL, X-Envoy-Original-Path, and
X-Rewrite-URL are read the same way; an explicit X-Forwarded-Prefix always
wins over inference. The standard Forwarded header (RFC 7239) is honored for
proto and host, below the X-Forwarded-* equivalents.
Setting DOLMEN_BASE_URL to the full public URL overrides all of it and is the
surest fix when a proxy cannot be changed.
Ask the server what it thinks its public URL is, through the proxy:
curl -s https://example.com/dolmen/skills | grep base_urlIf base_url is not the URL you typed, the proxy is not telling dolmen enough.
The server also logs a warning the first time it advertises a base URL that no
proxied client could reach:
level=WARN msg="advertising a base URL no proxied client can reach" base_url=http://127.0.0.1:8790
The proxy forwards the full path, including the sub-path, to dolmen. Run dolmen
with -prefix (or DOLMEN_PREFIX):
./dolmen -prefix /dolmennginx:
location /dolmen/ {
proxy_pass http://127.0.0.1:8790; # no trailing slash: do not strip
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
}Caddy:
handle /dolmen/* {
reverse_proxy 127.0.0.1:8790
}-base-url and -prefix can be combined when the configured base URL does not
already end with the prefix. For example, -base-url https://example.com -prefix /dolmen produces public links at https://example.com/dolmen. To avoid
double-prefixing, dolmen refuses to start when -base-url ends with -prefix.
When a browser-based MCP client connects via a proxy, the browser sends an
Origin header such as https://example.com. Add it to the allowlist:
DOLMEN_ALLOWED_ORIGINS=https://example.com ./dolmenlocalhost, 127.0.0.1, and ::1 are always allowed; the public origin of a
proxy is not.
Go programs can open dolmen in-process — no port, IPC, or subprocess. The module is github.com/lsm/dolmen; the executable's install path is now github.com/lsm/dolmen/cmd/dolmen (historical release instructions describe their own layout):
import "github.com/lsm/dolmen"
st, err := dolmen.Open(dataDir,
dolmen.WithEmbedding(myProvider),
dolmen.WithChangeRetention(168*time.Hour),
)
if err != nil { ... }
defer st.Close()Open never reads environment variables, loads a model, or starts a server; options are validated before touching the filesystem. Embedding is disabled unless a provider is supplied — implement dolmen.EmbeddingProvider (Identity, Embed, EmbedQuery); query and document embedding stay distinct for asymmetric models such as E5, and the identity string pins a vectorized table to its embedding space. The application owns the provider (a shared provider is not closed by Close).
The surface covers namespace/table lifecycle, Insert/Update/Delete/UpsertByKey, Batch, GetRows/Query, and SearchFulltext/SearchVector — the same semantic validation, embedding orchestration, error classification, and defaults as the HTTP and MCP transports (the conformance suite pins the parity). Records are map[string]any; reads return engine-typed values (int64/float64 numbers, bool, decoded JSON with json.Number precision, []float64 vectors) rather than a JSON round trip.
A file field is written with a dolmen.FileUpload{Reader, Name, MediaType} value in a record, an Update set or a Batch write; the library streams it into the namespace's file store exactly as a multipart part, and a failed write leaves nothing behind. A dolmen.FileCopy{Table, ID, Field} reuses another row's file, and a dolmen.FileURL{URL} has the library fetch one, from hosts named by Open(dir, dolmen.WithFileFetch(hosts, timeout)) (without that option it is refused); both take optional Name and MediaType overrides. OpenFile(ctx, namespace, table, id, field) returns an io.ReadSeekCloser over the bytes and a FileInfo with the reference.
Batch takes an ordered []dolmen.BatchWrite and applies them in one transaction — one ChangeRange for the whole call, one idempotency_key over all of it, and a result per write in the order sent. It is also the only route to the filter-matched upsert, which has no single method here. BatchOptions carries IdempotencyKey, Limit and Confirm; Limit and Confirm are hashed with the writes, so the same key with a different setting conflicts rather than replaying.
Errors are typed: match categories with errors.Is(err, dolmen.ErrConflict) (also ErrNotFound, ErrQuery, ErrInvalidRequest, ErrEmbedderUnavailable, ErrSQLEngineUnavailable, ErrStorageUnavailable, ErrCanceled, ErrForbidden, ErrInternal) and read *dolmen.Error with errors.As for the code and message; underlying causes stay wrapped. Close is terminal and idempotent — repeated Close returns the first result; operations after close return the local dolmen.ErrClosed sentinel — and one process holds at most one live store per data directory (symlinks resolve; opening a server and an embedded store on the same directory is unsupported).
While dolmen is on v0, breaking Go API changes land in minor releases only; patch releases keep source and behavioral compatibility.
A runnable end-to-end example lives in examples/basic.
Dolmen speaks MCP over two transports — one dispatcher, two framings: tools/list, tools/call, the JSON-RPC 2.0 envelopes, and the error taxonomy are identical either way.
HTTP (./dolmen) — one server, many clients, browser and curl access:
claude mcp add --transport http dolmen http://127.0.0.1:8790/mcpstdio (dolmen mcp) — for hosts that launch the server as a subprocess (Claude Desktop, Cursor, any non-Go client that can spawn a process). One process serves one data directory: newline-delimited JSON-RPC 2.0 on stdin/stdout, stdout carries protocol only (all logs go to stderr), and the process serves until stdin closes or SIGTERM/SIGINT arrives, then drains in-flight requests (a 5s grace, then cancellation, a bounded worker join, and a separate 5s cleanup cap); a second SIGINT/SIGTERM during a stuck drain kills the process immediately:
claude mcp add dolmen -- /path/to/dolmen mcp -data /path/to/datadolmen mcp takes the same flags and environment as dolmen; the HTTP-only ones (-addr, -max-subscription-age, DOLMEN_ALLOWED_ORIGINS) have no effect. The initialize handshake works as over HTTP (its instructions describe the stdio transport; set -base-url when an HTTP deployment also exists, and its links appear there). Over stdio there is no MCP-Protocol-Version header, so version negotiation happens in initialize alone.
The MCP server exposes the same operations as tools (tools/list shows them with input/output schemas and annotations). Successful tools/call results carry structuredContent — the result as a JSON object matching the tool's outputSchema — with no text mirror (content stays an empty array: the spec keeps it mandatory); tool errors are reported as text with isError: true.
Skill distribution is built into the server. GET /skills returns a JSON manifest with links to the layered skill markdown; GET /skills/dolmen is the end-user skill and GET /skills/dolmen-admin is the developer skill. Agents should fetch the skill from the running binary instead of copying a static file. Over dolmen mcp there is no HTTP listener — the skills are served by the HTTP deployment named by -base-url, when one exists.
GET /v1/subscribe streams a namespace's change feed over server-sent events: it replays the
durable change log from a cursor, then delivers every live commit as it happens. It is the push
counterpart of wait_for, which carries the same feed semantics as a request/response tool;
capabilities's subscribe field reports whether the server offers it.
curl -sN "http://127.0.0.1:8790/v1/subscribe?namespace=myapp"Query parameters mirror changes_since (names are trimmed and lowercased like every /v1
call): namespace (required — a namespace that does not exist is an in-stream not_found
error; the stream never creates one, as no read does), table (optional
filter to one table's feed; an explicitly empty value is rejected), and cursor (an opaque
resume token or the literal begin; omitted = start at the current head and receive future
commits only; an explicitly empty value is rejected). A Last-Event-ID request header, which a
browser EventSource sends on reconnect, is a resume token too and takes precedence over
cursor. Wrong method, an omitted namespace
parameter, and empty table / cursor values are ordinary HTTP errors (the standard envelope)
before the stream opens; every
other failure arrives inside the stream, because an open text/event-stream response can no
longer carry an HTTP status.
The frame protocol — named events whose data is one compact JSON line; change, ready and
close frames also carry their cursor as the SSE id:, so an EventSource reconnect resumes
where the stream left off:
event: ready,data: {"cursor":"..."}— the replay→live boundary. Replayedchangeframes (when resuming behind the head) arrive first, thenready, then live frames. Its cursor is the last replayed change's, or the head at registration when nothing replayed — reconnect with it to resume exactly there, gap-free. A stream that ends during replay never reachesready; itscloseframe carries the reached cursor instead.event: change,data: {"cursor":"...","table":"...","row_id":1,"kind":"insert"}— one frame per changed row: a batch write (multi-recordinsert, filter-matchedupdate/delete) mints one change-log record per affected row, and frames arrive in commit order.kindisinsert,update, ordelete. The payload is the same identity projectionchanges_sincereturns — re-read the row by id; adeletenames a row that is already gone.event: close,data: {"cursor":"..."}— sent before every server-initiated terminal, with the last-delivered cursor: the reconnect token.event: error,data: {"ok":false,"error":{"code","message","request_id"}}— the terminal frame: the standard error envelope. Nothing follows it.
Between change deliveries an idle stream sends a : keepalive comment frame every 20 seconds —
an SSE comment line with no event name and no data, ignored by every event parser, carrying no
cursor and never advancing one. Treat it as liveness: a connection that delivers neither a
change nor a keepalive for a couple of intervals (~40 s) is dead — close it and reconnect from
your last cursor.
Every server-initiated terminal is a close frame followed by a teaching error event whose
message is the recipe; registration failures — an unknown or foreign cursor, a missing namespace,
a missing table — send only the error event, since nothing was delivered. The causes, verbatim:
- Unknown or expired cursor: "cursor is unknown or past the change-log retention window (-change-retention, default 168h); catch up by reconnecting with no cursor to resume from the current head, or with cursor=begin to replay retained history"
- Cursor from another feed: "cursor was minted on a different feed (a specific table's, or the namespace-wide feed); pass it only to the feed you received it from — honoring it elsewhere would silently skip events — or start fresh with no cursor / cursor=begin"
- Buffer overflow: "subscription buffer overflow: commits arrived faster than this stream drained them; reconnect from the cursor in the preceding close frame — the durable log is the catch-up path, the buffer never was"
- Target ended: "the subscription's target ended (a dropped table, or a dropped or replaced namespace); reconnect against the current target — a same-named successor is a different feed"
- Authorization revoked (code
forbidden): "subscription authorization was revoked; reconnect once authorization is restored, or with another credential if this one was revoked". The server rechecks access on every change it delivers and on every keepalive (every 20 seconds), so a revoked caller's stream ends within about 20 seconds even when nothing is being written. - Subscription age bound (
-max-subscription-age, default 30m): "subscription reached the maximum subscription age (-max-subscription-age, default 30m); reconnect from the cursor in the preceding close frame to resume exactly where this stream ended — the fresh connection re-asserts your credentials"
Recovery is one rule for the cursor-reusable terminals — buffer overflow, revoked, age bound:
reconnect with the close frame's cursor (or the newest you hold, from ready or the last
change, when the connection died without one) and the new stream replays everything after it,
then goes live — no gaps, no duplicates. Target ended is the close-bearing exception: the feed
you were reading is gone (a dropped table's feed, or a dropped/replaced namespace whose cursor
store was deleted with it), so its cursor cannot resume anything — reconnect against the current
target and establish a fresh cursor (no cursor, or cursor=begin), as its message says; a
same-named successor is a different feed. The registration failures above are the other
exception: their cursor was rejected, so reconnect exactly as their message says — no cursor
(head) or cursor=begin — never the rejected token. Like every feed surface, cursor tokens are
minted per emission: the same commit carries a different token on each read that re-emits it,
while a read that emits nothing returns your cursor unchanged (the wait_for idle contract) —
and no frame field is a stable event id either: the same row updated twice yields two changes
with identical table/row_id/kind. Make processing idempotent and persist the cursor
atomically with your side effects rather than deduplicating on frame content.
| Tool | Purpose |
|---|---|
list_namespaces |
Namespaces on this server; an optional prefix (a namespace path) lists only that path's subtree, recursively |
create_namespace |
Reserve a namespace up front (the write ops create implicitly on first use otherwise; every read — including wait_for and the subscribe stream — never creates, and a missing namespace is not_found) |
drop_namespace |
Delete a namespace and all its tables; confirm must repeat the name; a namespace with child namespaces is refused — drop the children first |
rotate_secret_key |
Re-encrypt stored secret values under the active key in bounded batches; reports rotated, remaining, done and values per key id (admin on *) |
vacuum |
Reclaim free space in one namespace (SQLite: rebuild the file and truncate its WAL; PostgreSQL: VACUUM its tables) and report bytes_before / bytes_after; on every engine also remove stored files no row references and report files_removed, file_bytes_removed and file_bytes |
list_tables |
Tables in a namespace |
describe_server |
Server's embedding provider status — provider (none / local / openai), model, the identity that pins vectorized tables, whether server-side embedding is usable, and (local only) whether the model is cached; read-only, no secrets |
describe_table |
Schema, version, row count |
read_rows |
Fetch rows by id — each found row once, ascending id order; missing ids are simply absent (never an error); truncated is true only when the response budget dropped rows for existing ids (retry with fewer); at most 1,000 ids per request. Without ids, page through the table in id order: after_id (default 0) and limit (default 100, at most 1,000); while truncated, pass next_after_id as the next after_id. Needs only read on the table, or own-row access on a row_access table |
file_url |
A short-lived URL that downloads one row's file with a plain GET and no credentials, for a tool that holds none. ttl_seconds defaults to 900, at most 3,600; the link stops working at expires_at, and on the local store also once the row's file is replaced. With -files-store s3 it is the bucket's presigned GET, which works until it expires. Needs the same access as read_rows on the row. Served over HTTP and /mcp, not stdio |
capabilities |
The engine's static capability surface — vector_execution (exact / ann), ann_recall_bound (explicit null when exact), notifications, subscribe, query_dialect, filter_dialect; reported verbatim |
create_table |
Typed fields with fulltext / vector / vectorize / enum / shape / default annotations (enum restricts a string field to a closed vocabulary; shape restricts a json field to one JSON shape such as array<string>; default is stored by inserts that omit the field — a timestamp field may declare "now()", stamped server-side at write time so idempotent retries can omit the field and replay). constraints declares unique keys and foreign keys to another table's id (that table needs accept_references: true), and relation names a relation table's source, target and type fields; constraints work on SQLite, PostgreSQL and lakehouse; relation tables work on SQLite and PostgreSQL so far. With -auth on, constraint declarations require schema plus read on affected tables, and opting in to references requires namespace-wide read |
infer_schema |
Propose fields from sample records (creates nothing). Names are always valid for create_table on the running engine, which renames the keywords PostgreSQL or DuckDB reserve as it does SQLite's; warnings explains every rename or merge, provenance maps fields to source keys, and evidence counts presence, nulls and observed types per field |
insert |
Validated records; indexes and embeddings update automatically; idempotency_key makes retries replay the original ids |
import_records |
Import rows from another database keeping their source id and created_at; 1–1,000 records per chunk with a required idempotency_key; identical existing ids are skipped, differing ones conflict and roll the chunk back; needs table admin and create |
relate |
One directed edge into a relation table (source and target row ids, type, optional properties); both endpoints must exist; idempotency_key as for insert |
batch |
An ordered list of insert / update / delete / upsert / upsert_by_key writes to one namespace, applied in a single transaction — all commit or none does — with one result per write in order, one change-feed commit, and one idempotency_key for the whole batch. At most 100 writes and 1,000 rows touched in total; namespace and idempotency_key are set once per batch and refused inside a write, and dry_run, limit and confirm are refused inside a write because a batch is all-or-nothing. An error names the failing write as writes[i] and keeps that write's own error class. A batch update or delete may carry expected_rows (0 to 1,000), the exact count it must match (updated / matched), checked inside the transaction; any other count fails the whole batch with conflict, which makes read-decide-write a compare-and-set |
upsert_by_key |
Insert-or-update keyed by natural field(s) (on); converges instead of duplicating on retry |
query |
Read-only SQL (SELECT/WITH), parameter binding via args, typed results |
search_fulltext |
FTS5 MATCH over fulltext fields, relevance-ordered, typed results; optional filter + args restrict rows before ranking |
tokenize |
The terms a table's full-text index stores for a piece of text, in order: use it to find the stem a prefix search must start from. Returns no row data; any data grant on the table allows it |
search_vector |
Cosine KNN; text (server embeds; searches only the vectorize _embedding space) or raw vector (any vector column, caller owns the space); optional filter + args and min_score threshold; results carry _score and skipped_vectors |
changes_since |
Replay the namespace's durable change log: changes committed after a cursor, in commit order, as a bounded page plus next_cursor. No cursor = start at the current head (future commits only); "begin" = retained history; optional table filters to that table's current lifetime. Changes carry cursor/table/row_id/kind plus optional commit |
wait_for |
Long-poll the change feed: block until a change commits after the cursor or timeout_ms elapses (default 30000, max 60000, 0 = conditional poll with no polling wait; its storage read uses the operation deadline), then return exactly a changes_since page. A timeout is an empty page carrying the unchanged next_cursor — never an error; pass it back in to keep waiting. |
delete |
WHERE-filtered delete, cascades to search indexes |
drop_table |
Drop a table — rows, search index, schema, history, idempotency keys; confirm must repeat the name |
update |
WHERE-filtered field update; reindexes full-text rows and re-embeds changed vectorized fields |
upsert |
Update matching rows, or insert one record when the filter matches nothing |
migrate |
add_field (optional default backfills existing rows — required fields land on populated tables as NOT NULL DEFAULT; optional fields get a one-time backfill, later omitted inserts store NULL), rename_field, drop_field, set_fulltext, set_vectorize, set_enum (replaces a string field's vocabulary; rejects when a stored value falls outside the new list, naming it and its row count; an empty list removes the constraint), set_shape (sets or clears a json field's required shape; rejects when stored values do not fit, naming how many rows and their ids), set_file_constraints (changes a file field's max_bytes and media_types in schema metadata; rejects when stored files break them, naming how many rows and their ids); add_constraint (a unique or foreign-key declaration, checked against existing rows even during a dry run), drop_constraint (by name) and set_accept_references (explicit boolean value) each run as the only change in their migrate call; expected_version asserts the schema being migrated (required for rename/drop, conflicts surface as 409), expected_incarnation — the opaque token a dry run returns — asserts the table itself and is what a precondition must use when auth is on, dry_run previews the plan without side effects (its plan reports embed_rows, the rows still needing a provider call, and staged_rows, the ones an interrupted earlier attempt already embedded, so a vectorizing migration is resumable); versioned + logged |
list_migrations |
A table's migration history, newest first, with the exact recorded changes |
Change-feed commit is a stable positive integer within a namespace incarnation. All changes from one transaction share it, including multi-row writes and batch; it survives pagination, re-reads, wait_for, and SSE. It is absent for records written before the upgrade. Pages and filtered feeds may contain only part of a commit, so this field groups changes without signalling completeness. Cursor tokens remain resume handles; persist them atomically with idempotent processing.
-
Namespace = one SQLite file (
data/<ns>.db, WAL). Isolation is physical. Lifecycle is managed over the API:list_namespaces,create_namespace, anddrop_namespace(which closes the server's own connections, then deletes the file, its WAL sidecars and the files of its file fields —confirmmust repeat the namespace name, and any later write-op use of the name recreates the namespace empty (every read — includingwait_forand thesubscribestream (/v1/subscribe) — answersnot_founduntil it is recreated); a namespace with child namespaces is refused, the error naming the descendant count — children are dropped first, never deleted implicitly). Safety caveat: drop coordinates only within one server — another process holding the file open (a second dolmen instance, a backup tool) is not detected, and racing in-flight requests on the namespace may fail, so quiesce writers before dropping. A small registry inside each file holds table schemas, versions, and a migration log (surfaced bylist_migrations). -
Files: a
filefield holds a reference to bytes stored beside the namespace in<data>/<ns>.files/, content-addressed by SHA-256, so identical bytes are stored once however many rows reference them. A write attaches files in the same call:insert,import_records,update,upsert,upsert_by_keyandbatchacceptmultipart/form-data, with the JSON request in a first part namedrequestand each file in its own part, referenced as{"part": "<part name>"}:curl -s http://127.0.0.1:8790/v1/insert \ --form-string 'request={"namespace":"app","table":"meetings","records":[{"title":"standup","recording":{"part":"rec"}}]}' \ -F 'rec=@standup.ogg'
The media type comes from the part's
Content-Type, or from the file name's extension when the client sendsapplication/octet-stream; a value may setnameandmedia_typeitself.{"url": "https://…"}makes the server fetch the file, when the operator lists its host in-files-fetch-hosts.{"copy_from": {"table", "id", "field"}}reuses a file a row the caller can read already holds, in any write and over MCP, without uploading it again. Every read returns the reference{"sha256", "size", "media_type", "name"}, never the bytes.GET /v1/files?namespace=&table=&id=&field=downloads them with the same authorization asread_rowson that row, as an attachment withX-Content-Type-Options: nosniffand a sandbox CSP, supportingRangeandETag.file_urlreturns{"url", "expires_at"}, a link to that download that needs no credentials, forttl_seconds(default 900, at most 3600); it is signed with HMAC-SHA256 under a key kept at<data>/files.key, and stops working once the row's file is replaced. With-files-store s3it is the bucket's presignedGETinstead, with the same type and attachment headers, which works until it expires. A field may declaremax_bytesandmedia_types(exact, oraudio/*). MCP calls cannot carry files. File fields work on SQLite and lakehouse with either store, and on PostgreSQL with-files-store s3; PostgreSQL refuses them with the local store, which one server's disk would hide from the others. Running the stores, their limits, S3 permissions and backups: file storage.docs/design/object-storage.mdhas the design. -
Full-text via SQLite FTS5 shadow tables, maintained on insert/update/delete/migrate.
-
Enum-validated writes: a string field declared
enum: [...]accepts only those values on every write path (insert,update/upsertset,upsert_by_key) — exact match, no case folding, values stored as written. A rejected write names the field, the rejected value, and the allowed list, over/v1and MCP alike. A declareddefaultmust be a member; evolve the vocabulary withmigrateset_enum, which refuses to drop a value rows still store (naming it and its count). The annotation is declared and reported in the MCPtools/listschemas and/v1/openapi.json, so schema-validating clients see the vocabulary before the first write. -
Shape-validated JSON: a
jsonfield declaredshape(object,array,array<string>,array<number>,array<boolean>orarray<object>) accepts only values of that shape on every write path, on both engines and in the Go library alike, so a tags field declaredarray<string>refuses"db,sqlite"where["db","sqlite"]was meant. The refusal names the field, the expected shape and what arrived, down to the first misfit element's index.nullstays allowed unless the field isrequired. Evolve it withmigrateset_shape, which refuses a shape stored rows do not fit, naming how many and their ids. -
Idempotent writes for agent retries:
insertaccepts anidempotency_key(client-chosen, durably recorded with its ids in a side table, so a retry — even after a restart — returns the original ids; reusing a key for different records is an error), andupsert_by_keywrites records keyed by a natural field set (on), updating the matched row partially or inserting when nothing matches. -
Vectors are stored as float32 blobs, and KNN is an exact cosine search in Go with no index to build or tune. Each table's decoded vectors are kept in memory between searches (up to
-vector-cache-size, default 512 MiB) and kept current from the change log, so a search scores from memory across all cores instead of reading every vector back. On a recent laptop, a top-10 search over 20,000 rows takes about 1 ms at 384 dimensions (the defaultlocalmodel) and 3 ms at 1,536 (OpenAI'stext-embedding-3-small); 100,000 rows take about 4 ms at 384 dimensions. A table's cache costs about 4 bytes per dimension plus about 125 bytes per row, so 100,000 rows at 1,536 dimensions need about 620 MB: raise-vector-cache-sizeto cover your largest vector tables. Searches onrow_access: owntables use the cache too, since it also holds each row's owner. Afilteris evaluated by SQLite first, which reads the table (about 70 ms for 20,000 rows at 384 dimensions), and only the matching rows are then scored from memory. A table that does not fit reads the vectors from disk instead, at about 200 million row-dimensions a second (0.6 s for 100,000 rows at 1,536). -
Vector-search spaces are kept honest:
textqueries are embedded by the active provider and only search the server-managedvectorize(_embedding) space, whose model identity is pinned per table — a provider change is rejected until the table is re-embedded. Caller-providedvectorcolumns are searchable only with a rawvectorquery, because only the caller knows which embedding space produced them. Stored vectors that are corrupt, dimension-mismatched, or non-finite are skipped from scoring and reported asskipped_vectors, so a search never silently drops rows. -
Read-only SQL runs on a
mode=roconnection with a SELECT/WITH allowlist — defense in depth. -
Typed reads across
query,search_fulltext, andsearch_vector: results honor declared field types —boolean→true/false,json→ the decoded value,vector→ a number array,number→ integer or float, SQLNULL→null. In raw SQL, coercion is by result-column label (aliases count as their label); labels that match no declared field, or that different tables declare with conflicting types, fall back to raw values (blobs as base64). The hidden_embeddingcolumn (fromvectorize) is stripped fromSELECT *and search results — passinclude_hidden: trueto a search to include it. Naming it in the SQL (outside string literals and comments) also works where the backend exposes it to caller SQL, but that is backend-dependent. -
Pagination on
query,search_fulltext, andsearch_vectorviaoffsetandlimitparameters. Do not putLIMIT/OFFSETin raw SQL; use the parameters.search_fulltextandsearch_vectorhave stable, deterministic ordering. The response includestruncated: truewhen more results are available beyond the returned page — onquerythis is triggered by the 1,000-row result cap; on the searches, by results existing beyond the requestedlimit(default 10, max 200). -
Embeddings are pluggable:
none(caller supplies vectors),local(built-in in-process inference via rembed — pure Go, no cgo, model weights cached under the data dir), or any OpenAI-compatible endpoint. -
Namespaces are created implicitly on first use by the write ops —
create_table,insert,update,upsert,upsert_by_key,delete, andmigrate(one file per name;create_namespacejust reserves the name up front). Reads never create.list_tables,describe_table,read_rows,query,search_fulltext,search_vector,changes_since,wait_for,list_migrations, thesubscribestream, anddrop_tableall answernot_foundfor a namespace that does not exist, leaving nothing on disk — so a typo costs an error, not a stray database file. Tables are never implicit — callcreate_tablebefore inserting,drop_table(confirm-guarded) to remove one completely. No other management surface to operate.
The lakehouse engine (-engine lakehouse) uses local Iceberg tables: each namespace has its
own SQLite Iceberg catalog at <data>/<namespace>.lakehouse/catalog.db and a separate data/
directory for Parquet and Iceberg metadata. Lifecycle conformance runs alongside SQLite, including
reopen, leaf-only drop and drop/recreate lifetimes. An unpinned drop can remove an unreadable
catalog or incomplete namespace directory; listing skips incomplete namespaces and valid children
still block a parent drop. Generation-pinned drops require a readable, matching lifetime. Deploy one process
per local data directory.
Table lifecycle and schema evolution are available: create/list/describe/drop,
versioned add/rename/drop and metadata migrations, dry-run plans and migration history. Schema
properties and native Iceberg evolution publish in one catalog update; SQLite tombstones protect
drop/recreate lifetimes. The engine supports writes, populated-table backfills, the DuckDB query
sidecar and file fields with the same attachment, copy, download and vacuum behavior as SQLite.
See operations and the lane plan.
Storage sits behind the store layer, so engines like DuckDB-over-Parquet or Iceberg-over-S3 can be added as adapters without touching the API or MCP surface.
queryis read-only: onlySELECTorWITHstatements are allowed, and only one statement at a time.- Bind all values with
?placeholders and theargsarray. Identifiers and table names cannot be bound with?, so write them directly fromlist_tables/describe_tableand treat them as an allowlist, not user input.queryonly checks that the statement is read-only, not that the identifiers are safe. - SQL string literals use single quotes (
'value'). Escape a single quote by doubling it ('can''t'), or better, use a?placeholder. - Double quotes are for SQL identifiers, not string values.
idandcreated_atare real columns: you can filter, order, and select them.
Fields marked fulltext: true are indexed with a shadow SQLite FTS5 table. The tokenizer is the
porter stemmer wrapping unicode61: case-insensitive, diacritic-insensitive for most Latin
characters (some non-Latin or multi-diacritic characters may not normalize), and stemmed — both
the index and the query reduce English words to stems, so payments matches payment and refunds
matches refund without a prefix wildcard. Most punctuation, including hyphens, is a token boundary.
Stemming notes:
- Porter is a suffix-stripper, not a lemmatizer: it collapses inflections of the same root
(
payments/payment,refunded/refunds/refund) but not different derivations —paying,pays, andpaidstem topai/paidand do not matchpayment. - Phrases match on stems: each word of the phrase is stemmed before matching, so
"payments were"matches"the payments were refunded"(stemspayment,were). tokenizeshows the terms a table's index stores for any text (overheating→overh), so a prefix search can start from the right stem.- Prefix queries operate on stems:
pay*stems topai*, so it matchespaid/paying/paysbut notpayment(whose stem ispayment). - Stemming is English-focused. CJK text is untouched by the stemmer — an uninterrupted CJK run is still indexed as one opaque token, as before (#106).
- Tables created before stemming became the default keep their exact-token index and keep working.
Reindex one with
migrate:{"op": "set_fulltext", "name": "<fulltext field>", "value": true}— re-assertingtrueon an already-indexed field rebuilds the index under the current tokenizer (the migrate plan reportsrebuild_fulltext: true). BM25 rank ordering shifts after a reindex.
search_fulltext takes a raw FTS5 MATCH expression in query. It is not SQL, so do not wrap
the whole expression in single quotes.
Common syntax:
payment— a single token (paymentsmatches the same stem).payment gateway— implicitANDbetween tokens.payment OR gateway— either token.payment NOT gateway— must containpaymentand must not containgateway.title:payment— only in thetitlefulltext field.{title body}:payment— in any of the named fulltext fields."foo bar"— phrase (adjacent tokens, matched on stems). Because stored punctuation is also tokenized, a phrase matches token adjacency, not literal punctuation."foo-bar"— double-quote any term that contains spaces or punctuation (hyphens, dots, slashes, apostrophes). Barefoo-baris read by FTS5 as a column filter and errors.pay*— prefix match, applied to the stemmed term (pay*→pai*).NEAR(payment refund)— proximity search (default near span). The group formNEAR(term1 term2 ...)enforces proximity; writingterm1 NEAR(term2)instead parses as an implicitANDand does not enforce proximity.
Terms containing an apostrophe, such as "can't", must be inside a double-quoted phrase. Bare
single quotes in an FTS5 query are a syntax error.
Results are ordered by FTS5 rank (BM25 by default). More relevant documents have a lower — more
negative — rank value and are returned first. The rank value itself is not included in results.
vectorfields store caller-supplied float arrays. Pass them as JSON number arrays; the server stores them as float32 blobs and returns them as[]float64in reads.vectorize: trueon one string/text field makes the server embed that field into the hidden_embeddingcolumn. Only one field per table can be vectorized; only non-empty values are embedded, so rows withnull, empty strings, or missing values have_embeddingNULL and are excluded from vector search.search_vectorwithtextembeds the querytextwith the configured provider and searches only the vectorize_embeddingspace — a table without avectorizefield rejectstext. Withvectoryou supply the query vector directly and may search any vector column.columnis optional forvectorqueries: it names the stored-vectors column and defaults to_embeddingif a vectorized field exists, otherwise the first declaredvectorfield. The query and stored vectors must come from the same embedding space. For_embedding(fromvectorize) this means the same provider/identity; for caller-suppliedvectorfields it means the same model used to produce the stored and query vectors.- Every vector result carries
_score: cosine similarity, where higher is closer. For typical positive embeddings it ranges0–1; mathematically it ranges-1–1. _embeddingis hidden fromSELECT *and search results unless you passinclude_hidden: true, or the SQL names it explicitly on a backend that exposes it to caller SQL.
Every row has two implicit columns:
id—INTEGER PRIMARY KEY AUTOINCREMENT. It is assigned on insert, cannot be supplied or set, and is never reused after rows are deleted. Ids are safe to reference across sessions.created_at— a UTC millisecond timestamp in ISO/RFC3339 form, e.g.2026-09-03T12:34:56.123Z. It is set on insert and cannot be supplied or set. Use string comparisons or SQLite date/time functions on it.
SELECT * includes both columns.
| Resource | Limit | Behavior when exceeded |
|---|---|---|
| Namespace path | 1–3 segments (a/b/c), each ^[a-z0-9][a-z0-9_-]{0,63}$ (max 64 chars per segment) |
rejected |
| Table / field name | ^[a-z][a-z0-9_]{0,63}$ (max 64 chars); reserved names (id, created_at, _embedding, _score, _rank, rowid) are rejected, SQL keywords (key, order, group, from, values and the rest of SQLite's list) are rejected with a suggested replacement, as are the words the server's engine reserves: on PostgreSQL such as user and any, on lakehouse such as pivot, at and any (the error names the word and suggests a replacement), and a field named rank is rejected when fulltext: true (reserved by the FTS5 index); table also cannot contain __fts or start with sqlite_ |
rejected |
| Table fields | 100 user-defined fields (not counting the implicit id, created_at, _embedding columns) |
rejected |
Records per insert / upsert_by_key |
1,000 | rejected |
Full-text query |
2,048 bytes | rejected before any work, naming the limit; search for the few words that matter and narrow with filter |
NUL character (\u0000) |
not allowed anywhere in a request: a name, value, json value, key, cursor or argument | rejected with invalid_request naming the field |
Ids per read_rows |
1,000 | rejected |
Natural key fields per upsert_by_key |
8 | rejected |
| Idempotency key length | 1–256 bytes; use printable ASCII ([ -~]); omit the field for a non-idempotent insert |
empty and over-256-byte keys are rejected; the JSON Schema enforces non-empty printable ASCII for schema-validating clients |
Vector dimension (declared vector fields) |
1–4096 | rejected |
search_vector query vector |
at most 4096 numbers | rejected before the request is decoded |
Search limit (search_fulltext, search_vector) |
default 10, hard max 200 | omit limit for the default of 10; the tool schema enforces 1–200 for schema-validating clients, and the server clamps values above 200 to 200 (0 or negative selects the default on direct /v1 calls); every search response reports the limit it applied |
query result rows |
1,000 | truncated; truncated is true in the response |
query / search result bytes |
32 MiB | first row over budget errors; later rows truncate; a single BLOB value over 32 MiB always errors |
A single value built by SQL (in query or a search filter), SQLite engine |
64 MiB | query_error before the value is built |
| Request body size | 32 MiB | rejected with 413 Request Entity Too Large |
| One file | -max-file-size (1 GiB by default), and the field's max_bytes when set |
413 invalid_request naming the limit |
| Files per multipart write | 100 parts, -max-request-files-size (2 GiB by default) across them; the request part answers to the 32 MiB body limit |
invalid_request: 413 over the byte limit, 400 over 100 parts |
query args |
100 | rejected |
infer_schema samples |
1–50 | rejected |
Column label in query |
4096 bytes | rejected |
Coercion and validation rules:
number: JSON numbers and Go numeric types becomeint64when integral and within the int64 range, otherwisefloat64. A JSON number outside the int64 range is stored asfloat64, which loses precision — it is NOT rejected (9223372036854775808reads back as9223372036854776000). The separate rejection of unsigned values larger thanmath.MaxInt64applies only to the Go library, where the value arrives already typed.boolean: stored as0or1; returned astrue/false.timestamp: stored as RFC3339/ISO strings with minimal canonicalization (whitespace trimmed, lowercaset/zuppercased; offsets and date-only/space-separated forms are preserved as given). Accepted forms includeYYYY-MM-DD,YYYY-MM-DD HH:MM:SS, andYYYY-MM-DDTHH:MM:SS[±HH:MM|Z]; offsets must be ≤±23:59. Mixed-offset values do not sort chronologically as strings — normalize to UTC before storing if you order by this field.json: stored as JSON text; returned as the decoded value.vector: stored as a float32 blob. Input must be a number array of exactly the declared dimension;NaN,Inf, and out-of-range values are rejected. The 4096-dimension cap in the table above applies only to manually declaredvectorfields;vectorize: truerecords the provider's returned dimension.string/text: stored as TEXT.- Unknown field keys are rejected. Required fields missing or
nullare rejected oninsertand on the insert branch ofupsert/upsert_by_key;updateand matchedupsert/upsert_by_keyaccept partialsetmaps and only reject setting a required field tonull. queryonly acceptsSELECTorWITHstatements, rejects embedded semicolons (no multiple statements), and binds at most 100args.- On direct
/v1requests, namespace paths and table names are trimmed and lowercased before validation — a namespace per segment, sonamespace: " Production / EU "silently operates onproduction/eu. The MCP tool schemas require already-canonical names, so schema-validating clients must send trimmed lowercase names. insertwith anidempotency_key: the same key and the same records replay the original ids; the same key with different records is rejected. Use printable ASCII keys ([ -~]) up to 256 bytes.search_vectorwithtextrequires a provider and searches only the server-managed_embeddingcolumn produced by avectorize: truefield — the provider identity must match the one that embedded the table, and atextquery naming a declaredvectorcolumn is rejected. Searches with a caller-suppliedvectorneed no provider and are not checked against any embedding space — only you know which model produced the stored and query vectors.
Dolmen emits OpenTelemetry traces and metrics over OTLP (http/protobuf). Both are off by
default: nothing is recorded or exported until an OTLP endpoint is configured, and while they are
off the instrumentation is a no-op. They are configured only through the standard OTEL_*
variables (both dolmen and dolmen mcp), and each signal can be turned on or off on its own:
| Variable | Default | Effect |
|---|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT |
unset | Setting it turns traces and metrics on, e.g. http://collector:4318. |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT / OTEL_EXPORTER_OTLP_METRICS_ENDPOINT |
unset | A per-signal endpoint; setting one turns that signal on. |
OTEL_TRACES_EXPORTER / OTEL_METRICS_EXPORTER |
otlp when an endpoint is set |
otlp turns the signal on (endpoint defaults to http://localhost:4318); none keeps it off. prometheus is refused for metrics: scrape GET /metrics instead. |
OTEL_SDK_DISABLED |
false |
true turns traces and metrics off whatever else is set. |
OTEL_EXPORTER_OTLP_PROTOCOL / _TRACES_PROTOCOL / _METRICS_PROTOCOL |
http/protobuf |
Only http/protobuf is supported; gRPC is left out to keep the binary small. |
OTEL_EXPORTER_OTLP_HEADERS, _TIMEOUT, _COMPRESSION, _CERTIFICATE, and the _TRACES_ / _METRICS_ variants |
Passed to the OTLP exporters. | |
OTEL_METRIC_EXPORT_INTERVAL / OTEL_METRIC_EXPORT_TIMEOUT |
60000 / 30000 ms |
How often metrics are pushed, read by the OpenTelemetry Go SDK itself. |
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE |
cumulative |
delta or lowmemory for backends that want deltas; read by the SDK's exporter. |
OTEL_LOGS_EXPORTER |
unset (off) | otlp also sends every log line to the collector, at the same -log-level; stderr always gets them. Unlike traces and metrics, an endpoint alone does not turn log export on, since logs may carry namespace names and error details an operator may not want in a telemetry backend. |
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT / _LOGS_PROTOCOL / _LOGS_HEADERS ... |
The logs variants of the exporter settings. | |
OTEL_BLRP_* |
SDK defaults | Batch log processor tuning (OTEL_BLRP_SCHEDULE_DELAY, OTEL_BLRP_MAX_QUEUE_SIZE, OTEL_BLRP_MAX_EXPORT_BATCH_SIZE, OTEL_BLRP_EXPORT_TIMEOUT), read by the SDK itself. |
OTEL_SERVICE_NAME |
dolmen |
service.name. |
OTEL_RESOURCE_ATTRIBUTES |
Extra resource attributes (k=v,k2=v2, percent-encoded values). |
|
OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARG |
parentbased_always_on |
always_on, always_off, traceidratio, and the parentbased_* forms. |
OTEL_PROPAGATORS |
tracecontext,baggage |
tracecontext, baggage or none. |
OTEL_BSP_* |
SDK defaults | Batch span processor tuning (OTEL_BSP_MAX_QUEUE_SIZE, OTEL_BSP_MAX_EXPORT_BATCH_SIZE, OTEL_BSP_SCHEDULE_DELAY, OTEL_BSP_EXPORT_TIMEOUT), read by the OpenTelemetry Go SDK itself. |
DOLMEN_OTEL_INCLUDE_PRINCIPAL |
false |
true adds the caller's principal to operation spans as enduser.id. |
An unsupported value (a gRPC protocol, an unknown sampler or propagator) stops startup with an error
naming the variable. The resource carries service.name, service.version, service.instance.id
(random per process), host and runtime attributes. Spans are batched and flushed on shutdown, bounded
by the separate shutdown cleanup cap (at most 5s, shared with store cleanup).
What is traced:
- A SERVER span per request on
/v1/{op},/mcpand/v1/subscribe, following the HTTP semantic conventions (http.request.method,http.route,http.response.status_code,url.scheme,server.address, ...), continuing an inbound W3Ctraceparent. The subscribe span covers setup only and ends when the stream opens. - A
dolmen.op <name>span per operation, over HTTP, MCP over HTTP and MCP over stdio, withdolmen.op.name,dolmen.op.outcome(okor the error code),db.namespace,dolmen.tableanddolmen.request_id(theX-Request-Id). A failed operation sets the span's error status. - An
embeddings <model>span per embedding call (gen_ai.operation.name=embeddings,gen_ai.request.model,gen_ai.provider.name,gen_ai.usage.input_tokenswhen the provider reports it); CLIENT foropenai, which also sendstraceparentupstream, INTERNAL forlocal. - Storage spans under the operation span, SQLite engine, for the operations whose storage work sits
in one place: writes (
insert,update,upsert,upsert_by_key,delete), both searches, plain reads (read_rows,query), schema lifecycle (create_table,drop_table,create_namespace,drop_namespace),migrateandvacuum. The span is<db.operation.name> <table>(INSERT docs,SELECT notes,CREATE docs, ...) carryingdb.system.name=sqlite,db.namespaceanddb.collection.name. A write that waits for the namespace's single writer records that wait, so contention is visible; a vector search records the cache build or catch-up and the scoring pass separately, with the rows scored, the candidate count when afilternarrowed them, and whether the table was served from cache;migraterecords a span per step, andvacuumits own.queryis traced as a bareSELECT, since arbitrary SQL may span several tables: it names no table, and like every other storage span it never carries the SQL, its arguments or a row value.create_namespaceanddrop_namespaceare traced asCREATEandDROPwithdb.namespaceonly. The cheap metadata reads (list_*,describe_table,tokenize), the long-lived change feed (changes_since,wait_for,subscribe), amigratedry run androtate_secret_keyhave no storage span. - The PostgreSQL engine records the same span for the same operations, as a CLIENT span (the
database is a remote server) carrying
db.system.name=postgresql,db.namespace,db.collection.name,server.addressandserver.port; never the SQL, its arguments or the DSN. - Every log line written during a traced request carries
trace_idandspan_id. WithOTEL_LOGS_EXPORTER=otlp, exported log records carry the same trace context natively, so a backend links them to the request's trace.
What is measured (metrics are pushed every OTEL_METRIC_EXPORT_INTERVAL; with traces on too, the SDK
attaches exemplars linking a slow request's histogram bucket to its trace):
| Metric | Type, unit | Attributes |
|---|---|---|
http.server.request.duration |
histogram, s |
http.request.method, http.route, http.response.status_code, url.scheme, network.protocol.version, error.type on 5xx |
dolmen.operation.duration |
histogram, s |
dolmen.op.name, dolmen.op.outcome (ok or the error code); its count is the operation count |
dolmen.operations.in_flight |
up-down counter, {operation} |
dolmen.op.name |
dolmen.subscriptions.active |
up-down counter, {subscription} |
|
dolmen.namespaces.open |
observable up-down counter, {namespace} |
none (SQLite) |
dolmen.vector_cache.usage / dolmen.vector_cache.limit |
observable up-down counter, By |
none (SQLite) |
dolmen.db.size |
observable up-down counter, By |
none (SQLite; database files of the open namespaces, summed) |
dolmen.db.wal.size |
observable up-down counter, By |
none (SQLite; -wal files of the open namespaces, summed) |
dolmen.db.wal.largest |
observable up-down counter, By |
none (SQLite; largest single -wal, against the 64 MiB journal_size_limit the writer sets) |
db.client.connection.count |
observable up-down counter, {connection} |
db.client.connection.state (idle, used), db.client.connection.pool.name (PostgreSQL) |
db.client.connection.max |
observable up-down counter, {connection} |
db.client.connection.pool.name (PostgreSQL) |
gen_ai.client.operation.duration |
histogram, s |
gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, server.address, error.type on failure |
gen_ai.client.token.usage |
histogram, {token} |
the same, plus gen_ai.token.type=input, when the provider reports usage |
Metric attributes stay low-cardinality: never namespaces, tables, principals, request ids, paths or
client addresses, so the series count is bounded by the operation and error-code lists. The dolmen
names translate to the same Prometheus names GET /metrics serves (dolmen_operation_duration_seconds,
dolmen_operations_in_flight, dolmen_subscriptions_active), so dashboards carry over between a
scrape and an OTLP pipeline.
Privacy: spans never carry SQL text, filter arguments, row payloads, embedded text, API keys or other
credentials; error statuses carry the error code, not the message. The principal (enduser.id) is recorded only
with DOLMEN_OTEL_INCLUDE_PRINCIPAL=true. Outbound calls to the openai embedding provider carry only
traceparent; inbound W3C baggage is never forwarded to it.
docs/observability.md is the guide for running this: a one-command local
stack (Grafana + Tempo + Prometheus + Loki) to send traces, metrics and logs to, how to get from a
slow dolmen.operation.duration bucket to the trace behind it through its exemplar, and three starter
alerts (failure rate, p99 of a hot operation, embedding failures).
docs/otel-collector.yaml is a minimal OpenTelemetry Collector
configuration that receives dolmen's traces and metrics:
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 OTEL_SERVICE_NAME=dolmen-dev ./dolmen- Generate a new key:
openssl rand -base64 32. - Restart dolmen with the new key in
DOLMEN_SECRET_KEYand the old one added toDOLMEN_SECRET_KEYS_OLD(comma-separated; orDOLMEN_SECRET_KEYS_OLD_FILE, one per line). New writes use the new key at once; values written earlier still decrypt with the old one. - Call
rotate_secret_key(needsadminon*) until it answers"done": true. Each call re-encrypts at mostlimitvalues (default 10000) in short transactions, bounded by half the operation timeout, so writers are not blocked for long. It is idempotent: after a crash or a timeout, call it again and it continues where it stopped. - Check
keysin the last response: once the old key id shows 0 values (on a call withoutnamespace), remove it fromDOLMEN_SECRET_KEYS_OLDand restart.
Removing a key too early does not lose data as long as you still have it: reveals of values under
it answer internal_error (the log names the missing key id) and rotate_secret_key answers
conflict naming it; add it back and rerun. Idempotent inserts made before the rotation still
replay while the old key is configured; after it is removed, such a replay answers conflict
rather than inserting again.
dolmen import -namespace demo -table events -snapshot export-a < events.jsonl
preserves source IDs and canonical UTC-millisecond creation times in existing
tables. It sends row/byte-bounded chunks to /v1/import_records, prints committed
chunk numbers and stops on the first error. Use DOLMEN_TOKEN from the environment
for authentication. See the SQLite/Postgres snapshot recipes
for explicit mapping, counted timestamp truncation, retry keys and comparison.
A namespace is a SQLite file in WAL mode, so copying <ns>.db while the server runs can miss
commits still in the -wal file. Back up with the binary instead, which takes a consistent
snapshot of every namespace, and of the grant registry when auth is on, while writes continue.
It also copies every file the snapshot's file fields reference (unreferenced files a vacuum
would remove are left out), checking each against its SHA-256:
dolmen backup -data ./data -out ./backups/2026-09-23The backup directory must be new or empty. It gets one file per namespace, laid out as in the
data directory, and a manifest.json, written last, that records each file's size, SHA-256 and
catalog version, plus each file's path, size and SHA-256. A backup interrupted before its manifest is written is never mistaken for a
finished one.
dolmen restore -from ./backups/2026-09-23 -data ./dataRestore checks every file against its manifest, runs SQLite's integrity check, and confirms this
binary can read each catalog before it writes anything. It then moves each file into place
atomically, the files of file fields before the namespaces that reference them. It never overwrites: a namespace that already exists, or is open in a running
server, is refused. Restore into an empty data directory, or drop the namespace first. A file
already in place with the backed-up content is skipped, so a restore interrupted partway can
simply be run again.
A backup holds the data as of when it ran, so anything written afterwards is lost on restore. Run
backups as often as you can afford to lose writes, and keep them off the machine that holds
-data. The embedding model cache under <data>/models is not backed up, since it downloads
again. With -engine lakehouse (or DOLMEN_ENGINE=lakehouse) the backup also copies each table's Parquet and metadata files, and
restores only into the data directory it was taken from, because the catalogs record absolute
paths; see Lakehouse operations. On -engine postgres both commands refuse: back up with pg_dump, as
Running dolmen on PostgreSQL describes.
Dolmen uses modernc.org/sqlite, a pure-Go SQLite driver, so no CGO is required.
| Concern | Policy |
|---|---|
| Operating systems | Linux, macOS, and Windows are supported. |
| Filesystem | Local filesystems (ext4, APFS, NTFS, etc.) are required. SQLite WAL uses shared-memory coordination that does not work reliably over network or shared filesystems (NFS, SMB); these are unsupported. The server detects NFS, SMB/CIFS, AFP, WebDAV, FUSE, 9p and cluster filesystems (and Windows network drives) at startup and logs a warning naming the filesystem. |
| WAL | Enabled per namespace (journal_mode=WAL, with synchronous set by -sync: FULL by default, NORMAL when opted out). Expect <ns>.db, <ns>.db-wal, and <ns>.db-shm files. |
| Permissions | On Unix the data directory is created 0700 and namespace .db/-wal/-shm files are set 0600 (owner only). On Windows the data directory is given a protected, inheritable ACL on every open — full control for the running user, SYSTEM and BUILTIN\Administrators, and nothing inherited from the parent — so namespace files, nested namespace directories, the grant registry and the model cache are owner-only too; namespace files that already existed keep the ACL they were created with. A backup output directory gets the same treatment. Startup refuses a data directory it cannot secure or write to (for example a read-only mount), naming the directory and the fix. |
| Locking | Each namespace has one writer connection (MaxOpenConns=1) with BEGIN IMMEDIATE locking, plus a separate read-only connection pool. WAL mode allows multiple concurrent readers, but only one writer per file at a time. |
| Multi-process | SQLite's file locking makes concurrent processes safe in principle, but running two dolmen servers against the same data directory can cause database is locked errors and is not recommended. |
| Deleting a namespace | Prefer drop_namespace (confirm-guarded, closes the server's own connections first). Manually: stop the dolmen process, then delete the three <ns>.db* files. |
Quotas, replication, time travel, compaction, a UI, webhooks, an Iceberg adapter. The MVP exists to validate the tool surface with real agents.
make test # go vet + go test ./...
make race # go vet + go test -race ./...
make build # static binary
make run # run on :8790 with ./data
make vulncheck # govulncheck ./...
make release # cross-compile release binaries into dist/
make release-sbom # generate an SPDX SBOM for the source tree
make release-checksums # generate SHA256SUMS for dist/
make release-all # release binaries + SBOM + SHA256SUMS
make image # build a local container imagedolmen.Open(dir, lakehouse.With(lakehouse.Config{Sidecar: "/path/to/dolmen-duckdb", Extensions: "/path/to/duckdb-extensions"})) from github.com/lsm/dolmen/lakehouse opens a lakehouse data
directory in process, with the same rules as dolmen -engine lakehouse. The package keeps the
Iceberg and Parquet dependencies out of SQLite-only builds. Filters and Query run in the
dolmen-duckdb sidecar, so without Sidecar they answer ErrSQLEngineUnavailable.
The PostgreSQL backend is under development. It is selectable from the binary with
-engine postgres -pg-dsn <dsn> and from the Go API with postgres.With from
github.com/lsm/dolmen/postgres; SQLite remains the default on both. The implementation
includes pooled connections, namespace and table lifecycle, schema metadata, transaction
locking, native full-text search, and cross-process subscriptions, with PostgreSQL-backed
CI tests. See Running dolmen on PostgreSQL for
requirements, role provisioning, catalog layout, and operational notes, and the
implementation plan for the conformance matrix and test
instructions.
Full-text search uses each backend's native ranking: SQLite keeps FTS5
BM25, while PostgreSQL uses its native text-search index and ranking. Relevance and
linguistic matching may differ when moving datasets between backends.
Pushing a vX.Y.Z tag starts the release workflow:
git tag v0.3.0
git push origin v0.3.0Bump the fallback in internal/version/version.go when the release line changes. Release binaries
and the container image take their version from the tag via -ldflags; that constant is what a
plain go build or go install reports, so leaving it stale makes source builds misreport
themselves in -version, GET /version, MCP serverInfo, and the served-skill ETag.
The workflow creates a GitHub Release with static binaries for linux/darwin/windows on amd64/arm64, a SHA256SUMS file, and an SPDX SBOM. It also builds and pushes a multi-arch (linux/amd64 and linux/arm64) container image to ghcr.io/lsm/dolmen:
docker run --rm -it -p 127.0.0.1:8790:8790 -v dolmen-data:/data ghcr.io/lsm/dolmen:v0.3.0The image contains a single static Go binary in a gcr.io/distroless/static base. /data is exposed as a volume and is created by the container if not mounted. The default container command binds to 0.0.0.0:8790 so the port can be published from Docker. Auth is off unless you set DOLMEN_AUTH=on, so keep the published port on a private network until you do; the deployment guide covers running it for a team.
# Check a downloaded binary (skips entries for artifacts you didn't download)
sha256sum -c --ignore-missing SHA256SUMS
# Check the container image digest
oras manifest fetch ghcr.io/lsm/dolmen:v0.3.0The internal/conformance package is the contract-conformance suite: black-box
tests over the HTTP and MCP transports that pin the interface — transport
parity between /v1 and MCP tools/call, the golden error contract, the
limits table, typed-read coercion, write semantics, search invariants, and
migration guards. Behavior-changing PRs update the suite in the same PR.
Apache-2.0 — see LICENSE.