Skip to content
imbue-aiPublic

About

Datalib ✊ is about personal data sovereignty. It liberates your data from silos and helps you run powerful AI tools on it, locally, on your own terms.

Resources

Security policy

Stars

36 stars

Watchers

0 watching

Forks

Latest commit

 

History

5,675 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Datalib ✊

Organize the world's (struck out) your own information and make it universally (struck out) accessible and useful to you, your way.

Datalib is about personal data sovereignty. It liberates your data from silos and helps you run powerful AI tools on it, locally, on your own terms.

Note

Datalib is alpha. We don't promise stable bytes at rest yet: an upgrade may change how your data is stored on disk, and now and then one may ask you to re-download a source. We aim to carry every data folder forward through upgrades, and to make that a promise soon.

Datalib mirrors your personal data — chats, email, messages, contacts, documents, photos, health and fitness — out of the services that hold it and into one place you own: a folder on your own computer, in open formats, with history. Once it is there you can search it, join it across sources, and point whatever tools you like at it, without asking anyone's permission.

Think of it as the data warehouse and pipelines a big company runs over its own data, in laptop-sized packaging. Hat tips to Perkeep and Dogsheep, which got there first.

Datalib is an early Imbue project. It is read-only today: it brings data in and never writes anything back to a source.

Get started

Three ways in, from least to most hands-on:

  1. The desktop app (macOS, Apple Silicon). Download the .dmg from the latest release. It asks which folder to keep your data in, then opens the app; you add your first source from the Data sources card.
  2. The command-line tools (macOS or Linux). One curl | sh installs them. The first-time user guide walks through install, credentials, the config file, and your first sync.
  3. Hand it to your agent. Point an AI coding agent at this repository and ask it to set you up. docs/agent_user.md is written for an agent running Datalib on your behalf, and once the tools are installed datalib-step help prints it and every doc it links to, no checkout needed; AGENTS.md is for one working on the code.

Cautious? The Docker image keeps the binaries and your credentials inside a container that sees only the folders you mount, and it comes with a demo library already loaded, so you can look before you hand it anything of yours.

Read this before you point an agent at it

Datalib is Plain Old Software. Running a sync invokes no cloud AI model and no agent, and sends no telemetry: the only network traffic is Datalib reading from the services you configured, which means sending them your sign-in and your requests, plus one-time downloads the first time they are needed: the search model, its CUDA module on a Linux machine with an NVIDIA GPU, and (for the command-line install) the Node runtime that latchkey runs on. A sign-in that opens a browser also loads that service's pages, and may first download the browser. A package you add is fetched from the URL you gave it, and runs as you. The one exception is opt-in: a source you set to embed on Modal sends its text to your own Modal account, to be embedded there (what leaves the machine for Modal).

What it produces, though, is a very valuable pile of private data in one place — and most of it was written by other people. Three things follow:

  • Credentials. Web sources authenticate through latchkey, which keeps live session cookies and API tokens on your machine. Any process that can run commands as you can use them to act as you on those services, with no further prompt. Only do this on a machine you trust. macOS can lock a secret so only one signed app reads it, but Datalib doesn't do that yet, and such a lock is weaker than it sounds: who can use your sign-ins, and why.
  • Agents. Before you let an agent loose on the mirror, read about the lethal trifecta. Treat everything in your mirror as both private and untrusted content. And remember that an agentic harness sends what it reads to a model provider: ask yourself whether the people who wrote you those messages would be fine with that.
  • Terms of service. The Claude.ai, ChatGPT and Garmin sources talk to the same undocumented APIs those services' own apps use, signed in as you. It is your data, but check the terms of the services you use, and know that those APIs can change without notice.

Supported data sources

Every source links to its section of Getting your data: what it mirrors, and how to get at yours — a login kept by latchkey, an export, or a backup pulled off a phone.

Top tier

The sources we rely on ourselves, and the ones we keep working first.

Connected accounts


Calendar (CalDAV)

ChatGPT

Claude

Contacts (CardDAV)

Email (mbox or JMAP)

Fastmail

Fastmail Calendar

Fastmail Contacts

Gmail

Google Calendar

Google Drive

Linear

Notion

Slack

On this computer


Claude Code (Local)

Codex (Local)

Markdown (Local)

Best effort / community supported

These work, but few of us run them on our own data, so a change on the service's side can go unnoticed until someone who does reports it. Reports and fixes are welcome.

Connected accounts


Baby Connect

Calendly

Garmin

GitHub

GitLab

Todoist

YoLink

Exports & backups


Facebook

Google Chat (Takeout)

Google Takeout

Google Voice (Takeout)

LinkedIn

Signal backups

SMS Backup & Restore

WhatsApp backups

On this computer


Apple Mail (Local)

Apple Messages (Local)

Apple Photos (Local)

Beeper (Local)

File inventory (Local)

IQAir AirVisual (Local)

Lightroom (Local)

Media (Local)

PDFs (Local)

Perseus Digital Library

A source's type says what is being mirrored (claude, whatsapp, …). Its ingest step says how, with one table named for the method: [steps.params.api] reads the product's own API, [steps.params.export] an unpacked export, [steps.params.backup] a phone backup, [steps.params.fswalk] a folder on disk, and so on (jmap, mbox, caldav, …). So a claude source pulled from the API and one read from an export share a type and differ only in that table. Every shape, fully commented, is in all_sources.toml. Something not here? Any program that speaks the step protocol is a source.

How it works

Two layers.

The lower layer is a small, unopinionated pipeline runner. A data store is a file in a folder. A data processor is a program, in whatever language you like. datalib-pipeline arranges those programs into a graph (a DAG), runs it, and re-runs only the steps whose inputs changed. Any executable that speaks a small NDJSON protocol can be a step — see docs/dev/step_protocol.md.

The upper layer is the batteries. Each source above comes with ready-made steps:

  • ingest brings the raw data in.
  • render_markdown turns the raw records into readable markdown (most sources have one).
  • embed adds semantic search, which matches on meaning rather than words. Its embedding model runs on your own machine, unless you choose to run it on your own Modal account. It is slow, so you can turn it off per source.

Across all the sources, unified_grid_and_search_index builds one SQL table of every message and document (grid_rows) and puts their words in a keyword search index beside it, and a single search box reads both. A local web UI, also shipped as a desktop app (Tauri), manages your sources and syncs, searches, and browses the results.

The stores are doltlite: SQLite's engine over a versioned, content-addressed file format, so a database file is also a git-shaped history of itself. Every sync is a commit. That is what makes the pipeline incremental — each stage asks "what changed since the commit I last read?" rather than rescanning — and it is what keeps the record of what a source changed or deleted between syncs.

Deltas are things too. If you can render a collection of things, consider rendering the difference between two versions of that collection — what was added, what was removed, what changed and how. Code has had this for fifty years; almost nothing else does, and most of the questions people bring to their own data are questions about change. Two mechanisms carry it here:

  • Every store keeps its history. A raw store's commits are the syncs; a render store's commits are the renders. datalib-doltlite reads either at any commit or diffs any two (dolt_log, dolt_diff), and a source's commit history in the app shows what each commit did to each table.
  • A comparison is a source of its own. "Compare two versions…" on a source makes a diff group: the source's own renderer run at both commits and subtracted, written as an ordinary source. Its documents carry the changes marked — added and removed sections on green and red, edited words inside a modified one — and its grid rows say added, removed or modified and which columns moved, so everything that works on a source works on the difference.

What we are aiming for

Near term, we want to be able to ingest and understand many data sources:

  • Big tent — popular and unpopular sources alike, discovering each one's schema rather than forcing it into ours.
  • Local-first — file-based storage; views and processing work offline.
  • Incremental — cheap to keep up to date.
  • Stable identity — a message keeps its id through content edits, so links to it survive.
  • Versioned — notice when the upstream loses or edits data. The history is in the stores, and a diff group renders what changed between two syncs as a source of its own.
  • Legible — render raw data from many schemas into markdown.
  • Findable — search by metadata, keywords, and vectors.
  • Read-only (for now) — ingest-only views of every source.

Longer term: all your data in one place instead of one app per data type; your own apps that join data across sources; as much of your data as possible in your own hands, in formats you can use; and publishing it back into other apps.

Getting your data out again

A mirror you can't leave is just another silo, so the exits are plain:

  • Markdown — datalib-step markdown <data_root> --out <folder> writes every conversation or document as an ordinary .md file, one each, from the render stores that keep them.

  • Search — datalib-applet search <data_root> '<query>' answers as the app's search box does, as JSON, with no server: for scripts and agents.

  • SQL — the grid's table, unified_grid_and_search_index/unified_grid_and_search_index/db.sqlite, is plain SQLite. Each source's own stores are doltlite: datalib-doltlite ships with the tools and is a sqlite3 shell that understands the versioned format, and one pipe writes a plain SQLite file for any tool that wants one:

    datalib-doltlite -readonly slack/ingest/entities.doltlite_db .dump | sqlite3 slack.sqlite

    From Python, uv add imbue-datalib-reader reads a store directly (Reading the raw stores).

What an export carries and what it leaves behind, and the GUIs that open a store, are in the first-time user guide's Getting your data back out; the commit history and per-sync diffs in docs/dev/doltlite.md.

Documentation

License

MIT. A release carries the notices of the third-party software it bundles under licenses/ (in the .app, Contents/Resources/licenses/).

About

Datalib ✊ is about personal data sovereignty. It liberates your data from silos and helps you run powerful AI tools on it, locally, on your own terms.

Resources

Security policy

Stars

36 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages