diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml new file mode 100644 index 0000000..993ee60 --- /dev/null +++ b/.github/workflows/pages.yml @@ -0,0 +1,41 @@ +name: Pages + +# Publishes the marketing site in `site/` to GitHub Pages +# (https://dmux.github.io/minilake/). Requires the repository's Pages source to +# be set to "GitHub Actions" -- Settings -> Pages -> Build and deployment. +on: + push: + branches: [main] + paths: + - "site/**" + - ".github/workflows/pages.yml" + workflow_dispatch: + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: pages + cancel-in-progress: false + +jobs: + deploy: + runs-on: ubuntu-latest + + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + + steps: + - uses: actions/checkout@v4 + + - uses: actions/configure-pages@v5 + + - uses: actions/upload-pages-artifact@v3 + with: + path: site + + - id: deployment + uses: actions/deploy-pages@v4 diff --git a/README.md b/README.md index a7d81f5..cb2341c 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@

- Documentation · GitHub · Container Image (GHCR) + Website · Documentation · GitHub · Container Image (GHCR)

--- diff --git a/site/.nojekyll b/site/.nojekyll new file mode 100644 index 0000000..e69de29 diff --git a/site/README.md b/site/README.md new file mode 100644 index 0000000..87a99c9 --- /dev/null +++ b/site/README.md @@ -0,0 +1,27 @@ +# `site/` — the project website + +The landing page published to GitHub Pages at . + +It is a single self-contained `index.html` (inline CSS and JS, no build step, no +dependencies) plus the images in `assets/`. Edit it and push to `main`; the +`Pages` workflow uploads this directory as-is. + +Preview it locally with any static server: + +```bash +python3 -m http.server 4599 --directory site +# http://localhost:4599 +``` + +## Assets + +| File | What it is | +|---|---| +| `assets/minilake-logo.png` | `minilake_logo.png` from the repo root, background knocked out and trimmed | +| `assets/og.png` | 1200×630 social preview card (`og:image`) | +| `assets/favicon-32.png`, `favicon-64.png`, `apple-touch-icon.png` | Icons derived from the same mark | + +`.nojekyll` keeps GitHub from running the files through Jekyll. + +Content on the page is drawn from `README.md`, `FEATURES.md` and `docs/` — when a +feature's status changes there, update the "Coverage" table here too. diff --git a/site/assets/apple-touch-icon.png b/site/assets/apple-touch-icon.png new file mode 100644 index 0000000..d7cbed3 Binary files /dev/null and b/site/assets/apple-touch-icon.png differ diff --git a/site/assets/favicon-32.png b/site/assets/favicon-32.png new file mode 100644 index 0000000..85cf7bb Binary files /dev/null and b/site/assets/favicon-32.png differ diff --git a/site/assets/favicon-64.png b/site/assets/favicon-64.png new file mode 100644 index 0000000..af7dc78 Binary files /dev/null and b/site/assets/favicon-64.png differ diff --git a/site/assets/minilake-logo.png b/site/assets/minilake-logo.png new file mode 100644 index 0000000..b6ab598 Binary files /dev/null and b/site/assets/minilake-logo.png differ diff --git a/site/assets/og.png b/site/assets/og.png new file mode 100644 index 0000000..77cadf1 Binary files /dev/null and b/site/assets/og.png differ diff --git a/site/index.html b/site/index.html new file mode 100644 index 0000000..e04b0f7 --- /dev/null +++ b/site/index.html @@ -0,0 +1,1063 @@ + + + + + +MiniLake — The Databricks API, running on your laptop + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+ + +
+
+
+
+ Open source · MIT · Python 3.11+ · Docker +

The Databricks API,
running on your laptop.

+

+ MiniLake is a local Databricks API emulator with real SQL, + real Delta Lake and real Spark job execution. Point + databricks-sdk, the Databricks CLI, Terraform, Asset Bundles or an LLM agent at + http://localhost:8000 — no account, no API key, no cloud bill. +

+ +
+
+ $ + pip install minilake && minilake --port 8000 + PyPI + +
+
+ $ + docker run -p 8000:8000 ghcr.io/dmux/minilake:latest + GHCR + +
+
+ + +

Then open http://localhost:8000/ui/ for the built-in SQL workspace. The image downloads nothing at runtime — it works air-gapped.

+
+ +
+ MiniLake logo +
+
+
+
+ + +
+
+ Drop-in for +
+ databricks-sdk (Python) + Databricks CLI + Terraform provider + Asset Bundles + PySpark + Delta + MCP agents +
+
+
+ + +
+
+
+
16+Databricks API groups emulated, from Unity Catalog to Secrets
+
67MCP tools for LLM agents, served on the same port
+
138+tests driven by the real Databricks SDK, not mocks
+
$0cloud spend — everything runs on your machine
+
+
+
+ + +
+
+
+ What you get +

Not a mock server. A real engine behind a Databricks-shaped API.

+

Fake endpoints let broken code pass. MiniLake executes your SQL on DuckDB, writes real Delta files, and runs real spark-submit jobs in containers — so a green test locally means something.

+
+ +
+
+
+

Unity Catalog, for real

+

Catalogs, schemas, tables and volumes with full CRUD. Every catalog is its own DuckDB database file, so catalog.schema.table addressing is native — not a regex rewrite.

+ Real execution +
+ +
+
+

SQL that actually runs

+

Statement execution on real DuckDB: INLINE and EXTERNAL_LINKS dispositions, JSON_ARRAY / ARROW_STREAM / CSV formats, and a result manifest that carries true column types.

+ Real execution +
+ +
+
+

Real Delta Lake files

+

EXTERNAL Delta tables are genuine Parquet plus a _delta_log/. INSERT, UPDATE and DELETE are routed through a real Spark job; reads go through DuckDB's delta_scan().

+ Real Delta +
+ +
+
+

Jobs that really execute

+

spark_python_task runs spark-submit in a sibling container and returns actual stdout. Real DAG scheduling honours depends_on and run_if, with independent branches in parallel.

+ Real Spark +
+ +
+
+

Unity Catalog protocol for Spark

+

spark.table("cat.sch.tbl") resolves against MiniLake through the official UCSingleCatalog connector — the same protocol Spark speaks to a real workspace.

+ Wire-compatible +
+ +
+
+

A built-in SQL workspace

+

An Athena-style editor at /ui: Monaco with catalog-aware completion, a virtualized result grid, charts, CSV/JSON export, saved queries, query history — and an embedded JupyterLab.

+ Batteries included +
+ +
+
+

Terraform & Asset Bundles

+

Point the databricks provider at localhost and terraform apply creates real catalogs. databricks bundle deploy and bundle run work end to end — jobs execute for real.

+ Wire-compatible +
+ +
+
+

An MCP server for agents

+

67 tools, 8 resources and 4 prompts at /mcp — including composites that collapse a five-call sequence into one, and a SQL-dialect resource that stops agents writing Spark SQL.

+ Real execution +
+ +
+
+

Designed for CI

+

One POST /_minilake/reset gives every test a clean workspace without restarting the container. /health, /ready and /services round out the control plane.

+ Test friendly +
+ +
+
+

Workspace, DBFS, Files, Secrets

+

File-backed notebook and file storage with chunked uploads. Secrets are real CRUD whose values are only ever resolvable inside a job's environment — exactly like Databricks.

+ Real execution +
+ +
+
+

Air-gapped by design

+

DuckDB's delta extension and the Delta / Unity Catalog jars are baked in at build time. The image downloads nothing at runtime — provable with docker run --network none.

+ Offline ready +
+ +
+
+

Databricks-shaped errors

+

Failures come back as {"error_code", "message"}, so the SDK raises its normal typed exceptions. Anything out of scope returns a loud 501 NOT_IMPLEMENTED instead of a convincing lie.

+ Wire-compatible +
+
+
+
+ + +
+
+
+
+ Zero code changes +

Change the host. That's the migration.

+

+ MiniLake speaks the Databricks REST API on the wire, so every client you already use keeps working — + the same calls, the same response shapes, the same typed errors. +

+
    +
  • Any token is accepted. No sign-up, no PAT, no OAuth dance to develop against.
  • +
  • Types round-trip. STRING, LONG, DECIMAL(10,2), ARRAY<T>, MAP and STRUCT map to DuckDB and come back as the SDK's ColumnInfo.
  • +
  • Physical truth wins. Declared column types that disagree with the files lose, so mismatches are visible instead of believed.
  • +
  • Persistence when you want it. MINILAKE_PERSIST=1 snapshots state on shutdown and restores it on start.
  • +
+
+ +
+
+ + + + + +
+ +
+
+ + quickstart.py + +
+ +
from databricks.sdk import WorkspaceClient
+from databricks.sdk.service.catalog import ColumnInfo, TableType
+
+# The only line that differs from production
+w = WorkspaceClient(host="http://localhost:8000", token="dev")
+
+w.catalogs.create(name="sales")
+w.schemas.create(name="store", catalog_name="sales")
+w.tables.create(
+    name="orders", catalog_name="sales", schema_name="store",
+    table_type=TableType.MANAGED,
+    data_source_format="DELTA", storage_location="/data/sales/store/orders",
+    columns=[
+        ColumnInfo(name="id", type_text="BIGINT"),
+        ColumnInfo(name="customer", type_text="STRING"),
+        ColumnInfo(name="amount", type_text="DECIMAL(10,2)"),
+    ],
+)
+
+wh = w.warehouses.create(name="dev", cluster_size="Small")
+w.statement_execution.execute_statement(
+    warehouse_id=wh.id,
+    statement="INSERT INTO sales.store.orders VALUES (1, 'Ana', 459.90)",
+)
+
+result = w.statement_execution.execute_statement(
+    warehouse_id=wh.id,
+    statement="SELECT customer, amount FROM sales.store.orders",
+)
+print(result.result.data_array)   # [['Ana', '459.90']] — really stored
+ + + + + + + + +
+
+
+
+
+ + +
+
+
+ How it works +

One process. One port. Real engines underneath.

+

A FastAPI application serves the Databricks REST surface, the web UI and the MCP endpoint together. Behind it sit DuckDB, real Delta files on disk, and Docker for Spark.

+
+ +
+ + + + + + + + + + + + CLIENTS + + + databricks-sdkWorkspaceClient (Python) + + Databricks CLIbundle deploy · bundle run + + Terraform providerapply · destroy + + PySpark / JupyterLabspark.table("cat.sch.tbl") + + LLM agentsModel Context Protocol + + BrowserSQL workspace at /ui + + + + + + + + + + MINILAKE · ONE PROCESS + + MiniLake + FastAPI · uvicorn · :8000 + + + /api/2.0 · /api/2.1 + Databricks REST surface + + + /ui · /jupyter + SQL workspace + notebooks + + + /mcp + 67 tools for agents + + + /_minilake/* + health · ready · reset + + + + + + REAL ENGINES + + DuckDB + one database file per catalog + + + Delta Lake on disk + parquet + _delta_log/ + + + Docker → Apache Spark + spark-submit, sibling container + + + + Spark writes the files — + DuckDB reads the same bytes + +
+
+
+ + +
+
+
+
+
+ For LLM agents +

Give an agent a lakehouse it can't break.

+

+ A Databricks workspace is an awkward thing for an agent to drive: every useful action is four calls + deep, half the SQL it writes is the wrong dialect, and failures are silent. MiniLake's MCP server + closes those gaps — and it costs nothing when the agent gets it wrong. +

+
    +
  • setup_fixture()catalog + schema + warehouse in one idempotent call
  • +
  • seed_table()table + rows, exactly the rows you asked for
  • +
  • describe_catalog_tree()the whole hierarchy instead of dozens of calls
  • +
  • run_python_script()stage, submit to real Spark, poll, return stdout
  • +
+

+ Plus 8 resources — including a SQL-dialect guide that stops agents writing Spark SQL at DuckDB — + and 4 prompts that front-load the rules. Tools run through MiniLake's own ASGI stack, so an agent's + error is exactly the error an SDK client would have seen. +

+ +
+ +
+
+
+ + register + use +
+
# One line to register it with Claude Code
+claude mcp add --transport http minilake http://localhost:8000/mcp
+
+# Then just ask:
+#   "Use minilake: create a sales catalog with a seeded
+#    orders table, and tell me the total."
+
+# The agent does this — three calls, no glue code:
+#   setup_fixture({"catalog": "sales", "schema": "store"})
+#   seed_table({"full_name": "sales.store.orders", ...})
+#   run_sql({"statement": "SELECT SUM(amount) FROM sales.store.orders"})
+
+# Enable it on the server side:
+MINILAKE_MCP=1 minilake --port 8000
+
+
+
+
+
+
+ + +
+
+
+ Coverage +

What's emulated, and how honestly.

+

Every row below is either really executed or clearly labelled. Anything out of scope answers 501 NOT_IMPLEMENTED rather than faking a success your production code will not get.

+
+ +
+ + + + + + + + + + + + + + + + + + + + + +
ServiceStatusWhat that means
Unity CatalogRealCatalogs, schemas, tables, volumes — each catalog is its own DuckDB database, with native three-part addressing
SQL statementsRealExecuted on DuckDB; all three formats and both dispositions, with a typed result manifest
EXTERNAL Delta tablesRealGenuine Delta files; writes go through a generated Spark job, reads through delta_scan()
JobsRealspark-submit in sibling containers (or a subprocess fallback), real DAG scheduling, real logs
SQL warehousesRealFull CRUD and lifecycle; statements run regardless of state, as they do locally
Query history & saved queriesRealServer-backed, failures included — which is usually why you opened history
Workspace, DBFS & FilesRealFile-backed storage with chunked upload; raw-bytes sync is what makes bundle deploy work
SecretsRealReal CRUD; values are resolvable only inside job environment variables, never through the API
Web UI & notebooksRealAthena-style SQL workspace at /ui, with JupyterLab proxied at /jupyter
MCP serverRealOpt-in with MINILAKE_MCP=1: 67 tools, 8 resources, 4 prompts on the same port
PersistenceRealMINILAKE_PERSIST=1 snapshots state as JSON on shutdown and restores it on startup
ClustersState machineCRUD plus timed lifecycle transitions — deliberately no Spark compute; that lives in Jobs
PermissionsAllow-all CRUDReal create/read/update, but nothing is ever enforced — this is a single-user tool
Identity (SCIM)StaticA fixed current-user endpoint; there is exactly one user and any token is accepted
DLT, Model Registry, Vector Search, Dashboards, ReposNot implementedReturns 501 NOT_IMPLEMENTED — loudly, so you never mistake it for working
+
+
+
+ + +
+
+
+ Straight talk +

What MiniLake is not.

+

These are deliberate design decisions, not a backlog. MiniLake targets one developer running it locally — not a shared, multi-tenant server.

+
+
+

No authentication

Any token is accepted and there is exactly one user. Never expose it beyond localhost.

+

No access control

The Permissions API is real CRUD that always allows everything — a passing test here says nothing about grants in a real workspace.

+

DuckDB, not Spark SQL

Statements run on DuckDB's dialect. The MCP server ships a dialect resource for exactly this reason; the docs carry the translation table.

+

No Spark on clusters

The cluster lifecycle is a state machine. Real Spark happens through Jobs, in sibling containers.

+

Single process, no HA

DuckDB is single-writer, so concurrent load serializes on a lock. Fine for dev and CI; not a deployment target.

+

MANAGED tables are invisible to Spark

They are DuckDB tables with no files. Use EXTERNAL Delta when Spark has to see the data — that trade-off is what keeps SQL fast.

+
+

Being explicit about the edges is the point: an emulator that quietly fakes what it cannot do is worse than no emulator at all.

+
+
+ + +
+
+
+ +

Your lakehouse, on localhost, in one command.

+

Free and MIT-licensed. No account, no API key, no cloud bill — and nothing to uninstall from your cloud account when you're done.

+ +
+
+
+ +
+ + + + + +