Duckpond

An showcase for how a simple and Sovereign data stack can be constructed.

Some parts are inspired by: https://datalabtechtv.com/posts/data-lakehouse-dbt-ducklake/

What is inside:

• duckdb
• ducklake
• dependency handling: pixi
• secops
  • sops
  • age
• orchestration: dagster
• sql automation: dbt, sqlmesh
• data catalog: datahub and openmetadata
• object store: minio
• databases: postgres, pg_duckdb
• external compute: ray
• AI example
  • API calling
  • Local models

See:

• Slides
• Video
• Join one of the live sessions

Prerequisites

• pixi: curl -fsSL https://pixi.sh/install.sh | sh
• minio cli brew install minio/stable/mc
  • via i.e. homebrew https://brew.sh/
  • direct
    • https://docs.min.io/enterprise/aistor-object-store/reference/cli/
    • https://github.com/minio/mc

Starting the background services

# set up the environment variables
# this chooses insecure defaults; if you use this for anything else than the local demo - ensure secure passwords and certificates are used
mv example.env .env

# we want to enable local dev ssl
# duckpond.data-engineering.expert points to local host
# but we want some (locally trusted) SSL cert for it
pixi run -e ops --frozen setup

# starts all needed background services
docker compose build
docker compose up -d
# - databases
# - object stores
# - object store UIs

## catalogs
docker compose --profile catalogs up -d
### for open metadata
docker compose --profile om up -d
### for datahub
docker compose --profile datahub up -d


# quick check - ensure the minio CLI works
mc alias set duckpond https://objectstore.duckpond.data-engineering.expert ${OBJECT_STORE_ROOT_USER} ${OBJECT_STORE_ROOT_PASSWORD}
mc mb duckpond/test-bucket
mc ls duckpond

services:

• HTTPS
  • https://objectstore.duckpond.data-engineering.expert
  • https://objectstore-ui.duckpond.data-engineering.expert
  • data catalog
    • profile om, catalogs
      • https://catalog.duckpond.data-engineering.expert
        • user admin@duckpond.data-engineering.expert
        • password admin
    • profile datahub, catalogs
      • https://datahub.duckpond.data-engineering.expert
        • user datahub
        • password datahub
• TCP
  • database: db.duckpond.data-engineering.expert:5433, SSL enabled

Explore the tutorials in the projects folder.

The main combined one is described in ./projects/100_combined/README.md

The slides can be started with:

make slides

make slides slides-vde-2025-10.md

• https://georgheiler.com/post/dbt-duckdb-production/
• inspired by
  • https://datalabtechtv.com/posts/data-lakehouse-dbt-ducklake/
    • This license (apache 2) applies for the content https://github.com/DataLabTechTV/datalab/blob/main/LICENSE
  • https://datalabtechtv.com/posts/graphrag-with-kuzudb/
• https://github.com/l-mds/local-data-stack/
• https://medium.com/@jairus-m/materializing-multi-asset-rest-api-sources-with-dagster-and-dlt-e8da07dcdb83

Want to dig deeper?

Reach out to Georg to discuss how you can solve your challenges with data.