DuckDB extension for Apache Iceberg

This repository contains DuckDB's Apache Iceberg extension. It adds support for reading and writing Iceberg tables, inspecting Iceberg metadata, and attaching Iceberg REST catalogs.

User-facing documentation is available on the Iceberg extension page.

For format version 3 tables, rows inserted within a transaction have NULL _row_id and _last_updated_sequence_number until commit. An UPDATE preserves an already assigned _row_id, including through repeated updates in the same transaction, but its new _last_updated_sequence_number remains NULL until commit. Rows from before a v2-to-v3 upgrade also retain NULL lineage until a v3 snapshot assigns it.

Each data-changing statement publishes a snapshot at commit. A row inserted and then updated within one transaction can receive a different _row_id in each of those snapshots: the updates cannot preserve an ID that has not been assigned yet. After commit, further updates preserve its assigned ID. Use a business key to identify pending rows, rather than relying on their nullable lineage columns.

Clone the repository with its DuckDB and extension tooling submodules:

git clone --recurse-submodules https://github.com/duckdb/duckdb-iceberg.git
cd duckdb-iceberg

If the repository is already cloned, initialize the submodules with:

git submodule update --init --recursive

Building requires a C++ toolchain, CMake, and vcpkg. The repository supplies its own vcpkg manifest and ports; pass the vcpkg toolchain file when building.

Catalog-backed tests and test-data generation have additional requirements:

• Python 3 and the packages in scripts/requirements.txt
• Java compatible with the selected PySpark runtime
• Docker and Docker Compose for the local REST catalogs
• Git for catalog repositories cloned under .catalogs/

The catalog data targets create .venv-spark4 and install the pinned Python requirements automatically. make install_requirements installs those requirements into the currently active Python environment instead.

Set the vcpkg toolchain path for each invocation or export it in your shell:

export VCPKG_TOOLCHAIN_PATH=/path/to/vcpkg/scripts/buildsystems/vcpkg.cmake
make debug

Common build targets are:

make debug       # Debug build with assertions and sanitizers
make release     # Optimized release build; also the default `make` target
make reldebug    # Optimized build with debug information
make relassert   # Optimized build with assertions

For a release build, the resulting DuckDB shell and loadable extension are:

build/release/duckdb
build/release/extension/iceberg/iceberg.duckdb_extension

The top-level Makefile delegates standard build, test, and formatting targets to extension-ci-tools/makefiles/duckdb_extension.Makefile. It builds Iceberg together with the required test extensions configured in extension_config.cmake.

Build the matching configuration before running its tests. The default test target uses the release test binary:

make release
make test

Other configurations can be tested explicitly:

make debug
make test_debug

make reldebug
make test_reldebug

Use T to narrow a Make-based run, or invoke the sqllogictest runner directly:

make test_debug T=test/sql/local/iceberg_scans/

./build/debug/test/unittest \
  test/sql/local/iceberg_scans/iceberg_scan.test

Run formatting checks and fixes with:

make format-check
make format-fix

The catalog targets start local services, generate compatible Iceberg test data, and record the selected catalog in .catalogs/.active_catalog.

Starting a catalog stops the catalog currently named in .catalogs/.active_catalog. To stop one explicitly, use make <catalog>-stop. Catalog clones, runtime state, and generated data are kept in ignored directories.

To generate only the data needed by one registered generator, pass a pytest -k expression through TEST:

TEST=all_types_table make fixture-data

The fixture also supports generation for local file-based tests:

make fixture-data-local

Each REST catalog has a config in test/configs/. A config initializes credentials and the my_datalake attachment, statically loads required extensions, sets CATALOG_TEST_CONFIG_SETUP, and lists catalog-specific skips where necessary.

After starting a catalog, resolve its config from .catalogs/.active_catalog by running the helper directly:

./build/debug/test/unittest --order lex \
  "$PWD/test/sql/local/catalog_test_config_setup/*" \
  --test-config "$(scripts/catalog_test_config.sh)"

To run one file:

./build/debug/test/unittest \
  --test-config "$(scripts/catalog_test_config.sh)" \
  test/sql/local/catalog_test_config_setup/catalog_agnostic/create/test_create_table.test

scripts/catalog_test_config.sh accepts fixture, fixture-latest, gravitino, lakekeeper, nessie, polaris, or mock. It reports an error for a missing, empty, local-only, or unknown active-catalog marker.

For the mock, run the catalog-agnostic SQL suite excluding .test_slow, with capability flags and skips controlled by test/configs/mock.json. You can pass that config directly after make mock; see the mock catalog instructions. It has no Spark generator or Python integration profile. Stop it with make mock-stop.

The Python integration tests also use the active catalog and require a built unittest binary:

python3 -m pytest -vv test/python \
  --unittest-binary ./build/debug/test/unittest \
  --test-python-verbosity verbose

Verbose mode prints the resolved catalog and Spark runtime and explains capability-based skips. Cloud-only tests under test/python/cloud/ are excluded unless that directory is requested explicitly.

The generator under scripts/data_generators/ uses PySpark and Iceberg to populate the active catalog. Generator cases live under scripts/data_generators/tests/, and intermediate results are written below data/generated/intermediates/.

See scripts/data_generators/README.md for instructions on adding generator cases, catalog connections, skips, and expected failures.

This extension was initially developed as part of a customer project for RelationalAI, who agreed to open source it. We thank RelationalAI for supporting open source development and enabling us to share the extension with the community.