K2I Documentation

K2I is an open-source, standalone Kafka-to-Apache-Iceberg ingestion engine written in Rust. It consumes one configured Kafka topic, decodes raw or Confluent-framed Protobuf messages, keeps recent rows visible through an Arrow-backed local read path, and flushes Parquet data files through Iceberg catalog commits.

Use K2I when your Kafka events are already shaped like analytics rows and you want a single service/container instead of operating a Flink job, Spark micro-batch pipeline, or Kafka Connect cluster for this ingestion path.

Choose Your Path

I Want To Understand K2I

• Kafka to Iceberg with K2I
• Architecture
• K2I compared with alternatives
• FAQ

I Want To Run It Locally

• Getting Started
• Quickstart
• DuckDB Iceberg validation

I Need Typed Kafka Events

• Schema Registry Protobuf ingest
• Configuration reference
• CLI command reference

I Need To Operate It

• Deployment guide
• Production readiness review
• Troubleshooting

Current Release Scope

Area	Current behavior
Kafka ingest	rdkafka, manual offset management, batching, retry, and backpressure
Payload decoding	Raw values and Confluent-framed Protobuf
Schema Registry	Resolves Protobuf descriptors, caches schemas in memory and on disk, supports subject strategies
Schema evolution	Adds compatible nullable Protobuf fields and pauses readiness on breaking changes
Hot reads	Optional local read-state RPC over a Unix socket with Arrow IPC rows and committed file references
Iceberg writes	Writes Parquet files and commits real Iceberg REST metadata through iceberg-rust
Operations	HTTP health/readiness, Prometheus metrics, CLI commands, generated man pages, and Docker E2E scripts

K2I is production-oriented, but the current docs keep caveats visible. Review Production Readiness before broad rollout.

Local Proof

scripts/e2e-docker-iceberg.sh

A passing run ends with:

ok: DuckDB iceberg_scan validated real Iceberg metadata