Introducing K2I — The Kafka-to-Iceberg engine your lakehouse deserves.

Real-time streaming ingestion for Apache Iceberg

K2I bridges the gap between Kafka and your data lakehouse. Sub-second freshness. Exactly-once delivery. Zero operational overhead.

01

Stream

Kafka ingestion with exactly-once delivery.

02

Buffer

Sub-second freshness via Arrow hot storage.

03

Persist

Auto-maintained Iceberg tables.

MACOS / LINUX
> curl -fsSL https://k2i.dev/install | sh
Trusted by teams at
Problem Statement

Modern data teams want Apache Iceberg for their analytics workloads.

Open table formats, time travel, schema evolution, and cost-efficient object storage. But getting data from Kafka into Iceberg is harder than it should be.

The Latency-Cost Trade-off

Real-time streaming: milliseconds, but expensive and complex.

Batch ETL: cost-efficient, but hours of delay.

You shouldn't have to choose.

The Small File Problem

Streaming into Iceberg creates thousands of tiny files per hour.

  • Query performance degrades
  • Metadata explodes
  • Storage costs spiral

Compaction becomes a full-time job.

Exactly-Once Nightmares

Coordinating Kafka offsets, object storage writes, and catalog commits is complex.

One failure = data loss or duplication.

You need guarantees, not hope.

pipeline-01
RUNNING
Throughput 1.2 GB/s
Latency 42ms
pipeline-02
OPTIMIZING
Compacting small files... 75%
The Solution

K2I: Real-time Iceberg, simplified

K2I is a purpose-built streaming ingestion engine that solves the Kafka-to-Iceberg problem once and for all. A single Rust binary that runs anywhere, handles everything, and just works.

Feature Highlights

Sub-Second Freshness

K2I maintains an in-memory hot buffer using Apache Arrow. New events are immediately queryable while being accumulated for optimal file writes.

< 1 second end-to-end latency from Kafka to queryable

Exactly-Once Delivery

A write-ahead transaction log coordinates Kafka consumption, object storage writes, and catalog commits. Crash at any point? K2I recovers automatically.

Zero data loss even through crashes and restarts

Zero Operations

Auto-compaction, snapshot expiration, and orphan file cleanup. No cron jobs. No manual intervention.

1 binary no cluster, no dependencies, no ops
How It Works

A single process that does everything right

K2I is a single-process, embedded engine inspired by the Moonlink architecture. It combines in-memory hot buffering with durable cold storage, delivering real-time freshness with lakehouse economics. 

+-----------------------------------------------------------------------------+
|                              K2I ENGINE                                     |
+-----------------------------------------------------------------------------+
|                                                                             |
|      INGEST                   BUFFER                    PERSIST             |
|                                                                             |
|   +------------------+      +------------------+      +------------------+  |
|   |                  |      |                  |      |                  |  |
|   |  Kafka Consumer  |----->|   Hot Buffer     |----->|  Iceberg Writer  |  |
|   |                  |      |   (Arrow)        |      |  (Parquet)       |  |
|   |  - Backpressure  |      |                  |      |                  |  |
|   |  - Retry logic   |      |  - O(1) lookups  |      |  - Atomic commit |  |
|   |  - Batch collect |      |  - TTL eviction  |      |  - CAS conflicts |  |
|   |                  |      |  - Memory bound  |      |  - Multi-catalog |  |
|   +------------------+      +------------------+      +------------------+  |
|            |                        |                        |              |
|            |                        |                        |              |
|            v                        v                        v              |
|   +---------------------------------------------------------------------+   |
|   |                          TRANSACTION LOG                            |   |
|   |                                                                     |   |
|   |      Append-only  |  CRC32 checksums  |  Periodic checkpoints       |   |
|   |      Crash recovery  |  Exactly-once  |  Idempotency tracking       |   |
|   +---------------------------------------------------------------------+   |
|                                                                             |
+-----------------------------------------------------------------------------+
                                      |
                                      v
+-----------------------------------------------------------------------------+
|                          YOUR INFRASTRUCTURE                                |
+-----------------------------------------------------------------------------+
|                                                                             |
|   +------------------+      +------------------+      +------------------+  |
|   |  Kafka           |      |  Iceberg Catalog |      |  Object Storage  |  |
|   |                  |      |  - REST API      |      |  - Amazon S3     |  |
|   |  Any Kafka 2.8+  |      |  - Hive Metastore|      |  - Google GCS    |  |
|   |  Redpanda, etc.  |      |  - Nessie        |      |  - Azure Blob    |  |
|   |  MinIO / Local   |      |                  |      |                  |  |
|   +------------------+      +------------------+      +------------------+  |
|                                                                             |
+-----------------------------------------------------------------------------+
STEP 01

Consume

K2I pulls messages from Kafka in intelligent batches. If the buffer is full, backpressure automatically pauses consumption. No data is acknowledged until it's durably logged.

STEP 02

Buffer

Events are converted to Apache Arrow columnar format and held in an in-memory hot buffer. Hash indexes enable O(1) lookups by key or offset. Data is immediately queryable.

STEP 03

Flush

When the buffer reaches size, time, or count thresholds, K2I encodes records as Parquet and uploads to object storage. The transaction log tracks every step.

STEP 04

Commit

K2I atomically commits the new data files to your Iceberg catalog. Only after successful catalog commit are Kafka offsets acknowledged. Exactly-once, guaranteed.

STEP 05

Maintain

Background tasks automatically compact small files, expire old snapshots, and clean up orphans. Your table stays healthy without manual intervention.

Features

Everything you need for production streaming

Hot Buffer + Cold Storage

Sub-millisecond queries on recent data via in-memory Arrow buffer. Cost-efficient analytics on historical data via Parquet on object storage. Best of both worlds.

Transaction Log

Every operation is logged before execution. Crash at any point and K2I recovers automatically. No manual intervention, no data loss, no duplicates.

Intelligent Flow Control

When the buffer fills up, K2I automatically pauses Kafka consumption. No OOM crashes, no dropped messages. Resume seamlessly when capacity returns.

Your Catalog, Your Choice

Native support for REST, Hive Metastore, AWS Glue, and Nessie catalogs. OAuth2, IAM roles, and custom auth. Works with your existing infrastructure.

Self-Maintaining Tables

Built-in compaction merges small files. Snapshot expiration controls metadata growth. Orphan cleanup removes failed operation debris. All automatic.

Full Visibility

Prometheus metrics for throughput, latency, buffer utilization, and catalog health. Structured JSON logging. Kubernetes-ready health endpoints.

Common Use Cases

Built for real-world data pipelines

Real-Time Product Analytics

Scenario: E-commerce platform tracking user behavior

Before

Spark batch jobs with 15-minute delay. Dashboard data always stale. Marketing can't react to trends.

After

K2I streams clickstream events to Iceberg with sub-second freshness. Real-time dashboards. Instant A/B test insights.

See case study
Kafka clickstream_events
K2I Engine streaming ingest
Apache Iceberg hot + cold storage
TRINO
DASHBOARD
View setup guide

Ready to start?

Get started in minutes with our open source streaming engine. Free forever, no strings attached.

Get Started Read the Docs

100% Open Source

Apache 2.0 licensed. No vendor lock-in, no hidden fees. Your data, your infrastructure, your control.

View on GitHub

Need Enterprise Support?

Running K2I at scale? OSO provides 24/7 support, advanced integrations, and dedicated Kafka expertise.

Contact Sales

Stay connected

Stay in the loop, get product and feature updates.