Stream Kafka to Iceberg with Kafka Connect

Pipe events from Apache Kafka into Iceberg tables using the Kafka Connect Iceberg Sink, with Lakekeeper as the governed REST catalog.

Runnable source: https://github.com/lakekeeper/lakekeeper/tree/main/examples/kafka-connect-iceberg-sink.

If your events already flow through Kafka, the shortest path into the lakehouse is the Kafka Connect Iceberg Sink — a connector that consumes topics and writes Iceberg tables. Pointed at Lakekeeper, those tables land in a governed catalog instead of a loose pile of files, so the data is immediately discoverable, permissioned, and served with vended credentials.

See it running: the kafka-connect-iceberg-sink example provisions the whole pipeline on a local Kubernetes cluster.

What this example runs

Unlike the other examples, this one deploys onto Kind (Kubernetes in Docker) via a Makefile rather than a single compose file:

cd lakekeeper/examples/kafka-connect-iceberg-sink
make all

make all provisions the full stack:

  • Apache Kafka (via the Strimzi operator) and a Schema Registry
  • Kafka Connect running the Iceberg Sink Connector (v1.9.2)
  • Lakekeeper as the Iceberg REST catalog, with Postgres for metadata and MinIO for storage
  • A producer application generating events, plus Trino and a Kafka UI

Because it runs on Kubernetes, use kubectl port-forward to reach the UIs (for example, Trino and the Kafka UI). Check the example’s Makefile and manifests for the exact port-forward targets, since they’re defined there rather than fixed ports on your host.

How the sink points at Lakekeeper

The Iceberg Sink Connector speaks the Iceberg REST protocol, so connecting it to Lakekeeper is a matter of catalog configuration on the connector — the REST catalog URI, the warehouse, and the authentication token. Conceptually the connector config includes something like:

iceberg.catalog.type=rest
iceberg.catalog.uri=http://lakekeeper:8181/catalog
iceberg.catalog.warehouse=demo

The exact connector manifest (topics, table routing, credentials) lives in the example’s Kubernetes resources — read those for the authoritative, runnable configuration rather than copying the snippet above verbatim.

Why route ingestion through the catalog

The connector could write Iceberg files directly to a bucket. Writing through Lakekeeper instead is what turns raw ingestion into governed data:

  • Discoverable on arrival — the table shows up in the catalog the moment the sink creates it, alongside every other table.
  • Same access model — centralized permissions and vended credentials apply to Kafka-sourced tables exactly as they do to everything else.
  • One source of truth — Trino, Spark, and StarRocks read the freshly-landed events through the same catalog, with no separate registration step.

Where to go next