The Data-AI Control Plane: Iceberg REST Catalog's Journey to Intelligence

How the Iceberg REST Catalog evolved from a simple metadata store into an intelligent control plane that governs both data and AI assets across the modern Lakehouse.

Originally published at https://www.linkedin.com/pulse/data-ai-control-plane-iceberg-rest-catalogs-journey-intelligence-t07fe/.

TL;DR:

What if your data catalog could think for itself? The Iceberg REST Catalog has evolved from a simple metadata store into an intelligent control plane. It will autonomously manage tables, optimize queries, and enforce dynamic security policies across both data and AI assets like embeddings and models. This “Catalog 2.0” bridges data and AI to power future-ready, governed Lakehouses.

History — Why a Catalog?

Before Iceberg, the data lake was largely a collection of Parquet or ORC files in object storage. These formats were great for efficient storage—but they lacked structure, versioning, and governance. There was no consistent way to manage schema evolution, partitioning, or ensure transactional consistency—especially across tools.

Apache Iceberg solved that by introducing a metadata layer on top of the data, bringing ACID guarantees, time travel, versioned snapshots, and schema evolution to the lakehouse. But even with Iceberg, there was still a missing piece: a reliable, open way to store, discover, and govern those metadata objects.

In the early days, teams had to rely on the Hadoop Catalog or the Hive Metastore (HMS). But both came with serious drawbacks. They were tightly coupled to specific compute engines, hard to scale across environments, and came with a heavy operational burden. Hive Metastore in particular had an ever-growing list of critical CVEs, limited authentication options, and a vanishing open-source community around it. The writing was on the wall: this model wouldn’t scale into the cloud-native, multi-tenant, multi-engine future.

Enter the REST Catalog.

Its purpose was clear: provide a stateless, vendor-neutral HTTP API to manage Iceberg’s table metadata—without requiring a tightly-coupled runtime, a proprietary JAR, or heavyweight infrastructure. It decoupled the metadata layer from compute and storage, giving teams the freedom to run Iceberg anywhere: across tools, clouds, and organizational boundaries.

And because it speaks REST, any system—SQL engines, schedulers, AI services—can interact with it easily and safely. REST is universal. The REST Catalog was the right abstraction for a new generation of composable, open, and governed data architectures.

Catalog 1.0 — First Attempts and Extensions

The earliest implementations of the Iceberg REST Catalog were intentionally minimal. They demonstrated a simple but powerful idea: metadata management through a stateless, RESTful API, decoupled from compute engines and cloud vendors. Teams could use it to create, update, and delete Iceberg tables with no need for tightly coupled services like Hive Metastore or embedded catalog clients.

It was fast to deploy, portable across environments, and engine-agnostic by design.

Security was kept lightweight but flexible. The catalog adopted vended credentials—short-lived access tokens injected into each request by an external identity provider—and remote signing, a mechanism that allowed trusted systems to cryptographically verify the source and integrity of metadata operations. Both of these patterns were borrowed from cloud object storage, where they were already used to avoid exposing long-lived access keys.

But the key shift was this: the catalog itself began taking responsibility for access control.

Rather than simply accepting metadata operations, it became an active participant in authorization—deciding who could interact with what, when, and how. That led organizations to extend their REST Catalog deployments with additional layers: lightweight RBAC, tenant-aware routing, and support for multiple catalogs across projects or domains.

What began as a simple interface for table lifecycles was quickly evolving into a gateway for metadata governance and access mediation. REST Catalog 1.0 had validated the concept—and opened the door for what came next.

From Tool to Infrastructure: The REST Catalog Matures

As Apache Iceberg matured into the backbone of modern Lakehouse architectures, the REST Catalog had to evolve with it. What began as a simple tool for managing table metadata was no longer enough. The ecosystem was growing. So were the expectations.

Teams needed more than create, update, and delete operations. They needed secure, multi-tenant environments with table- and project-level RBAC. They needed robust orchestration of concurrent transactions—handling conflicts, retries, and commits across distributed systems. They needed metadata visibility that transcended tools, enabling lineage, observability, and governance. And they needed stable, predictable APIs to build production pipelines, data products, and platform features on top of the catalog.

This was the turning point.

The REST Catalog was no longer just a developer utility—it became a core infrastructure component. A persistent, governed control layer that served as the metadata and transaction orchestration backbone of the Lakehouse. It now sat at the center of the data platform, managing the lifecycle and access of data assets across users, domains, pipelines, and services.

It acted like a brain: ensuring that metadata, access, and transactional state remained consistent and trustworthy—even as compute and storage became more dynamic and distributed.

This marked the beginning of Catalog 2.0—a shift from simple metadata tooling to enterprise-grade control infrastructure for modern data platforms.

Control Plane for Data & AI

Autonomous Data Management: The Foundation

At the same time, the catalog is becoming far more active in runtime orchestration. It no longer just stores metadata—it autonomously manages tables, much like how a mature database such as Postgres takes care of its tables behind the scenes without users needing to think about it.

These autonomous table management services handle compaction, vacuuming, and retention policies automatically, ensuring tables remain optimized, performant, and cost-effective without manual intervention. This shift means teams can rely on the catalog to maintain healthy tables, reducing operational overhead and improving reliability.

Additionally, the catalog offers a powerful Scan API that empowers query engines with the best possible execution plans. By intelligently leveraging partition pruning, snapshot versions, and column filters, the Scan API reduces query latency and resource usage, enabling faster, more efficient analytics.

AI Governance and Contextual Control

Beyond data management, the modern stack demands governance and seamless integration with AI workloads. The REST Catalog is evolving into a programmable control plane that manages context and enforces policy for AI applications alongside traditional data.

AI systems—such as retrieval-augmented generation (RAG) pipelines, LLM agents, and model serving infrastructure—rely on access to governed, high-quality data. Using the same authentication and access control mechanisms trusted by Iceberg engines, the catalog acts as a secure gateway for AI workloads.

Importantly, the same access mechanisms—vended credentials, remote signing, and token delegation—used to grant secure access to Parquet files can be extended to AI-related data assets, including PDFs, text documents, and images. This unified access layer prevents silos and simplifies governance across data and AI.

The catalog also supports contextual governance: the system dynamically enforces different access policies depending on who or what is querying data. For example, a product manager asking “What’s our projected revenue for next fiscal year?” may receive a different answer than a compliance analyst, with sensitive data redacted or filtered in real time. Catalog-backed roles and policies ensure AI respects organizational boundaries at inference time.

Finally, the catalog governs AI-native metadata such as embeddings, pipeline artifacts, model outputs, and training lineage. These assets become first-class citizens in the data ecosystem, with lifecycle management, auditability, and policy enforcement on par with traditional tables.

Join the Journey: Building the Future of Data and AI Governance

The convergence of data and AI demands a new kind of control plane—and the Iceberg REST Catalog is evolving to become exactly that. From a simple metadata store, it has matured into a unified, intelligent control plane for governing both data and AI assets—essential for building scalable, secure, and intelligent data ecosystems of tomorrow.

Don’t get left behind in this critical evolution. Experience the future of data and AI governance with Lakekeeper, our open-source Iceberg REST Catalog control plane. Get started today by exploring our documentation, starring us on GitHub, and joining the community that’s shaping the next generation of Lakehouse innovation.