Centralized access & vended credentials for every engine

Why routing every query engine through one catalog is the security win — and how Lakekeeper vends short-lived, scoped credentials so no engine ever holds a long-lived cloud key.

Runnable source: https://github.com/lakekeeper/lakekeeper/tree/main/examples/minimal.

Most data platforms have the same quiet security problem: every engine holds its own copy of the cloud keys. Spark has an S3 access key in its config. Trino has another. A notebook has a third. StarRocks, your ingestion jobs, that one analyst’s laptop — each one carries long-lived credentials to the bucket. Nobody can say with confidence who can read what, and rotating a key means chasing it across a dozen systems.

Lakekeeper removes that problem by making the catalog the single point of access. This tutorial explains how — and why it matters more than it first appears.

See it running: the minimal example starts SeaweedFS with IAM/STS enabled and points Spark, Trino, and StarRocks at one Lakekeeper catalog — all three get vended credentials, none hold a storage key.

The problem with key sprawl

When each engine authenticates to storage directly, you inherit three liabilities:

  • No central authority. Storage IAM policies are coarse (“this role can read this bucket”). They don’t understand tables, namespaces, or who is asking. Access decisions are scattered across cloud IAM, engine configs, and tribal knowledge.
  • Long-lived secrets everywhere. Every key is a standing risk. Leak one and an attacker has durable access; rotate one and you break jobs you didn’t know depended on it.
  • No usable audit trail. “Who read the payroll table last quarter?” is nearly unanswerable when ten engines each talk to the bucket with the same shared role.

How Lakekeeper centralizes it

Lakekeeper sits between the engines and the storage. Every engine speaks the Iceberg REST protocol to one catalog, and the catalog is the only thing that holds the real storage credential.

Spark ─┐
Trino ─┤                          ┌── vends short-lived, scoped creds
Note.  ─┼──►  Lakekeeper catalog ─┤     (or signs each request)
StarR. ─┤     (holds the key)     └──►  S3 / GCS / ADLS / SeaweedFS
Flink ─┘

Because access flows through the catalog, three things become true at once: authorization is decided in one place, engines stop holding keys, and every access is attributable to an identity. That’s the payoff — the credential mechanics below are how it’s delivered.

Vended credentials: zero-trust access, one request at a time

When an engine wants to read or write a table, it doesn’t reach for a stored key. It asks the catalog, and Lakekeeper hands back access in one of two ways:

  • Vended credentials — Lakekeeper calls the cloud’s STS (Security Token Service) and returns temporary, time-bound credentials scoped to just the location that table lives in. They expire automatically. This is the zero-trust model: the engine gets exactly the access it needs, only for as long as it needs it, and never sees a long-lived secret.
  • Remote signing — for S3, the engine prepares a request and sends the headers to Lakekeeper’s signing endpoint (/<warehouse-id>/v1/aws/s3/sign). Lakekeeper checks permission, signs with its own credential, and returns the signing headers. The engine then talks to S3 directly. The key never leaves Lakekeeper at all.

The engine chooses the mode per request with the X-Iceberg-Access-Delegation header:

X-Iceberg-Access-Delegation: vended-credentials   # get short-lived STS creds
X-Iceberg-Access-Delegation: remote-signing        # have Lakekeeper sign requests
X-Iceberg-Access-Delegation: client-managed        # use my own creds; vend nothing

In a Spark REST catalog config, requesting vended credentials is a single line:

spark.conf.set(
  "spark.sql.catalog.lakekeeper.header.X-Iceberg-Access-Delegation",
  "vended-credentials",
)

Any Iceberg-compatible engine — Spark, Trino, StarRocks, PyIceberg, Flink, DuckDB — sends the same header and gets the same treatment. That’s how you connect five engines and hold zero keys in five configs.

Storage profile vs. storage credential

Two concepts do the work, and it’s worth keeping them straight:

  • Storage credential — how Lakekeeper itself authenticates to the cloud (an access key, a GCP service-account key, an Azure managed identity). This is the secret you centralize; it lives in the catalog and nowhere else.
  • Storage profile — how a warehouse’s storage is organized and delegated: bucket, region, key-prefix, and which delegation methods are allowed. You can even disable credential vending entirely at this level, so a warehouse issues no temporary credentials at all.

Supported backends: S3 and S3-compatible (MinIO, SeaweedFS, Cloudflare R2, NetApp StorageGRID), Azure Data Lake Storage Gen 2, OneLake (Microsoft Fabric), and Google Cloud Storage.

One hard rule: never share a storage location between warehouses. Vended credentials are scoped to a warehouse’s location, so overlapping locations can leak access across warehouse boundaries.

Why this matters

Centralizing access through the catalog turns three chronic problems into properties of the system:

  • Least privilege by construction. Engines receive narrowly-scoped, expiring credentials per request — not blanket bucket access.
  • Instant, complete revocation. Change a permission in the catalog and the next request is denied. There’s no key to hunt down across engines, because the engines never had one.
  • Real auditability. Every access is tied to an authenticated identity and a catalog authorization decision, so “who touched this table” has an answer.

Vended credentials are the mechanism; the point is that governance stops being scattered across cloud IAM and engine configs and becomes one thing you can reason about, enforce, and audit.

Where to go next