Govern AI agents on a multimodal medallion

Build a raw → bronze → silver → gold medallion that mixes Iceberg tables, S3 objects, and Lance vectors, then let two AI agents hit the same tool and get different answers because the catalog decides who gets credentials.

Runnable source: https://github.com/lakekeeper/lakekeeper/tree/main/examples/agentic-medallion.

Agentic applications break the usual governance story. A RAG agent doesn’t read tables — it reads images, captions, and vector embeddings, and it decides at runtime what to fetch. If access control lives in the agent’s code, then every prompt injection and every new tool is a new hole. This tutorial puts the boundary somewhere an agent can’t argue with: the catalog either vends storage credentials or it doesn’t.

You’ll build a four-layer medallion that spans Iceberg tables, plain S3 objects, and a Lance vector dataset — all under one Lakekeeper catalog and one authorization model — then watch two agents call the identical tool and get different results.

See it running: the agentic-medallion example runs Lakekeeper, Keycloak, SeaweedFS, Ollama, and a JupyterLab workbench together, and drives the whole flow across three notebooks.

The medallion, one layer at a time

Each layer is a namespace in a single warehouse. What’s interesting is that the format changes per layer while the governance model doesn’t:

Bronze and silver are ordinary Iceberg tables. Raw and gold are generic tables — Lakekeeper’s way of governing datasets that aren’t Iceberg (a bucket prefix of JPEGs, a Lance vector dataset) with the same grants and the same credential vending. That’s what makes a multimodal medallion governable end to end instead of governed-for-tables-and-hoped-for-the-rest.

Prerequisites

This one is heavier than the other tutorials: budget ~12 GB of disk and ~8 GB of RAM for the Docker/Podman VM. The first run builds a workbench image and downloads model weights; later runs are cached.

1. Start the stack

cd lakekeeper/examples/agentic-medallion
./up.sh

Use ./up.sh, not docker compose up. The script detects your host’s LAN IP and sets S3_ENDPOINT to it, so the same signed URL resolves from both the in-network notebook kernel and your host browser; it also sets KEYCLOAK_BROWSER_URL so the device-code login works from the browser.

Then pull the two local models in a second terminal:

docker compose exec ollama ollama pull moondream    # vision → captions
docker compose exec ollama ollama pull gemma2:2b    # chat → RAG answers

Endpoints:

• JupyterLab — http://localhost:8888/lab/tree/notebooks
• Lakekeeper console — http://localhost:8181

Everything runs inside the workbench kernel: the S3 data plane is only reachable on the Docker network, so external compute won’t work here.

2. Two kinds of identity

The setup notebook (00-setup.ipynb) provisions both patterns Lakekeeper supports, and the distinction matters once agents are involved:

• A human, interactively — peter, the platform admin, logs in with the OAuth2 device grant. The notebook prints a URL and a code, you approve in the browser, and no secret is ever pasted into a cell:

  peter = mlib.device_login()
  

• Agents, autonomously — PIPELINE (the ETL job), analyst-agent, and contractor-agent are service accounts using the client-credentials grant. There’s no human in the loop to approve anything, which is exactly why their permissions have to be pinned down in advance.

Then the grants, which are the whole experiment — the analyst may read only gold, the contractor may read everything except gold:

The contractor is not a locked-out user — it has broad access to the earlier layers. Only the gold embeddings are walled off. That asymmetry is what proves the wall is at a specific object, not at the door.

3. Build the layers

01-build-medallion.ipynb runs the pipeline as PIPELINE, and every write goes through credentials the catalog vends for that layer:

1. Raw — downloads ~40 CC0 Met Museum thumbnails and uploads them as plain S3 objects.
2. Bronze — writes metadata to Iceberg, with image_uri linking each row to its raw object.
3. Silver — the local vision model (moondream) reads each image through its URI and writes a caption.
4. Gold — CLIP embeds each image into a 512-d vector; the vectors land in a Lance dataset, again via vended credentials.

Note what silver does: it resolves an s3:// URI from an Iceberg row and reads the bytes behind it. The link between structured metadata and unstructured objects stays inside the governed perimeter, rather than becoming a side channel around it.

4. Same tool, two answers

02-agent-governed.ipynb gives both agents an identical search_gallery tool — embed the query, search gold, answer from what came back. Then it runs both:

• analyst-agent — Lakekeeper vends gold credentials, the vector search returns matches, and the chat model answers from them. Normal RAG.
• contractor-agent — Lakekeeper returns 404, no such table. No credentials, no bytes, no answer. The agent isn’t blocked by a guardrail in its prompt; the data is simply not reachable.

Two details are worth pausing on:

• It’s a 404, not a 403. A 403 would confirm that gold.image_embeddings exists — useful information for anyone probing. Lakekeeper doesn’t leak the existence of resources you can’t see.
• The contractor can still read raw and silver in the same notebook. The refusal is scoped to gold, which rules out “the token was just broken.”

Flip the grant — in the console or in code — and rerun the cell. The contractor’s next call succeeds. No agent restart, no redeploy, no code change on the agent’s side.

Why enforce at the credential layer

Because it’s the one place an agent can’t reason its way past. Prompt-level rules (“don’t query the embeddings”) are instructions; a denied credential is an absence of capability. The check happens when load(vended=True) asks the catalog for STS credentials, which means:

• The blast radius of a compromised agent is its grants, not its imagination.
• Every tool the agent gains inherits the same boundary — you don’t re-implement access control per tool, per framework, or per model.
• Revocation is immediate and global, the same as it is for Spark or Trino. An agent is just another principal.

Be precise about the scope, though: what’s governed is credentialed access to the stored gold dataset. A contractor with raw image access could re-embed those images itself. The catalog controls access to your assets — it doesn’t stop someone from recreating derived data from inputs they’re allowed to read. That’s a modeling decision for your grants, not a gap in enforcement.

Housekeeping

If your machine’s IP changes (Wi-Fi switch, sleep/wake), re-point the warehouse without tearing anything down:

./refresh-ip.sh    # uses PIPELINE's modify right — no login needed

And when you’re done:

./down.sh           # keep the Ollama models
./down.sh --purge   # remove everything, including ~6 GB of models

The demo credentials in this example are throwaway values. Never carry them into anything real.

Where to go next

• Access control with Lakekeeper — the grant model this example leans on, without the AI layer on top.
• Centralized access & vended credentials — how short-lived, scoped credentials work, and why the agent never holds a key.
• Swap the chat model for something stronger with CHAT_MODEL=qwen2.5:7b (needs ~4.7 GB) and compare the RAG answers. </content> </invoke>