The Database is Dead: The Plot Twist in Data's New Era

Monolithic OLAP databases broke under modern scale. Apache Iceberg made data the open foundation, turning databases into specialized query engines in a new open ecosystem.

Originally published at https://www.linkedin.com/pulse/database-dead-plot-twist-datas-new-era-vakamo-com-aq43e/.

For decades, the database was the ultimate data hub. It handled everything: transactions (OLTP) and analytics (OLAP), all in one monolithic box. You stored your data, and you queried your data. The database was the undisputed center of gravity.

But the world changed.

A terabyte used to be “big data.” Now it’s a rounding error in a petabyte era. Instead of a few hundred users, thousands of people—analysts, engineers, and even AI agents—are hammering the same data. This explosion of scale and concurrency broke the traditional OLAP database model. You simply can’t box all that into a single system anymore.

The database wasn’t “bad.” It was just designed for a smaller world.

The New Paradigm: Data Becomes the Foundation

When the old model broke, something new had to emerge. That “something” was Apache Iceberg, and it flipped the script.

Data no longer had to be trapped in a proprietary database. Instead, it lives openly in cheap, elastic, and practically infinite object storage. Iceberg added the missing pieces: schema, ACID transactions, time travel, and evolution.

Suddenly, the database wasn’t the center of gravity. The data itself became the foundation, and multiple engines could connect to the same Iceberg tables. For the first time, OLAP wasn’t bound to one monolithic system; it became an open ecosystem.

New Formats Expand the Story

Iceberg opened the door, but the story doesn’t end there. The next generation of columnar formats is pushing the boundaries of what’s possible with open storage.

• Vortex is a high-performance columnar format designed for object storage, offering incredible speed through custom encodings and optimized layouts. It’s built for efficient processing of complex queries.
• Lance is built specifically for AI and machine learning. It supports multimodal data—vectors, images, video, and text—with built-in versioning and highly efficient vector search for similarity queries.

These formats prove a bigger trend: OLAP and AI workloads no longer rely on a single database. They thrive on open formats and specialized engines, each optimized for a specific workload.

Databases: From Silos to Query Engines

The story of databases isn’t over—it just got a plot twist.

When Iceberg opened up OLAP, databases had to adapt. Data wasn’t trapped anymore; it was open, structured, and queryable by anyone. Snowflake, BigQuery, Redshift, and others raced to read Iceberg tables, competing not on owning data, but on speed and efficiency.

There’s an even bigger shift happening. The same databases that once controlled every byte of OLAP data are now learning to write back. Old kings like DB2 and Postgres (data kings) can now take OLTP tables and publish them directly to Iceberg.

This “shift left” in data architecture pushes analytics closer to source systems. Complex ETL pipelines are no longer mandatory. Operational databases can now publish analytics-ready formats directly, reducing latency and enabling real-time insights with leaner, simpler architectures.

But managing all this open data at scale is still a challenge. That’s where a Control Plane like Lakekeeper comes in. It provides a centralized layer for governance, access control, and operational insights, making it easy for multiple engines and teams to collaborate on the same Iceberg tables without stepping on each other’s toes. With Lakekeeper, organizations can enforce data contracts, monitor usage, and automate maintenance, turning the open data ecosystem into a manageable, enterprise-ready platform.

What was once a threat—open, object-store-based analytics—has become the next frontier. Databases are no longer just silos; they are engines that write, read, and orchestrate this new open data ecosystem.

The Key Takeaways

• Monolithic OLAP is dead. Traditional databases can’t handle today’s scale, thousands of users, and AI workloads in a single box.
• Data is open and foundational. Apache Iceberg moves analytics from proprietary silos into object storage while preserving essential features like ACID and schema.
• Databases are becoming specialized query engines. Their value now lies in performance, indexing, caching, and federation, not in owning the data storage.
• Next-gen formats extend the ecosystem. Vortex and Lance are building on Iceberg’s foundation to support high-performance and multimodal AI/ML workloads.
• “Shift left” enables leaner architectures. Operational databases can now publish directly to analytics formats, reducing complexity and enabling real-time insights.

A new decade of data is here. The era of open data has begun, and the databases themselves are learning to play in this new world.