Apache Iceberg: From File Format to Business-Oriented API
Apache Iceberg's new File Format API, shipping in 1.11.0, lets new storage formats plug in cleanly across engines, deepening Iceberg's promise of vendor-independent data.
Originally published at https://www.linkedin.com/pulse/apache-iceberg-file-format-api-accessible-business-oriented-tt0te/.
🧊 Apache Iceberg just quietly solved one of the most frustrating problems in data infrastructure — and most people haven’t noticed yet.
Let me explain what happened, why it matters, and what it means for your data stack.
A bit of background
Apache Iceberg is the open-source standard that sits underneath modern data lakehouses — it’s the layer that lets tools like Spark, Flink, Snowflake, and dozens of others read and write the same data reliably, without stepping on each other.
One thing Iceberg has always needed to handle is file formats — the way data is physically stored on disk. Think of file formats like different container types for shipping goods. Parquet, Avro, and ORC are the classics. Each is optimized for different things.
The old problem, in plain terms
Until now, every time Iceberg wanted to support a new file format, engineers had to essentially rebuild the same plumbing in multiple places — once for Spark, once for Flink, and again for the core library. It was repetitive, error-prone, and meant that some features only worked with some formats, on some engines.
For teams building on Iceberg, this created a hidden tax: capabilities weren’t always consistent, and adopting anything new was slow and risky.
What just changed
The Iceberg community has merged the File Format API — shipping in the upcoming 1.11.0 release.
In business terms, think of it like this: instead of every department doing their own custom filing system, the company adopted one standard framework that everyone plugs into. New formats can now be added cleanly, without touching anything else.
What does that unlock?
→ Faster innovation. New file formats optimized for AI workloads — like Vortex (built for GPU-accelerated analytics) and Lance (designed for ML datasets and ultra-fast random access) — will be able to integrate with Iceberg cleanly, without months of custom engineering per engine. Vortex integration is already underway as an early proof of this.
→ More consistent behavior. Features like filtering and data deletion will work the same way regardless of the format or engine underneath. Less “it works in Spark but not Flink” conversations.
→ Smarter storage. The foundation is now in place for a capability called Column Families — which will allow partial updates to data without rewriting entire files. For teams with high-frequency updates, this is a significant performance and cost improvement.
The business bottom line
The data tooling landscape is moving fast. New formats built for AI and real-time analytics are emerging regularly. With this change, Iceberg doesn’t just support today’s formats well — it can adapt to tomorrow’s without requiring your team to migrate off the platform.
I wrote earlier this year about how Apache Iceberg represents the end of the era of painful data migrations — the decade-long cycles of moving from DB2 to Oracle to Teradata to Snowflake, each one a high-cost, high-risk, multi-year project. Iceberg’s open format and decoupled architecture means your data is no longer hostage to any single vendor or technology wave.
The File Format API deepens that promise. It’s not just that Iceberg works with today’s tools — it’s that the architecture is now open enough to absorb whatever comes next. New storage formats, new compute paradigms, new AI-native workloads. Your data stays put. The tools evolve around it.
That’s the compounding value of betting on open infrastructure.