AI-Ready

Accept Fragmentation. Invest in Meaning.

Thiru Arunachalam, Founder & CEO, WALT

August 19, 2026

Iceberg just killed the three-year migration. Iceberg started out as a Netflix side project from Ryan Blue and Daniel Weeks, who built it to fix the correctness and performance problems of Hive tables at Netflix scale. Netflix donated it to the Apache Software Foundation in 2018, and still runs core data platform workloads on it today.

From being a top-level Apache project, Iceberg quickly became a threat to the big warehouse players, and then remarkably, an asset embraced by them.

In June 2024, Databricks acquired Tabular, the company founded by Iceberg's creators, in a deal reported at over $1 billion. That same month, Snowflake made Iceberg tables generally available and announced Polaris Catalog, an open catalog built on Iceberg's REST API.

Both Databricks and Snowflake supporting Iceberg as a first-class table format is the industry's most decisive signal that vendor lock-in on storage is ending.

Fragmentation is no longer a problem to be solved by migration, and instead, it’s the shape of the future data estate. Before we explore this future, let’s quickly recap why fragmentation exists in the first place.

Fragmentation is a story of prudence

Every time a new data platform arrived (Hadoop, Redshift, Snowflake, Databricks, real-time streaming stacks), leaders made careful, defensible choices to move only the most impactful workloads to the new place.

This approach seemed impactful at the time. However, when multiplied across three or four platform waves, you end up with the fragmented, messy data estate every CDO now inherits.

Why the instinct to centralize won’t work

For three decades, the CDO's default move on inheriting a data estate has been to pick a platform, run a multi-year migration, and centralize. This approach takes 3-4 years, costs millions. By the time the project ends, the metric definitions you migrated have drifted, the sponsoring executive has moved on, and half the source systems have been replaced.

What changes with the arrival and adoption of Iceberg

With Iceberg and the broader open-format wave (Delta Lake, Apache Hudi), the physical location of your data stops being the strategic question. Any compute engine can read the same tables. Spark, Trino, Snowflake, and Databricks can all query one copy of the data, in place.

The migration was always a means to an end: one engine, one copy, one bill. Open formats deliver the "one copy" without the migration. The lock-in that justified the three-year program has evaporated, which means the program itself no longer clearly pays off.

Where does the platform business model go next?

Platforms have historically monetized compute, with storage as the sticky part. If storage goes vendor-neutral, the way to keep the compute business is to own the layer of meaning above it: register the Iceberg tables in your catalog, host the query engine, provide the operating experience.

That explains the Tabular acquisition by Databricks, the rise of Polaris and the reason why every major platform is pouring money into catalogs. Metadata and context is the new lock-in surface.

What the traditional data catalogs got wrong

The Collibra and Alation generation was built for governance, and it shows. These tools sit off to the side of the analytical critical path: an inventory of tables, definitions, and owners that starts going stale the day it is created. Schema drifts. Business meaning evolves quarter to quarter. Human curation cannot keep the ‘expensive shelfware’ updated.

In an agent-driven world, this model breaks entirely. An agent doesn’t talk to a data steward to find out whether the documentation is current. It needs context that is machine-readable and continuously verified at the speed of business.

The new center of gravity: Meaning, context, inference

If storage is a commodity and compute is heading the same way, what remains is meaning.

Meaning is how you tackle questions like “what does "revenue" mean at this company?” or “what defines an active customer this quarter?”

These questions cannot be answered by any of the storage or compute layers. They live above both layers, in the data context graph. This layer is where tribal knowledge goes to survive. Today it lives in your senior analyst's head, and it walks out the door when they do.

Does the "best query engine" even matter, in the long run?

If meaning now sits above storage and compute, one question remains for the compute layer: how much does raw engine speed still buy you? Let’s take stock of analytics today.

Analytics is mostly not real-time. In three years, the primary consumer of analytics will be agents, not humans.

Agents don’t care whether a query returns in 900 milliseconds or 90 milliseconds. They care whether the answer is reliable, reproducible, and traceable. Ask the same question twice and an agent needs the same answer both times, which LLM-generated SQL cannot promise because transformer architectures are stochastic by design.

So the "best query engine" narrative starts to lose force, and the "reliable inference engine + stable context graph" narrative gets stronger.

The new CDO playbook: Stop centralizing the data, start centralizing the meaning of that data

Stop asking which platform to standardize on. Start asking four different questions:

1. Leave the data in place: Which workloads can stay exactly where they are now that open formats give any engine federated access to them? You still shape data for consumption, the physics has not changed, but the shaped data can stay wherever it already sits.

2. Invest in meaning: How fast can I stand up an ontology, semantic layer, and canonical metric definitions that outlive any platform decision?

3. Stay vendor-neutral: How do I keep that meaning layer independent, so a platform catalog does not re-lock me in through the back door?

4. Pull AI-readiness forward: How do I get agent-ready in months, instead of the three years the migration would have taken?

How does autonomous data engineering help

Once your meaning layer is stable and your agents can operate on any topology, migrations become opportunistic rather than strategic. You no longer plan around data gravity. Meaning gravity is the place to start, and the data follows workload by workload.

A workload that would benefit from lower latency? Move it.

A dataset that would benefit from a cheaper storage class? Move it.

The autonomous data engineer handles the mechanics. For instance, WALT’s crew of specialized autonomous data engineers ingest, transform, reason, and govern across whatever topology you already run. WALT's Operator agent, in particular, decides when to trickle migrate a workload, weighing consumption, cost, and latency.

WALT builds the data context graph autonomously and keeps it in sync as schemas drift, in production in about 8 weeks. No lift and shift required.

Your engineers stay, and their judgment stays. The grunt work of ontology construction, lineage mapping, and continuous sync moves to agents, and everything the agents build belongs to you.

The result: The three-year migration project dies. AI-readiness gets pulled forward by years.

Bottom line

Iceberg didn't kill the warehouse. It killed the reason for the three-year migration. The three things replacing it, namely a meaning layer, an inference engine, and an agentic runtime, are things you can start building now, on whatever you already run.

The CDO's job shifts: stop centralizing the data; start centralizing the meaning of the data. Invest in a vendor-neutral data context graph. Let agents opportunistically migrate workloads later, only when the business justifies it.

See how WALT works, or book a demo and bring your messiest use cases.