The End of the Data Warehouse? Convergence, Fragmentation, and the Future of Analytical Systems¶

See also: Parquet vs CSV, ORC, and Avro — the file format layer on which lakehouses depend, and Lakehouse vs Warehouse vs Database — the architectural trade-offs between these paradigms.

Themes: Data Architecture · Economics · Ecosystem

Opening Thesis¶

The data warehouse is not dying. It is mutating. What looks from the outside like warehouse displacement — lakehouses replacing Redshift, DuckDB running analytics locally, open table formats fragmenting the storage layer — is more precisely a restructuring of where analytical capabilities live, who controls them, and what economic model governs their operation. The architectural principles that made warehouses successful (schema enforcement, columnar storage, query optimization, transactional consistency) are not being discarded. They are being rebuilt on different substrates with different cost and control properties. The question for organizations is not whether to use a warehouse, but whether the warehouse abstraction — a single, centrally managed analytical store — is still the correct organizational unit for analytical data.

Historical Context¶

The Enterprise Data Warehouse Era¶

The enterprise data warehouse (EDW) emerged in the 1980s and 1990s as a solution to the analytical query problem: operational databases (OLTP systems) optimized for transactional throughput could not serve complex analytical queries without degrading transaction performance. The EDW separated analytical workloads from operational workloads by creating a dedicated store, populated by Extract-Transform-Load (ETL) processes, where analytical queries could run without competing with transactional writes.

Bill Inmon and Ralph Kimball articulated competing architectural philosophies for EDW design (corporate information factory vs dimensional modeling) that would dominate data architecture thinking for two decades. Both approaches shared the assumption that the warehouse was a single, integrated, carefully governed store — the "single source of truth" for organizational analytics. This assumption was valid when data volumes were measured in gigabytes and when the number of data sources was small enough for a dedicated ETL team to manage integration.

MPP and the Scale Era¶

As data volumes grew through the 2000s, row-store databases (Teradata, Netezza, Oracle) reached their performance limits. Massively parallel processing (MPP) systems distributed query execution across many nodes, enabling analytical queries over terabytes of data within human-patience response times. MPP warehouses became the standard for large enterprises whose data volumes exceeded the capacity of single-node systems.

The MPP model introduced significant cost: hardware acquisition, licensed software, specialized DBAs, and long-running procurement cycles. The economics of MPP warehouses were viable for large enterprises but prohibitive for smaller organizations and for new use cases that required analytical capabilities without the capital investment.

Cloud-Native Warehouses and Serverless Economics¶

Amazon Redshift (2012), Google BigQuery (2010), and Snowflake (2012) fundamentally changed the economics of analytical systems. Columnar storage on cloud object storage (S3, GCS) with separated compute and storage meant that organizations could store unlimited data at object storage prices and pay for query compute only when queries ran. The capital investment of on-premises MPP hardware was replaced by operational expenditure proportional to actual use.

Snowflake's virtual warehouse model — compute clusters that spin up and down in seconds, independently scalable per workload — produced a new operational pattern: multiple independent compute pools serving different stakeholder groups from a shared storage layer. This model eliminated the resource contention that characterized shared MPP hardware and enabled workload isolation without the cost of dedicated hardware per team.

The Lakehouse Narrative¶

The lakehouse narrative, articulated by Databricks in 2020, proposed that open table formats (Delta Lake, Apache Iceberg, Apache Hudi) could bring warehouse-grade guarantees (ACID transactions, schema enforcement, time travel, efficient query planning) to data stored in open object storage formats. If the warehouse's core contributions were its guarantee layer rather than its proprietary storage format, those guarantees could be built on top of Parquet files in S3 without a proprietary storage engine.

  Data warehouse vs lakehouse stack:
  ─────────────────────────────────────────────────────────────────
  Traditional Warehouse                  Lakehouse
  ─────────────────────────              ──────────────────────────
  ┌────────────────────┐                 ┌────────────────────────┐
  │   Query Engine     │                 │   Query Engines (many) │
  │ (proprietary SQL)  │                 │  Spark / Trino / DuckDB│
  ├────────────────────┤                 ├────────────────────────┤
  │  Storage Engine    │                 │   Table Format Layer   │
  │ (proprietary format│                 │ Delta / Iceberg / Hudi │
  │  columnar, indexed)│                 ├────────────────────────┤
  └────────────────────┘                 │   Object Storage       │
                                         │   (Parquet on S3/GCS)  │
                                         └────────────────────────┘

  Warehouse: proprietary stack, managed by vendor
  Lakehouse: open formats, composable compute, governed by you

Delta Lake (Databricks) uses a transaction log (_delta_log) to record all table operations, enabling ACID transactions, time travel queries (reading the table as it was at a prior point), and efficient metadata scanning. Delta files are Parquet files annotated by the transaction log.

Apache Iceberg (Netflix-originated, now Apache) uses a catalog of manifest files to track which Parquet files belong to a table partition, enabling hidden partitioning (queries need not know partition structure), schema evolution without rewriting data, and concurrent write safety.

Apache Hudi (Uber-originated) focuses specifically on incremental data processing — updating and deleting specific records within Parquet files efficiently, enabling CDC (change data capture) patterns without full table rewrites.

These formats share the architectural commitment to separating the guarantee layer (metadata about transactions and table structure) from the storage layer (the actual data, stored as open Parquet files). The separation means different compute engines can read the same table: a Spark cluster can write to an Iceberg table, and Trino and DuckDB can read it. Query engine interoperability is the architectural property that lakehouses provide that proprietary warehouses cannot.

Performance Trade-Offs¶

Lakehouse architectures are not performance-equivalent to purpose-built warehouses. The design choices that enable openness and flexibility introduce performance trade-offs.

Metadata scanning: Iceberg and Delta Lake use catalog metadata to determine which files contain data relevant to a query, a process called metadata filtering. For tables with many small files or complex partitioning, metadata scanning can add latency. Managed warehouses with proprietary metadata structures (Snowflake's micro-partitions, BigQuery's internal clustering) can apply metadata filtering more aggressively because they control both the metadata format and the query engine.

Query optimization differences: Managed warehouse query optimizers have been tuned to the proprietary storage format for years. The optimizer knows the storage layout, can exploit internal statistics that are not exposed through open APIs, and can apply physical plan optimizations that are not possible when the storage format is opaque to the query engine. Open table format query engines are improving rapidly, but there remains a gap in optimizer sophistication for certain query patterns.

Cold storage latency: Object storage (S3, GCS, Azure Blob) has higher first-byte latency than SSD-backed block storage. For queries that must scan many small files — a common pathology of lakehouse tables that have accumulated many small writes — the aggregate overhead of many object storage requests can produce query times meaningfully higher than an equivalent query on a warehouse that stores data in a single large file with internal indexing.

File compaction: lakehouse tables require periodic file compaction (merging small files into larger files) to maintain query performance. This is an ongoing operational process that proprietary warehouses handle automatically internally. Lakehouse operators must schedule and monitor compaction jobs, adding operational overhead.

Governance Implications¶

The openness of lakehouse architectures introduces governance complexity that managed warehouses handle internally.

Schema enforcement: managed warehouses enforce schema on write — inserting data with the wrong column type produces an error. Lakehouse tables can be configured for schema enforcement, but many deployments allow schema evolution without explicit policy, leading to schema drift where the same column name carries different semantics across different batches of data.

Access control: managed warehouses integrate access control (GRANT/REVOKE on tables, columns, rows) into the query engine. Lakehouse access control is split between the object storage layer (S3 bucket policies, IAM roles), the table catalog (Unity Catalog, AWS Glue, Hive Metastore), and the query engine (Spark SQL GRANT, Trino system access controls). Coordinating access control across these layers requires careful design and is a common source of security gaps.

Data contracts: lakehouses require explicit data contracts (schema definitions, column semantics, SLO assertions) because the openness of the format means that any client can write any data. Proprietary warehouses can enforce contracts through the loading layer. Lakehouse contracts are typically enforced through tooling (Great Expectations, dbt tests, Soda) that operates outside the storage layer — contracts are asserted at ingestion or transformation time, not enforced by the storage format itself.

Economic Realities¶

The storage cost comparison favors lakehouses significantly: object storage is an order of magnitude cheaper than managed warehouse storage. But the managed warehouse storage premium includes query performance optimizations (internal columnar indexing, statistics, micro-partition pruning) that are not free in a lakehouse. Comparing raw storage prices is misleading — the relevant comparison is total cost per query-hour of analytical work delivered, which requires accounting for engineer time spent on lakehouse operations.

Cloud egress costs are a significant but frequently underestimated component of both warehouse and lakehouse economics. Querying a Snowflake or BigQuery warehouse from a client outside the cloud region, or moving data between a lakehouse and downstream consumers in another region, incurs egress fees. These fees can dominate operational costs for analytics workloads with large query result sizes or cross-region data pipelines.

Convergence vs Fragmentation¶

The analytical data landscape is moving simultaneously toward convergence and fragmentation, in different dimensions.

Convergence on open formats: Parquet and open table formats are becoming the substrate that multiple analytical tools interoperate through. BigQuery Omni can query Parquet files in GCS. Redshift Spectrum queries Parquet in S3. Snowflake External Tables read Iceberg and Delta. The open format layer is becoming the neutral interchange plane on which proprietary systems interoperate — a genuine convergence.

Fragmentation of compute: the single warehouse as the universal compute layer for analytical workloads is fragmenting. DuckDB has demonstrated that analytical query workloads that were previously sent to centralized warehouse clusters can often be executed locally on laptop or single-node server hardware with competitive performance and zero infrastructure cost. Polars provides vectorized dataframe computation in Python without requiring a cluster. The "every query to the warehouse" assumption is being replaced by a tiered model: local computation for small/medium analytical work, cluster computation for large-scale processing, warehouse for governed query serving.

Edge analytics is an emerging dimension of fragmentation: analytical computation running at the network edge (IoT gateways, CDN nodes, embedded devices) rather than in centralized cloud infrastructure. The scale of data generated by embedded sensor networks makes centralized ingestion and processing economically untenable for many use cases, pushing computation toward the data source.

Decision Framework¶

The question the warehouse was answering: where is the governed, consistent, performant store for organizational analytical data? The lakehouse answers the same question with different cost and control properties. The question itself remains valid. The organizations that declare "the warehouse is dead" and adopt a lakehouse without investing in governance, schema contracts, and operational discipline reproduce the warehouse's problems with different tooling.