Lakehouse vs Warehouse vs Database: Convergence, Marketing, and Reality¶

Themes: Data Architecture · Economics · Governance

The Problem with the Terminology¶

The terms "data lake," "data warehouse," and "data lakehouse" are not neutral technical descriptors. They are products of vendor marketing cycles, and the precision with which they are used in practice varies inversely with the ambition of the marketing claims surrounding them. A meaningful analysis requires untangling the historical development of each concept from the vendor positioning that has accumulated around it.

The underlying question — where should analytical data live, in what format, under what governance model, with what query semantics — is real, important, and genuinely difficult. The terminology is not the best entry point into it.

Historical Arc: OLTP to OLAP to Lake¶

The separation of transactional and analytical workloads is not a recent innovation. It is a recognition that the access patterns of a system recording business events (OLTP: many small, fast, random-access writes and reads) are fundamentally at odds with the access patterns of a system analyzing those events (OLAP: sequential scans over large datasets, aggregation, few writes).

The data warehouse as an architectural concept emerged in the late 1980s with the work of Ralph Kimball (dimensional modeling, star schemas) and Bill Inmon (corporate information factory, normalized warehouse). Both models shared the same premise: extract operational data from source systems, transform it, and load it into a separate analytical store optimized for query.

  Classic Data Warehouse Architecture:
  ─────────────────────────────────────────────────────────────
  [OLTP Systems]                [OLAP Warehouse]
  ERP, CRM, POS  ──► ETL ──►   Dimensional Model
                               (star schema, fact/dim tables)
                                      │
                                   SQL queries
                                   from analysts

This model worked well for structured, well-understood business data at the scale of the 1990s and 2000s. Its limitations became apparent as data volumes grew, data types diversified (logs, images, sensor streams, text), and the latency of ETL pipelines (typically daily or weekly batch jobs) became incompatible with operational analytics needs.

The data lake concept, popularized around 2010–2012 with Hadoop's rise, proposed a different premise: store everything in raw form on cheap, scalable storage (HDFS, later S3), and apply schema on read rather than schema on write. The appeal was obvious — no data was discarded, no transformation decisions needed to be made upfront, and storage costs were orders of magnitude lower than MPP warehouse hardware.

The data lake's failure mode was equally obvious in retrospect: schema on read without governance produces a swamp. Data with unknown provenance, undocumented schemas, contradictory definitions, and no lineage is not an asset — it is a liability with good intentions. The "data swamp" critique of poorly governed data lakes was valid and predictable.

The Snowflake Moment¶

Snowflake's commercial success beginning around 2015–2016 represented the resurgence of the warehouse model with a critical innovation: separation of compute and storage. Traditional MPP warehouse systems (Teradata, Netezza, Vertica) coupled storage and compute physically — scaling required buying more hardware. Snowflake's architecture stored data in cloud object storage (S3, GCS, Azure) and provisioned compute elastically.

  Snowflake Architecture:
  ─────────────────────────────────────────────────────────────
  Cloud Object Storage (S3/GCS/Azure Blob)
         │ data files (Snowflake-proprietary format)
         ▼
  Virtual Warehouses (compute clusters, elastic)
         │ query results
         ▼
  Clients (SQL)

  Metadata Services: schema, access control, optimization

This architecture delivered the warehouse's strong guarantees — SQL semantics, ACID transactions, schema enforcement, fine-grained access control — with cloud-native elasticity. It also retained the warehouse's primary limitation: proprietary storage format. Data in Snowflake is stored in a Snowflake-specific columnar format that is not directly accessible to external tools. The "data lake" promise of open, format-agnostic storage did not apply.

The Lakehouse Emergence¶

The lakehouse concept, articulated most prominently by Databricks (and formalized academically in the 2021 "Lakehouse" paper by Zaharia et al.), proposes a third path: open file formats on object storage, with a metadata layer providing ACID transaction semantics, schema enforcement, and optimization information.

  Lakehouse Architecture:
  ──────────────────────────────────────────────────────────────────
  Object Storage (S3/GCS)
  Raw files in open format (Parquet/ORC)
         │
  Table Format Metadata Layer
  (Delta Lake / Apache Iceberg / Apache Hudi)
  Transaction log, schema, partition info, column statistics
         │
  Query Engines (Spark, Trino, Athena, DuckDB, Flink)
         │
  Data Products, ML pipelines, BI tools

The three dominant table formats represent slightly different design philosophies:

Delta Lake (Databricks): the transaction log is a series of JSON files stored alongside the data in object storage. Each commit appends a new log entry. Checkpoints compact the log periodically. Delta Lake has the deepest integration with Spark and the Databricks ecosystem.

Apache Iceberg (Netflix origin, now Apache): designed for maximum engine neutrality. The manifest structure separates file tracking from partition tracking, enabling partition evolution without rewriting data. Iceberg has broader multi-engine support (Spark, Flink, Trino, Athena, Snowflake external tables, Dremio).

Apache Hudi (Uber origin): designed primarily for record-level upserts — the ability to update or delete individual records in an otherwise immutable dataset. Hudi's copy-on-write and merge-on-read table types trade write amplification against read performance differently from Delta and Iceberg.

Control Plane vs Data Plane¶

Understanding the architectural distinction between control plane and data plane clarifies what each of these systems actually provides.

  Data Plane: physical bytes
  ─────────────────────────────────────────────────
  Parquet files on S3
  (compressed columnar data, no semantics)

  Control Plane: semantics and governance
  ─────────────────────────────────────────────────
  Table format metadata (Delta/Iceberg/Hudi)
  Schema registry
  Access control policies
  Query optimizer statistics
  Transaction log (what commits have occurred)

A data lake without a table format has a data plane (the files) but no control plane. A query engine can read the files, but it cannot know which files represent the current state of a table after deletions and updates, which schema version applies, or whether a concurrent write has occurred.

A data warehouse provides both planes, but couples them tightly and proprietary. The control plane (Snowflake's metadata service, Redshift's catalog) knows everything about the data plane (the proprietary storage format), but external tools cannot participate.

The lakehouse's architectural bet is that an open control plane (Delta/Iceberg/Hudi table format) layered over an open data plane (Parquet on S3) enables multi-engine participation without vendor lock-in at either layer.

Cost Structures¶

The cost structure of each model differs substantially.

Traditional warehouse: pay for compute and storage as a coupled unit. Scaling requires upgrading the system. Data must be loaded (ETL), incurring transformation cost. The warehouse vendor's proprietary format means data cannot leave without conversion.

Snowflake-era warehouse: pay separately for compute (per-second, per-warehouse-size) and storage (per TB on S3). Compute can be shut down when not querying. The proprietary format limits external tool access; egress from Snowflake to other systems requires external tables or data sharing features.

Lakehouse on object storage: pay for storage (S3 object storage is the cheapest tier of compute-adjacent storage), pay for compute only when queries run (Athena per-TB-scanned, EMR on-demand, Databricks per-DBU). No transformation required before storage — raw data lands in open format. The metadata layer (Delta/Iceberg) is open source.

At small scale, the differences are noise. At petabyte scale, the cost difference between storing data in an open format on S3 versus in a proprietary warehouse format is meaningful — both in absolute dollars and in the architectural flexibility to choose query engines based on workload rather than vendor relationship.

Governance Complexity¶

The lakehouse model's openness is also its governance burden. A warehouse enforces schema at write time — data that doesn't fit the schema doesn't land. An open lakehouse stores anything; governance must be applied as a separate operational concern.

Schema enforcement: Iceberg and Delta support schema enforcement at the table level, but it requires explicit configuration and is not universally applied by all writers.

Access control: object storage ACLs (S3 bucket policies) are coarser than database-level row and column security. Fine-grained access control in a lakehouse requires a catalog service (AWS Glue, Apache Atlas, Unity Catalog) with engine-side enforcement.

Data lineage: who wrote which data, from which source, with which transformation — is not tracked by the file format or the table format by default. Lineage requires instrumentation at the pipeline level.

Data quality: schema enforcement prevents type errors; it does not prevent semantically wrong values. Data quality validation (Great Expectations, dbt tests, Soda) is a separate operational layer.

A well-governed lakehouse is not simpler than a warehouse. It is more flexible. The governance responsibility shifts from the database vendor to the data engineering team, with open-source tools substituted for vendor-provided enforcement mechanisms.

Where Convergence Is Real and Where It Is Marketing¶

The genuine convergence between warehouse and lakehouse is at the query interface: SQL semantics, ACID transactions, schema enforcement, and analytics-optimized query execution are now available in open-format lakehouse architectures through Iceberg + Trino, Delta Lake + Spark SQL, and similar combinations. This was not true of first-generation data lakes.

The genuine remaining distinction is at the storage layer. Snowflake's proprietary format and Redshift's internal storage cannot be read by external tools without explicit data sharing features or data export. Open table formats can be read by any engine that implements the spec.

The convergence is partial because the control plane in a managed warehouse (catalog, access control, optimization, monitoring) is more integrated and lower-operational-overhead than the equivalent stack assembled from open-source components. The trade-off is lock-in for operational simplicity, which is a legitimate engineering decision for organizations without large data engineering teams.

Decision Framework¶

Small team, modest data volume (<1 TB), SQL-centric analytics: a managed warehouse (Snowflake, BigQuery, Redshift) provides maximum operational simplicity. The lock-in premium is worth paying for the reduced engineering overhead.

Large engineering team, high data volume (>10 TB), multi-engine workloads (BI + ML + streaming): a lakehouse on object storage with an open table format (Iceberg or Delta) provides the flexibility to use best-of-breed engines for each workload without re-ingesting data.

Regulated industry with strict data residency and audit requirements: evaluate whether the warehouse vendor's compliance certifications or the lakehouse's control plane auditability better serve your regulatory model.

ML/AI workloads with large training datasets: object storage + Parquet + feature stores is the standard architecture. Warehouse systems are not optimized for the unstructured-read-pattern of model training.

The question to ask is not "lakehouse or warehouse" but "which control plane trade-offs can we operate?" — and that is a question about team capability and organizational governance maturity, not about data volume or query patterns alone.