What the Lakehouse Must Preserve and Add
A lakehouse is not defined by putting files in cheap cloud storage. It is defined by the relationship among open data objects, a layer that manages their logical state, and the many systems that need to use that state. The question here is what each part is responsible for, and what a reader should demand before calling a collection of files a reliable shared table. The answer is a three-part model: open storage preserves access and flexibility; a metadata and control layer supplies identity, history, and rules; compute clients consume the same data through interfaces suited to their work.
Three responsibilities, one table
Start with a distinction that is easy to miss. A physical data object is not the same thing as a logical table. A Parquet file contains encoded records and can be read by a compatible tool, but it does not by itself say whether it belongs to the current version of a table, whether a newer object replaces it, or whether a multi-object change was committed as one operation. The physical object answers “what bytes are here?” A table state must also answer “which bytes count, under which schema, at which point in time?”
The open data layer is responsible for storing those bytes in a format that multiple systems can access. In the supplied design, Parquet is attractive because it is column-oriented, supports nested data, compresses well, and already has implementations in many engines. The broader lakehouse idea could use another standard format, but the architectural requirement is the same: the bulk data should not be trapped inside one proprietary query service. Direct access lowers the friction for a new consumer and lets storage and compute scale separately.
Direct access is not the same as unrestricted interpretation. If every reader simply scans whichever objects it can find, two readers may disagree about the table they are querying. One may see a file that another regards as removed. A reader may observe half of a change that was intended to be atomic. The open layer makes bytes available; it does not decide which bytes form a valid snapshot. That decision belongs to the second responsibility, the metadata and control layer.
The control layer gives the logical table a durable identity. It can record which data objects belong to a table version, coordinate changes, preserve earlier states, and expose information that makes reads selective rather than blind. It can also carry schema information, statistics, provenance, audit records, and other management state. These responsibilities are related but not identical. A version tells a reader which state to see. Statistics help a query avoid irrelevant objects. A schema check can reject a bad write. An audit record can explain who or what produced a change. They become more useful when they describe the same committed transition.
The crucial ownership question is what happens between a proposed change and a visible change. A writer may create new data objects successfully, but those objects should not become part of the logical table merely because the bytes exist. The control layer decides when the change is accepted and how readers learn about it. If the write fails before that point, the candidate objects may be left for cleanup or a retry, but they are not supposed to masquerade as current table state. If it succeeds, readers need one coherent account of the new membership. This separation between preparation and commit is what lets physical storage remain flexible while logical state remains controlled.
This is why the control layer is more than a catalog that helps users find files. A catalog may answer where an object is. A table protocol must establish when an object becomes part of the table, how a replacement is made visible, and what a concurrent reader is allowed to see. It owns the logical boundary between candidate data and committed data. In Delta Lake, the transaction log is the concrete mechanism for that boundary; its details matter because the log becomes the table’s account of state, not merely a directory index.
The control layer also does not need to become a new closed warehouse. Its job is to describe and protect logical state, while the data plane can continue to serve large reads from object storage. That division is important for cost and elasticity: different applications can start compute resources when needed and access the same underlying data. It also makes the control layer a potential bottleneck or compatibility boundary, so the design must keep its metadata operations efficient and its protocol available to the clients that need it.
The third responsibility belongs to compute and access clients. A SQL engine may need a stable snapshot and a way to push filters toward the data. A DataFrame system may represent transformations as a plan that can be optimized before it reads. A streaming job may need to discover new committed data and resume after a failure. A machine-learning library may need direct access to large batches of records without translating every operation into warehouse SQL. These clients do different work, but they should be able to refer to a common logical table rather than maintain unrelated copies.
The three responsibilities form a system, not a stack in which the upper layer owns everything. Open objects remain a valuable interface. The control layer owns the logical state that makes them a table. Clients own computation and may run on different clusters or hardware. A machine-learning job can therefore use a compute environment suited to its workload while reading the same managed data that a BI query uses. The benefit comes from shared state with separate compute, not from forcing every workload through one execution engine.
Imagine a table fed by operational changes, queried by analysts, and sampled by a model-training job. The analyst needs a stable answer while the ingestion job prepares its next batch. The model may need to reproduce the snapshot used for an earlier experiment. The ingestion process, meanwhile, needs a place to record a new state without making the analyst see half of it. The storage objects, control layer, and clients each have a different role in that situation. Treating them as interchangeable is what causes the original lakehouse problem to reappear inside the supposedly unified system.
This arrangement also clarifies what the lakehouse gives up. A traditional relational system can hide its storage representation and change internal layouts without asking every external reader to understand them. A directly accessible lakehouse format becomes part of the public contract. Changes to the format, log protocol, or connector behavior can affect clients outside the main engine. The trade is deliberate: less storage-layer independence in exchange for portability, high-bandwidth reads, and the ability to use the data from multiple kinds of systems.
The contract with consumers
The first consumer is the analytical SQL path. BI users need more than files that happen to contain familiar columns. They need a stable relation, predictable visibility of updates, and enough metadata to plan selective queries. The lakehouse therefore adds database-like behavior without removing the open representation. A SQL client may ask the table layer for a snapshot, use its statistics to identify relevant objects, and then read those objects from storage rather than stream every byte through a proprietary front end.
The second consumer is the DataFrame and machine-learning path. Advanced analytics often manipulates large datasets with programmatic operations and non-SQL code. Routing those reads through ODBC or JDBC can create an awkward boundary: the warehouse must expose data through an interface designed for relational queries, while the ML system wants batches, columns, or records in forms that fit its own execution. Direct access to an open format can remove that extra movement. Declarative DataFrame APIs add another possibility: they can describe data preparation as a plan, allowing the underlying engine to apply selection, projection, caching, or layout optimizations before materializing the data for the model.
The important point is that “direct” does not mean “unmanaged.” A machine-learning consumer still needs to know which files belong to the intended snapshot. In the described design, a client can consult the table metadata to identify the current objects and then pass those objects to a library that understands the open format. A reader that understands only ordinary file directories may need a consistent manifest representing a particular snapshot. That is a useful compatibility path, but it is read-only in character and does not give the external system the full table protocol.
This makes interoperability a spectrum rather than a yes-or-no property. At one end, a protocol-aware client can read and write managed tables and participate in their version rules. In the middle, an engine may read a table through a connector or a generated manifest. At the other end, a generic Parquet reader can open individual objects but cannot know whether it has reconstructed the current table. All three forms of access can be useful, but they support different guarantees. Saying that a table is open without naming the access level hides the most important practical distinction.
That spectrum also prevents a common category mistake. An external-table feature in a warehouse can make files queryable from SQL, but queryability is not the same as management. The external engine may still lack a way to perform atomic changes over the files, enforce the lake’s table contract, or make those files equally accessible to a non-SQL training job. Conversely, a table layer that preserves open objects may offer strong state management while leaving some advanced query features to the consuming engine. The right comparison is therefore not “can both systems run a SELECT?” but “who owns state, which clients can participate, and what happens when data changes?”
Streaming consumers expose the same issue over time. A batch query can select a snapshot and finish. A streaming job needs a durable way to determine which new committed changes it has already processed and where to resume. The table layer must therefore expose change history or an equivalent protocol, while the data objects remain readable by the processing engine. The fact that the same storage can support both batch and streaming is not automatic; it depends on how the control layer represents and exposes transitions.
Governance and data quality are another part of the contract. A table layer can enforce that new records match the expected schema, preserve schema history, reject records that violate constraints, and attach provenance or audit information to committed changes. It can also check whether a client is authorized before giving it credentials to read the underlying objects. These controls are valuable because an unmanaged file directory often accepts a write without knowing whether the records fit the logical table or how the write should be investigated later.
But the boundary must remain visible. A raw object-store client that bypasses the table protocol can also bypass its schema checks and access controls. A table audit record does not automatically describe every transformation performed in an external system or every copy made downstream. Governance is stronger when readers and writers respect the same control layer, but the lakehouse is not a complete identity, policy, or lineage system by itself. It provides a place where important state changes can be made visible and enforceable; surrounding systems still determine how far that visibility reaches.
The same qualification applies to the promise of simplification. If one managed open table can serve BI, engineering, streaming, and machine learning, an organization may avoid some duplicated storage and separate loading jobs. Yet every consumer must still implement or use the relevant access protocol, and the table layer must continue to meet its performance and reliability responsibilities. A unified substrate reduces boundaries only when the remaining boundary is clear enough for all participants to trust.
That is the lakehouse contract in its most compact form. The storage layer keeps data inexpensive and directly reachable. The control layer says which data is valid, current, versioned, and governed. The clients use that shared state for different kinds of computation. The next problem follows immediately: cloud object stores are excellent at scaling bytes, but they do not natively supply the atomicity and metadata behavior that this contract requires. Understanding that mismatch is the key to seeing why Delta Lake needs a transaction log at all.