Why a Bucket of Files Is Not Yet a Table
The lakehouse contract depends on a difficult premise: open cloud storage must behave enough like a managed table for many readers and writers to trust it. This chapter traces that difficulty from the object-store interface to its table-level symptoms. The useful judgment is causal. When a write becomes partial, a query becomes slow, or a reader sees an uncertain state, we should be able to identify which storage property produced the problem and which kind of control layer could address it.
What object storage makes difficult
A cloud object store is designed to scale a very large number of independent objects. Its basic abstraction is a key and a value: a key names an object, and the value is a blob of bytes. The key may look like a filesystem path, but that appearance can mislead. A path-like prefix does not necessarily represent a directory with atomic rename, locking, or a transaction boundary. Creating or replacing one object is a different operation from changing the logical contents of a table spread across many objects.
That difference from a conventional filesystem is not merely a vocabulary issue. A filesystem hierarchy often gives applications an operational story about renaming a temporary file into place or locking a directory while a change is made. An object store generally exposes operations at the object or key level, and a table assembled from many keys must build its own story about grouping them. The storage system can faithfully preserve each blob while remaining unable to say whether a group of blobs represents one completed business operation.
This interface is attractive for the lakehouse because storage and compute can scale separately. An organization can keep a large collection of data without running a query cluster continuously, then start compute resources when a workload arrives. Objects can be read in parallel, and range reads can retrieve only part of a large object. Those properties support inexpensive capacity and high aggregate bandwidth. They do not, however, answer which group of objects constitutes the current table or how a multi-object change becomes visible.
The first failure appears when an update touches more than one object. Suppose records for one customer are distributed across several columnar files. A deletion or correction may need to rewrite each affected file. If the storage system makes each replacement visible separately, a concurrent reader can observe a mixture: one file contains the new state while another still contains the old state. If the job fails after some replacements, there may be no single operation to roll back. The problem is not that any individual object is corrupt. The problem is that the table has no atomic boundary around the collection.
A related failure occurs when a new object is written successfully but the process that should announce its membership fails. The bytes now exist, yet one reader may include them and another may not. Conversely, a removal may be recorded in one place while a stale listing still suggests that the old object is present. Correctness therefore requires two distinct decisions: when a physical object is safe to reference, and which committed decision is authoritative for readers. Treating object existence as table membership collapses those decisions and makes recovery depend on guesswork.
Consistency across keys makes the situation harder. Object stores have different guarantees, and some widely used systems have historically offered eventual visibility for listings or limited guarantees across keys. A client may write a new object and not immediately discover it through a list operation. One client may see an update before another. The precise behavior varies by provider and by operation, so a protocol cannot assume that a directory listing is an instant, authoritative picture of table state. It must instead define how a reader finds a trustworthy version despite delayed or incomplete visibility.
Atomicity and visibility are related but not identical. An operation can be atomic from the writer’s point of view and still be difficult for another client to discover if metadata visibility is delayed. A list operation can eventually show every object and still fail to tell the reader which combination was intended as one transaction. A useful table protocol must make a small piece of metadata—the committed version—serve as the root of the read, rather than asking the reader to infer intent from an unstable set of objects.
The second failure is metadata cost. A query rarely needs every object in a large table. Columnar files may contain min/max statistics that can help skip irrelevant files, but reading those statistics one object at a time requires many metadata requests. Listing a large prefix is also paginated and relatively slow. At modest scale, the delay can be hidden inside a query. At millions of objects or partitions, discovering what to read can take longer than reading the selected data. A table can therefore contain the right information and still be operationally unusable because the path to its metadata is too expensive.
This overhead grows in ways that are easy to underestimate. A partitioning scheme may reduce the search space for one common predicate while multiplying directories for combinations of fields. A streaming pipeline may produce many small objects because small writes reduce ingestion delay, then force every later query to pay the cost of discovering and opening them. Reading each file footer can also turn a selective query into a metadata workload before any useful data reaches the execution engine. The remedy cannot be only “add more compute”; it must reduce the amount of object-store discovery the compute performs.
The third failure is a conflict between update granularity and read efficiency. Large objects make sequential analytical reads efficient: fewer requests and longer transfers allow the storage system and compute nodes to use their bandwidth. But a point update to a large object may require rewriting all of it. Small objects make inserts or streaming writes easier, yet they increase request count, metadata overhead, and the amount of work a query must perform to assemble a scan. Object storage does not choose the right compromise for a table; the table design must manage it.
This tension explains why compaction and layout are not cosmetic maintenance. If a writer favors low latency, the table accumulates fragments. If a reader favors long sequential transfers, updates become expensive. A managed table needs a way to reorganize physical objects without changing the logical state seen by concurrent readers. It also needs metadata that lets a query avoid data whose values cannot match its predicate. Those are table responsibilities even though the underlying store only knows objects.
This is why a directory of Parquet files can be perfectly readable and still fail as a shared analytical table. It has data, but not necessarily a committed state. It has file statistics, but no efficient central path to those statistics. It can accept a new object quickly, but that does not make a multi-object update atomic. The visible symptom—wrong results, slow planning, stale discovery, or difficult recovery—can have the same root cause: the file interface is being asked to represent relationships it does not own.
The diagram shows two different problems that are often conflated. A log can establish which objects belong to a committed version, addressing the correctness boundary. A checkpoint and statistics can make it cheaper to find the relevant objects, addressing the metadata path. Neither removes the need to read the data, and neither makes every workload equally suitable for object storage. They change where the table’s state and selection information are found.
Three ways to supply the missing table behavior
The simplest approach is to keep the directory model and add conventions. Writers place columnar files in predictable prefixes, perhaps partitioned by date or another field. Readers list the relevant prefixes and inspect file metadata. This preserves maximum compatibility: any tool that understands the file format can open the objects without consulting another service. It is also the approach with the weakest table semantics. Multi-object updates are exposed piece by piece, listings may be stale or expensive, and there is no built-in history, rollback, or audit trail. The system remains “just a bunch of files,” even when teams build elaborate procedures around it.
Conventions can be sufficient for append-only data that is rarely corrected and can tolerate readers discovering changes asynchronously. They become fragile when the same table supports privacy deletion, late-arriving records, repair of an incorrect upstream value, or concurrent analytical reads. In those cases, operators need to know not only whether a file exists but whether it should count, what it replaced, and which earlier state can be restored. Procedures can imitate those answers, but each additional writer or reader becomes another chance to interpret the convention differently.
A second approach is a closed-world storage or warehouse engine with a separate, strongly consistent metadata service. The service becomes the source of truth for which objects belong to a table and can coordinate updates, indexes, searches, and layout decisions. This can provide a clean database experience because every access goes through a component designed to mediate table state. The cost is operational and architectural. The service must remain available, may add overhead to reads, and can become a dependency for every engine that wants to use the data. External tools may have to stream data through it rather than read the objects directly, and the proprietary boundary can make migration or multi-engine access harder.
A separate service can also be the right choice when the workload needs capabilities that an object-store-native log cannot provide economically, especially high write rates or richer cross-table coordination. The trade-off is that the service becomes part of the data path and the recovery plan. It must be operated, scaled, secured, and kept consistent with the objects beneath it. The lakehouse argument is not that such a service is always wrong; it is that many analytical workloads may prefer to keep the data plane directly on object storage and add only the coordination needed for table state.
A third approach places the table’s transaction metadata in the object store alongside the data objects. Delta Lake follows this direction: Parquet remains the data format, while an ordered transaction log records the objects that belong to each version and the operations that change that membership. The design aims to let clients coordinate through storage operations rather than require a heavyweight always-on service for every read. A client can first identify a committed version and then read the referenced objects directly. That preserves much of the lake’s access model while adding a logical table boundary.
This approach is not a free escape from object-store constraints. A log write still needs a way to ensure that only one client claims the next version. Different storage systems provide different atomic operations, and some deployments may need a lightweight coordination service for commits. The log also introduces a write-rate ceiling tied to object-store latency and optimistic conflicts. The design is therefore most naturally aligned with analytical workloads that perform relatively few, relatively large transactions, not with a high-frequency transactional database that commits tiny changes continuously.
The three approaches also place responsibility in different locations. With raw directories, each writer and reader must behave carefully, and correctness is partly a convention. With a closed metadata service, one provider owns the authoritative state, but the service becomes a persistent operational and interoperability boundary. With an object-store-native log, the table protocol travels with the data and can be shared by multiple clients, but every client that wants full guarantees must understand that protocol. None is universally superior; each answers the question “who owns table state?” differently.
For a lakehouse, the third option is attractive because it preserves the subject’s central relationship: open data remains directly reachable while management is added above it. But the transaction log should be understood as a carefully bounded control plane, not as proof that object storage has become a general-purpose database. It can define versions, coordinate supported changes, and centralize useful metadata. It cannot by itself provide unlimited transaction throughput, instant streaming, automatic cross-table atomicity, or compatibility with a reader that ignores the table protocol.
The diagnosis therefore gives us a precise test for the next mechanism. If Delta Lake works, its log must do more than record filenames. It must let readers reconstruct a coherent table version, let writers publish changes without exposing partial state, and give query planning a cheaper source of metadata than repeated object listing. The following synthesis will use those requirements to distinguish a managed open table from both a raw data lake and a closed warehouse before examining the protocol itself.