How Delta Lake Makes Object Storage Transactional
Delta Lake answers the lakehouse’s mechanism question by separating physical data from logical table state. Parquet objects continue to hold the records, while an ordered transaction log says which objects belong to each table version and how that membership changes. This separation lets readers access open files directly without inferring table state from a directory listing. It also gives writers a place to publish a change as one logical event. Understanding that representation and the read/write protocol explains what “transactional object storage” means—and where the guarantee stops.
A table version is a log-derived state
A Delta table is a collection of data objects plus a log of transaction actions. The data objects contain the table’s records, often organized into partitions, but their names and physical existence do not alone determine whether they are current. The log is the authoritative account of membership. An add action brings an object into a logical version; a remove action takes it out while preserving information needed for retention and historical reads. Metadata actions describe the table’s schema and configuration, while protocol and application information can record compatibility requirements or the identity of a writing process.
This arrangement changes the meaning of a write. A job can first create new Parquet objects in the object store, but those objects are only candidates until a successful log record refers to them. The table becomes the new table version when the corresponding metadata transition is committed. If a job prepares objects and then fails before that transition, a reader following the log does not treat the candidates as current data. Physical cleanup and retry are separate operational concerns; logical membership is decided by the committed history.
The log is ordered by increasing record identifiers. That ordering gives clients a way to describe a table’s evolution without asking the object store to perform a multi-object rename. A reader can reconstruct a state by applying the actions that follow an earlier known state. The design keeps the data objects in an open format while moving the difficult question—what counts as the table—into a small, durable control structure.
The important consequence is that a table version is a logical result, not a physical folder snapshot. A version can refer to objects that were written at different times, so long as the log makes their membership part of one committed state. A later version can remove an object from current membership without immediately erasing its bytes, which is what makes historical reads and delayed cleanup possible. The physical layout can therefore change while the logical history remains intelligible. This is a control-plane solution to a data-plane problem: the system does not need to make every object operation atomic if it can make the state transition that names those objects authoritative.
Replaying every historical action forever would eventually make reads expensive. Delta therefore periodically writes checkpoints in Parquet. A checkpoint summarizes the non-redundant state up to a particular log record: the live data objects, relevant removals that must remain as tombstones during retention, table metadata, and other current actions.