Making Open Tables Fast—and Useful Beyond SQL

A managed table can be correct and still be too slow to use. The performance question is how Delta Lake and the broader lakehouse design make open files behave well for selective analytical work without hiding them behind a closed warehouse. The answer is layered: transaction metadata identifies the table, statistics reduce the objects considered, layout improves locality, caching moves hot data closer to compute, and the execution engine turns those choices into a result. The same layers also determine whether direct access is genuinely useful to machine-learning and other non-SQL consumers.

The performance stack above open files

A transaction log is not a query engine. It tells the system which objects belong to a committed table state, but a query still needs to find the objects that might contain matching records and then read them efficiently. The lakehouse therefore builds an optimization stack above the open format. Each layer answers a different performance question, and a good result usually comes from their interaction rather than from one feature in isolation.

The first question is selection: which objects can be skipped? Checkpoints and log metadata can store statistics for each data object, including ranges or counts associated with its columns. If a query asks for a narrow range of values and an object’s statistics show that the range cannot occur there, the engine can avoid reading that object. This is a form of data skipping. It reduces I/O without changing the underlying Parquet representation, but its value depends on how informative the statistics are and how the records were laid out.

The second question is locality: when several objects must be read, are records likely to be near one another in the physical layout? A single global sort can help queries on one leading dimension, but it may offer little locality for filters on other dimensions. Directory partitioning has a similar limitation when too many combinations create an unwieldy number of partitions. Delta Lake’s described Z-ordering aims to cluster records across several selected attributes so that min/max statistics can skip more objects for multidimensional predicates. It is not a general secondary index. It is a layout choice whose benefit depends on access patterns.

The third question is where the bytes should be served from. Cloud object storage is inexpensive and highly available, but repeated reads of hot data pay network and object-store latency. Because Delta data and log objects are immutable from the table’s perspective, a compute cluster can cache them on local SSD or memory without risking that a cached object will silently change underneath a snapshot. The cache can include a representation convenient for the engine, such as partially decompressed data. This can reduce repeated I/O, but it is an optimization for recurring access, not a substitute for a correct table state.

The fourth question is whether the physical object layout remains healthy as data arrives. Streaming or incremental writers often produce small objects because they favor low write delay.