From Prototype to Production: Recovery, Scale, and Boundaries

A temporal policy is only useful if the running system can preserve it when the process stops, the workload changes, or several workers share the computation. This chapter examines Pathway Live Data Framework at that operational boundary: what the Python-to-Rust design contributes, how workers divide a logical dataflow, what persistence must remember, and why scaling, observability, and licensing change the practical production envelope. The goal is not to treat deployment as a separate checklist, but to show how it carries the same changing state that the earlier chapters described.

The execution layer owns state, progress, and recovery

Pathway uses Python to describe the pipeline, but its documentation places table contents and many transformations in a Rust engine. The division has a clear purpose. Python offers a familiar way to express schemas and relationships; the engine maintains the state and executes the dataflow without making the developer implement every update loop in Python. The repository's packaging describes a PyO3 extension module, while the Rust manifest retains direct dependencies on Timely Dataflow and Differential Dataflow. Those details support the architectural boundary, not a claim that every piece of application code becomes equally fast.

The boundary also tells us where a guarantee stops. Built-in table operations are part of the engine's dataflow model. A user-defined Python function, a remote API call, or a language-model request can introduce blocking, retries, side effects, or costs that the engine cannot erase. The Python surface makes those components convenient to combine with a pipeline, but convenience is not the same as identical execution semantics. A production design still has to decide what happens when an external call is slow or fails and whether repeating a call is safe.

Pathway's worker model keeps one logical computation while dividing its work. The architecture documentation describes each worker running the same dataflow over a different shard or partition. Workers can read the portions of a partitioned source assigned to them, exchange data needed by downstream operations, and communicate progress as computation advances. A file-based source can be divided by assigning files to workers; a source with explicit partitions can use those partitions; a source without a natural partition may require one worker to read and distribute data.

Progress is not merely a performance counter. Each node needs to know how far relevant input has advanced before it can interpret the results it has received. Workers exchange progress information so that the distributed graph can determine which portion of the input a result depends on. That is what keeps a parallel computation from treating an early partial view as though it were a settled global state. The exact implementation is below the Python interface, but the system-level consequence is visible: parallelism must preserve the meaning of time and dependency, not only divide CPU work.

The state is normally kept in memory while the pipeline runs. That supports fast access to the current tables and the retained state needed by stateful transformations. It also means that a process failure can erase the working state unless the application has a recovery mechanism. Persistence supplies that mechanism by saving enough information to reconstruct the computation from a known point rather than replaying the entire source from the beginning.

Pathway's persistence model is more than a cache of the latest output. The documented configuration can store metadata about advanced times, source positions, and the computation graph, along with a larger snapshot of internal state. It combines state serialization with a write-ahead logging mechanism and can use filesystem or S3 storage. A snapshot interval creates a trade-off: more frequent snapshots can make recovery more current but consume more resources, while less frequent snapshots leave more work to replay after a failure.

The source position is part of the recovery contract. A snapshot of operator state is not enough if the connector cannot tell which input records have already been consumed. The source must be replayable, and the framework must be able to match the current source with the source represented in the saved state.