When Data Keeps Changing, the Pipeline Must Change With It

Pathway Live Data Framework is a Python-facing system for building data-processing and AI pipelines over information that does not stay still. The focus here is not simply how to process a stream quickly, but how to keep an answer coherent as its inputs change. That distinction gives this series its central question: what does it take to describe a computation once and keep its results aligned with new, revised, late, or newly indexed data?

A conventional batch pipeline has a reassuring shape. It reads a set of records, applies transformations, writes an answer, and stops. That shape remains useful for reports, tests, backfills, and many data products. But it becomes an incomplete mental model when the source is a Kafka topic, a directory that receives new files, a database change stream, or a document collection whose contents are edited after they were first indexed. In those settings, yesterday's answer is not necessarily wrong because the original calculation failed. It may simply be out of date.

Pathway's importance lies in treating that change as part of the computation rather than as a separate emergency procedure. Its documentation describes a Python API, a Rust execution engine, connectors to external systems, stateful transformations, persistence, and tools for live LLM and RAG pipelines. These pieces are easiest to understand as one system: a dataflow whose tables represent the current state implied by the events received so far, and whose downstream results are revised when those tables change. The framework is the subject; incremental dataflow is the perspective that explains how its parts relate.

The problem is not only speed; it is revision

Suppose a pipeline receives measurements and a second source supplies the threshold for each measurement. The first source may be live while the threshold source is a file that can be edited. An alert is produced when a measurement exceeds its current threshold. If a threshold changes, the right output is not a second copy of the old alert alongside the new one. The old conclusion has become invalid and must be withdrawn; another conclusion may need to appear.

That small example exposes the difference between processing a stream and maintaining a result. A system that merely appends each new calculation can leave contradictory answers for downstream users to reconcile. A system that rebuilds every table from the beginning can preserve a snapshot view, but it may spend unnecessary work repeating unaffected calculations. The harder requirement is to retain enough state to understand what changed, propagate the consequences through the relevant parts of the computation, and communicate the revised state to the destination.

This also changes how “correct” should be understood. A result can be correct for the data received at one moment and still be legitimately revised later. The system needs a way to make that transition explicit, so a consumer can tell whether a row is new, withdrawn, or the latest version of an existing conclusion. That is a different contract from a report that is complete only when its file has been written. It makes the path from input to output part of the meaning of the result: the same calculation must account for what it has seen, what it has not yet seen, and how it represents a correction.

The distinction matters operationally as well as mathematically. If an input changes but an unaffected branch is recalculated from scratch, the result may still be correct, but the system has paid for work that did not contribute to the change. If a downstream sink cannot accept removals, the framework may need to adapt the update into a database write or another destination-specific operation. Incremental dataflow therefore connects computation to interfaces: the benefit is not only less work inside the graph, but a more honest way to communicate changing state outside it.

Pathway's documented output model makes that revision visible. In streaming mode, a result is sent as changes rather than as a complete table after every input. An addition is represented as an insertion, a removal as a deletion, and an update as the removal of the previous row followed by the insertion of its replacement. Streaming outputs may carry logical time and diff fields; an output connector can preserve those events or translate them into a destination-specific update. The exact representation depends on the sink, but the underlying idea is stable: a live answer has a history of changes, not just a final-looking file.

This matters well beyond alerts. A temporal aggregate can change when a late event arrives. A customer profile can change when a database record is corrected. A search index can become stale when a document is edited. In each case, “real time” means more than low latency. It means that the system has a defined way to revise what it previously believed.

What Pathway brings together

Pathway presents that problem through a Python interface. Developers describe schemas, connect to inputs, select or join tables, apply reductions or other transformations, and connect results to outputs. The repository describes a Rust engine based on Differential Dataflow and positions the framework for both batch and streaming workloads, real-time analytics, data indexing, and RAG. The Python surface is therefore not the whole runtime: it is the accessible language for expressing the relationships that the engine will maintain.

The most useful relationship is among five roles. An input connector reads from an external system. A schema gives incoming records a known shape. A Pathway table represents the current state of that input as the system has observed it. Transformations derive new tables from existing ones. An output connector sends the resulting changes elsewhere. Calling pw.run() starts execution; constructing those objects beforehand describes the graph but does not itself process the data.

Those roles also divide ownership of change. The connector knows how to observe the outside system and turn its observations into input events. The table holds the framework's current representation of those events. A transformation states how that representation should be filtered, combined, grouped, or otherwise derived. The output connector decides how a changed derived table can be expressed in the destination. This division is what lets a developer reason about a pipeline as a connected system instead of hiding every concern inside one polling loop.

It also explains why the graph can be built before data arrives. At construction time, Pathway knows the relationships among tables and operations, but it has no complete input state to process. Execution supplies that state and keeps it available for future changes. The distinction is subtle but important: writing a pipeline describes what should happen when data exists; running it creates the ongoing relationship between data arrival and result updates.

This vocabulary prevents several common misunderstandings. A table is not necessarily a static dataset, even though it has a stable schema. A transformation is not simply a line that mutates a variable; in the documented functional model it returns another table. A connector is not just an import shortcut; it determines how the outside world becomes a sequence of additions, updates, or removals, and how results are represented when they leave the framework. The runtime is not an invisible afterthought, because it owns the ongoing work of propagating changes.

The framework also keeps the interface deliberately broad. The documentation describes connectors for files, message systems, databases, cloud storage, HTTP, and other destinations, as well as custom Python connectors. It is unnecessary to memorize that catalog to understand the design. The important questions are narrower: Is the source finite or unbounded? How are changes detected? Can the source be replayed after a restart? Does the sink accept revisions, or must it translate them into its own update model? Those choices sit at the boundary of the dataflow and cannot be made by the transformation logic alone.

A mental model that is useful—and bounded

The basic picture is a graph rather than a script that repeatedly starts over:

flowchart LR S["External sources"] --> C["Input connectors and schemas"] C --> T["Dynamic tables"] T --> X["Transformations and maintained state"] X --> O["Output connectors"] O --> D["External destinations"] S -.-> U["New or revised data"] U -.-> C

The graph explains why Pathway can support a common processing design for static and streaming data. A finite input can be loaded, processed, and allowed to terminate. A streaming input can remain open, with new data entering the same kind of graph and causing affected results to change. The documented batch-to-streaming example changes the connector's mode while leaving the reduction logic intact. That is a meaningful simplification: development and testing can begin with finite data, while the processing relationships need not be rewritten merely because the source becomes live.

It is not a claim that the two modes are identical in every respect. Streaming computations normally continue until the process is terminated and expose results through output connectors. Static computations can finish and can use debugging outputs that wait for all available data. Connectors that assume finite completion cannot simply be mixed with connectors that wait forever for more events. Time-related operations also introduce questions that do not arise in the same way for a closed dataset. A unified pipeline reduces one class of architectural duplication; it does not abolish the semantics of arrival, revision, or completion.

That boundary gives a practical test for the mental model. A source belongs comfortably in a live dataflow when the system can observe its changes, represent them with a stable enough schema, and—especially for recovery—read them again when necessary. A destination belongs comfortably at the other end when it can accept a stream of changes or has a clear way to turn those changes into its own state. If either side lacks those properties, the transformation logic may still be useful, but the surrounding connector design becomes the difficult part of the system.

Nor does an engine described as incremental make every application concern disappear. External Python functions, model calls, and destination systems have their own latency, failure, and side-effect behavior. The selected project material supports a strong architectural account of how Pathway builds and updates a dataflow. It does not independently establish that Pathway is universally faster than other systems, or that a live RAG pipeline will always produce a better answer. Those judgments require workload-specific evidence.

The practical conclusion is more precise. Pathway offers a way to make changing inputs first-class in a Python-described computation. Once that model is clear, the next issue is control flow: what exactly does the framework construct before it runs, and how do connectors, tables, transformations, and outputs divide responsibility for a result that must keep changing?

References