How One Update Rewrites an Answer
Pathway Live Data Framework's most important behavior becomes visible when a single input change is followed all the way to an output. The question is not whether the framework can read another event, but how it maintains the relationship between existing state and a new fact. This chapter traces additions, removals, and updates through a join and filter, then uses the same example to separate a finite batch run from a streaming computation that remains open to revision.
A revision has a lifecycle
An input event first changes the state represented by an input table. That event might add a new measurement, replace information associated with an existing record, or remove information that no longer belongs in the source. Pathway's documented model carries the consequences of those changes through downstream tables. A table is not simply an append-only log exposed to every transformation. It is a current representation whose rows can be added or withdrawn as the framework's understanding of the source changes.
That gives three useful states to distinguish. An addition creates a row that was not previously part of the represented table. A removal takes an existing row out of that state. An update is a transition from one row version to another: the previous version is removed and the new version is inserted. The two events may describe one logical change even though the representation has two sides. If those sides are collapsed into “another row arrived,” a consumer can mistake correction for duplication.
The distinction is important because downstream transformations depend on state, not only on the latest event in isolation. A filter evaluates whether a row currently satisfies a condition. A join relates the current state of two tables. A reduction summarizes the rows that remain relevant. When one input changes, each operation must reflect the new relationship and withdraw consequences that no longer follow. That is why the output of a live computation can contain a removal even when the application only appears to be adding new information.
Consider a pipeline with two sources. A measurement stream supplies names and values. A second input supplies a threshold for each name, perhaps from files that an operator can edit. The pipeline joins the sources on the name and filters for measurements whose value is above the current threshold. At first, the measurement table and threshold table establish a joined state; the filter then emits an alert for every joined row that satisfies the rule.
Imagine the measurements are A=8 and B=10, while both thresholds are 9. The current alert state contains only B, because 10 exceeds 9 while 8 does not. This is not merely a calculation that happened once. It is the result of a relationship among two changing tables: the measurement, the threshold, and the condition that connects them. The output is valid for the state those inputs currently represent.
Now change the threshold source so that A has a threshold of 7 and B has a threshold of 11. No new measurement is needed. The source that defines the rule has changed, and that change is enough to alter the alert set. The previous alert for B no longer satisfies the relationship, so it must be removed. The measurement for A, which was present all along, now satisfies the revised threshold, so a new alert must be inserted.
The important point is that the framework does not need to treat the threshold edit as an unrelated calculation. The threshold table is already a node in the dataflow, and the join records its dependency on the measurement table. The change travels through that dependency: the joined state is reconsidered, the filter's result changes, and the output receives the affected consequences. Unrelated branches of the graph do not become conceptually relevant merely because one threshold changed. The exact work performed by an operator depends on its implementation, but the documented dataflow model is organized around propagating affected changes rather than rewriting every output snapshot.
The update lifecycle can be read as a sequence of state transitions. Before the threshold edit, the join contains the measurement for B paired with its threshold of 9, and the filter marks that pair as an alert. After the edit, the same measurement is still present, but its relationship to the threshold has changed. The joined row is no longer the same logical result for the filter, so the old alert is withdrawn. For A, the measurement was already in the input state; what changed was the other side of the relationship, which makes a previously suppressed consequence visible. This is why a streaming system must retain the relevant table state: the new event may change the meaning of information that arrived earlier.
The table view and the output log serve different purposes. The table view answers what state the pipeline currently represents. The output log explains how the pipeline moved from one state to another. A downstream database may use the log to maintain its own current table, while an audit process may keep the log as a history of corrections. Both are valid consumers, but they must interpret the event fields according to the connector's contract. Treating a change event as a permanently independent fact loses the relationship that made it a change in the first place.
At the output boundary, Pathway documents this change model through diff and logical time. A row with diff=1 represents an addition to the output state; a row with diff=-1 represents a removal. The pair makes an update legible to a downstream system without requiring that system to rerun the entire pipeline. A CSV-like sink may preserve the events as a log of changes. A database connector may apply them as an update to a table-like destination. The connector determines the physical form, but the reason for the form comes from the dataflow's changing state.
This also explains why an output log can look temporarily confusing if it is read as a static table. Seeing an earlier B alert and then a removal of B does not mean the framework is asserting both states at once. It means the log records how the state changed. A consumer that reconstructs the current state applies the removal and arrives at the latest answer. A consumer that only appends rows without interpreting the change fields will preserve history but lose the intended current view.
The same lifecycle applies to more complicated transformations. In a grouped reduction, one input row can change a group total, which means the previous aggregate row is no longer current. In a join, a new matching row can create a result while a corrected key can withdraw one. In a windowed computation, a late event can revise a result that was already emitted. The details differ, but the core question is the same: which represented state changed, and which derived conclusions depended on it?
This is why “incremental” should not be reduced to a promise that every operation touches exactly one row. The evidence supports Pathway's model of forwarding affected changes and avoiding full-table output on every update. It does not establish a universal work bound for every operator or every external function. Incremental computation is best understood as a way of organizing state and dependencies; the cost still depends on the transformation, its retained state, and the connector behavior around it.
The same logic, different runtime
The measurements-and-thresholds pipeline can be developed against finite data before it is connected to an unbounded source. In static mode, the available input is loaded, the graph is evaluated, and the computation can terminate. The output is naturally read as the result for the finite data present at that run. This is useful for checking whether the schemas, join condition, filter, and expected rows are correct without waiting for a live source or debugging an indefinite process.
In streaming mode, the graph remains active. Input connectors wait for new data, and each committed update can propagate through the tables and transformations. The output is no longer one final table produced at the end of a finite computation. It is a sequence of changes that describes how the current result evolves as the framework receives more information. The process continues until it is terminated because the source has no natural point at which it can say that no more events will arrive.
The documented batch-to-streaming path is intentionally modest. A pipeline that reads a CSV directory in static mode can switch the connector to streaming mode while retaining the reduction and output logic. That example is valuable because it shows what Pathway unifies: the developer can express the data relationship once and change the source's execution mode rather than maintain separate transformation code for batch and stream processing.
What remains unchanged is the meaning of the table operations. A filter still selects rows by a condition. A join still establishes a relationship between columns. A reduction still summarizes a table. The same schema and transformation logic can therefore be tested against a finite fixture before the application is exposed to live updates. This continuity reduces one kind of duplication and makes static testing a useful first step rather than a separate programming model.
What changes is the contract around time and completion. A static run has a finite input frontier: once the available data has been consumed, the computation can finish. A streaming run has no such automatic ending. The result at any moment reflects the data received so far and can be revised by a later event. The output consumer must be able to interpret change, and the application must decide what it means for a result to be provisional, current, or eventually stable. Those questions become even more important when events can arrive out of order, but they exist at a basic level as soon as the source is live.
The connector boundary limits how far the unification goes. The documentation distinguishes connectors that work in streaming mode from those intended for static data, and it warns that the two kinds cannot simply be mixed. A debugging operation that waits until all data has been ingested makes sense for a finite table; in a never-ending stream, waiting for completion would mean waiting forever. Conversely, a live output needs a destination that can receive updates or a connector that can adapt those updates to the destination's constraints.
That difference changes how a developer reads a test result. A static test can answer whether a particular finite input produces the expected table. It cannot by itself prove that the live connector observes every relevant update, that a late event is handled as intended, or that the sink interprets removals correctly. The shared transformation logic is valuable, but the streaming system also includes source detection, commit boundaries, state retention, and output delivery. Moving to streaming is therefore a change in runtime conditions even when the central computation remains recognizable.
For example, a finite threshold file can confirm that the join and filter produce the expected alert for A and B. It does not show what happens when the file is modified after the pipeline has already emitted an alert. A streaming test must exercise that transition: change the threshold, allow the connector to detect the modification, and inspect whether the output removes the no-longer-valid alert and inserts the newly valid one. The point is not to turn the article into a test script. It is to identify the behavior that distinguishes a live dataflow from a batch calculation that happens to be run repeatedly.
The same reasoning applies to outputs. A static CSV result may be read as a finished artifact, while a streaming CSV can become an append-only record of changes with diff and logical time fields. If a reader opens the second file as though it were the first, they may conclude that the pipeline produced duplicate alerts. If a database connector translates the events into updates, the database may look like a single current table while the underlying computation has still processed a removal and an insertion. The destination's visible state can therefore conceal the change protocol that maintains it.
A sound migration keeps those questions explicit. First verify the relational logic on a small finite input. Then verify that the chosen streaming connector exposes the relevant kind of change and that the output connector has a clear interpretation for additions and removals. Finally, test a correction rather than only a new row: edit a threshold, change a source record, or withdraw an input and observe whether the output moves from the old state to the new one. The correction is the real test of a live dataflow because it reveals whether the system is maintaining a relationship or merely appending observations.
Pathway's value here is not that batch and streaming become indistinguishable. It is that the same graph can express the intended data relationship while the runtime changes how data enters, how long execution lasts, and how results are delivered. A batch result answers “what does this finite input produce?” A streaming result answers “what is the current consequence of the input received so far, and how should it change when the next event arrives?” The next question follows directly: when arrival time and event time diverge, how long should that current consequence remain open to revision?