Time Is Part of the Data
Pathway Live Data Framework can revise a result when a new event arrives, but revision alone does not answer whether that event should still count. The focus here is the difference between event time and processing time, and the policy that follows when records arrive late or out of order. By the end, the choice among waiting, revising, and closing a result will be concrete, as will the reason an ASOF join expresses a different relationship from an ordinary equality join.
When arrival time and event time disagree
Every streaming record participates in at least two timelines. Event time is the time assigned to the event itself: when an order was placed, a sensor reading occurred, or a corporate announcement was made. Processing time is when the framework receives and processes that record. The two can differ because a source may buffer messages, a network may delay them, or a producer may emit records in an order that does not match the events those records describe.
That difference turns the output of a live computation into a provisional statement. At processing time, Pathway can calculate from the data it has received. But an event with an earlier event time may still arrive later and change an aggregate or a join. The result was not necessarily an error when it was first produced; it was the best result supported by the available prefix of the input. Once the late record is accepted, the dataflow may need to revise that result so it reflects the newly observed history.
A simple average makes the problem visible. Suppose a system is collecting readings for an interval and produces an average after the first few records arrive. If another reading belonging to that interval appears later, the average should change if the application wants a result based on all relevant readings. The engine cannot infer from a quiet period whether the interval is complete or whether the next relevant record is merely delayed. Silence is ambiguous: it can mean absence, or it can mean that the channel has not delivered the information yet.
The same ambiguity appears in a more consequential form when an aggregate drives an alert. An operational monitor might flag unusually low power usage while a window is still receiving readings. If the missing readings would have raised the average, an early alert could be misleading. Conversely, waiting for every possible late record can postpone an alert indefinitely and force the system to retain old state indefinitely. A live dataflow therefore needs more than an operator that knows how to aggregate; it needs a policy for how long the aggregate remains open to change.
Pathway's documented default favors beginning computation promptly and continuing to account for potential late data. That supports results that remain consistent with the data received, but it has a cost. To revise a result, the system must retain enough historical state to know what the late record affects. If the maximum lateness is unknown, the amount of information that might remain relevant can grow without a natural bound. The framework cannot manufacture a safe memory limit without changing the meaning of “still allowed to revise.”
This is the underlying trade-off behind temporal behavior. Low latency means producing a result before all likely records have arrived. High revision tolerance means holding state and accepting that the result may change. Resource release means deciding that some late information will no longer alter a particular computation. None of those choices is universally correct.