The Dataflow Behind a Live Result
Pathway Live Data Framework becomes easier to reason about once its dataflow is treated as an operating system for changing tables rather than as a collection of input and output helpers. The focus here is the boundary between an external source and a running computation: how a connector, a schema, a table, a transformation, and pw.run() divide responsibility. Understanding that construction lets a reader predict what Pathway knows before execution, what state it maintains afterward, and where a live result leaves the framework.
The input boundary: shape first, arrival second
An input connector is the first point at which an outside system becomes part of a Pathway computation. A file directory, message topic, database change feed, or another source has its own format and rules for observing change. Pathway's connector turns that source into a table that the rest of the pipeline can use. The connector therefore owns the question “what has arrived, and how should it be read?” It does not, by itself, decide what the arriving records mean for the application's result.
A schema supplies the stable part of that contract. It names columns and gives them types so the connector can interpret incoming records consistently. A measurement might have a name and a numeric value; a document input might have binary content plus metadata. The exact fields vary with the application, but the relationship is the same: the schema describes the shape of an event, not the moment when the event will arrive. A streaming source can keep delivering records to a table whose schema does not change from one update to the next.
That distinction is easy to miss because ordinary tabular data makes structure and content appear inseparable. In Pathway, a table has a static schema but dynamic content. It represents the latest state implied by the events the framework has received so far. If a new record is observed, the table can gain a row. If an existing record is corrected or removed, the represented state can change. The table is therefore closer to a current view of an evolving source than to a file frozen at the time the program started.
This design gives the rest of the dataflow a stable object to work with. A filter can ask which rows satisfy a condition; a join can relate columns from two tables; a reduction can summarize a group. None of those operations has to know whether a row came from a file, Kafka, or a database log. They receive a table with a declared shape and operate on the relationship expressed by that table. The connector remains responsible for translating the outside world's events into that representation.
Consider the measurements-and-thresholds example more closely. One input table contains a measurement name and value. A second table contains the same name and its current threshold. The first connector may listen to a message stream while the second watches files that can be changed by an operator. The schemas make the two inputs comparable, but they do not perform the comparison. The join transformation owns the relationship between matching names; the filter owns the rule that a value must exceed its threshold; the output connector owns the publication of an alert. Each stage has a distinct reason to exist, and each stage can be inspected without pretending that the source systems are identical.
This separation is particularly useful when only one input changes. A new measurement should not require the pipeline to rediscover how thresholds are read. A changed threshold should not require the connector to know how alerts are defined. The tables provide the shared language: both sources become structured state, and the transformations express the dependency between those states. The fact that one table is live and the other is editable does not alter the conceptual shape of the join, although it does affect the runtime behavior and the kinds of corrections that may be produced.
The boundary still matters when choosing a source. The documentation describes connectors for both finite and streaming data, but they do not all have the same runtime behavior. A static connector can read what is available and let the computation finish. A streaming connector waits for future updates and keeps the computation alive. A source that cannot expose changes, or cannot be read again after a restart, may impose constraints that the transformation layer cannot solve. Connector choice is therefore part of the dataflow's semantics, not merely a deployment preference.
The same questions apply at the other end. An output connector is responsible for translating a derived table into a destination's language. Some destinations can accept a stream of insertions and removals. Others expect a table-like state and need the connector to apply a change directly. Pathway's core transformations can remain focused on the data relationship, but the connector determines how that relationship becomes an external side effect.
From table relationships to execution
Once input tables exist, Pathway's transformations build new tables from them. This is a functional style of dataflow: a transformation returns a derived table rather than mutating the input table in place. The distinction is more than a matter of programming taste. Each derived table becomes a visible node whose relationship to its inputs can be followed, updated, and connected to another operation.
Consider a pipeline that filters measurements and then reduces the survivors into a summary. The filtered table is one state representation; the reduced table is another. If a join is added, the joined table is another node with a dependency on both inputs. The code describes these relationships before any particular measurement or threshold has been processed. Pathway can then reason about how a change at one input may affect the nodes downstream from it without treating the whole program as an undifferentiated loop.
The graph is the central construction. Connectors and transformations add nodes and edges that describe where data can flow and how each table depends on the previous one. At this point, there is no complete dataset in the graph. There is a recipe for producing and revising one. That explains why defining a pipeline does not immediately print a result or write a file: the graph has been described, but the computation has not yet been started.
This distinction separates two kinds of reasoning that are often mixed together in ordinary scripts. The first is relational: if a row appears in this table, combine it with matching information from that table and retain only the rows that satisfy a condition. The second is operational: watch the source, decide when an input update is committed, keep the relevant state, and deliver the resulting change. Pathway's table operations express much of the first kind, while the connectors and engine supply the second. A pipeline can look short in Python because the operational machinery is below the surface; it is still present and it still determines what the program does when data arrives.
The graph also provides a natural place to identify dependencies. If an output depends on a joined table, and that joined table depends on two inputs, then a change in either input may matter while an unrelated input branch should not. This is more informative than reading a script from top to bottom because it exposes relationships rather than just execution order. It gives a reader a way to ask which state belongs to which node, what event can trigger a change, and where the change should stop if it has no downstream effect.
Calling pw.run() changes the status of that description. The framework begins ingesting data, maintains the tables represented in the graph, and propagates the effects of input updates through the relevant transformations. In streaming mode, the process keeps listening until it is terminated. In static mode, the available finite data can be ingested and processed, after which the computation can finish. The same graph vocabulary covers both cases, but execution supplies different assumptions about whether more data will arrive.
The difference can be seen in what the output means. A static run is naturally read as the answer for the finite input available at launch. A streaming run produces an answer for the current frontier of observed data and remains prepared to revise it. The transformation may be written in the same style, but a caller must decide whether the destination should receive a one-time table, a sequence of changes, or a connector-mediated current state. “Run the same pipeline” therefore describes the shared logic, not a promise that a consumer can ignore the execution mode.
Execution also gives otherwise abstract relationships a lifecycle. The connector begins observing data; the input table becomes populated; derived tables become meaningful as their dependencies receive data; and the output connector starts publishing what the graph produces. In a live computation, those stages may overlap and repeat as new commits arrive. The graph remains the same logical object, but its tables move through successive states. That is the foundation needed for the incremental update story: a later change can be compared with the state already held by the graph rather than treated as an entirely new program.
This separation also clarifies what Pathway's Python API does and does not promise. Python is the language in which the developer expresses schemas, table relationships, and transformations. The framework's documentation describes a Rust engine that maintains table contents and performs many transformations below that interface. That division supports an accessible way to describe a system whose runtime must manage state and updates. It does not mean that arbitrary external work—such as a slow network request, a user-defined function with side effects, or a model call—automatically has the same behavior as a built-in table operator.
The dataflow model is strongest when each boundary has a clear owner. The connector knows how to observe and decode the source. The table holds the framework's current representation. The transformation declares a derived relationship. The runtime manages propagation. The output connector adapts the derived state to a destination. When those responsibilities are blurred, debugging becomes harder: an apparently stale result might arise from source detection, schema interpretation, transformation logic, or the sink's handling of updates.
A useful diagnostic follows from that ownership model. If no rows appear, the first question is whether the input connector has observed and committed data that matches the schema. If the input table looks right but a derived table is empty, the join or filter relationship deserves attention. If the derived table changes but the external system does not, the output connector or destination contract may be responsible. These are not exhaustive troubleshooting rules, but they show why the graph is more than a diagram: it gives each visible symptom a boundary at which its cause can be investigated.
The model also keeps failure from being hidden behind a single word such as “streaming.” A source can be live while a transformation is logically simple; a transformation can be stateful while its source is a finite file; an output can be correct inside the graph but unusable at a destination that does not understand revisions. Treating the pipeline as connected responsibilities makes those distinctions visible before production traffic turns them into surprises.
That is why a connector catalog is less useful than a connector question. Before selecting one, ask whether it can represent the source's changes, whether its mode matches the intended execution, and what happens when the process restarts. Before selecting an output, ask whether the destination accepts revisions or requires the connector to translate them. The framework offers many integrations, but it cannot decide those application-specific contracts on the reader's behalf.
The result of this construction is a pipeline that can be inspected conceptually before it is run. A table is not the source itself, a transformation is not an imperative rewrite of its input, and pw.run() is not a decorative final line. Together they establish a graph in which state can be maintained and changed. The next difficulty is to follow one such change: when a measurement or threshold moves, which parts of the graph change, which parts remain untouched, and how does the destination learn that the previous answer is no longer current?