The Deeper Change: A New Geometry for Sequence Computation

This series has followed the Transformer from the constraint that motivated it to the attention operation, the encoder–decoder stack, the computational trade-off, and the experiments that made the proposal consequential. The final question is what, precisely, changed when sequence-aligned recurrent layers were replaced by multi-headed self-attention. The answer should give you more than a historical slogan: it should let you trace the new information paths, judge the parallelism claim, and carry the evidence limits with the conclusion.

The durable change is a change in geometry. A recurrent model makes a chain of state updates the organizing route through a sequence. The Transformer makes globally addressable relationships the organizing route for its representation layers. That opens shorter paths and more parallelizable computation, but it also creates a quadratic global-interaction cost, requires explicit positional information, and leaves autoregressive generation sequential. The architecture is powerful because of this exchange, not because it abolishes the constraints that make sequence computation difficult.

flowchart LR Need["Long-range or structured sequence need"] --> Choice["Transformer design"] Choice --> Mix["Global multi-head attention"] Mix --> Paths["Short relational paths"] Mix --> Parallel["Parallelizable known-position computation"] Choice --> Position["Explicit positional encoding"] Choice --> Causal["Causal decoder output control"] Paths --> Evidence["Reported translation and parsing evidence"] Parallel --> Evidence Evidence --> Conditions["Task, data, model, and evaluation conditions"] Conditions --> Judgment{"What does the evidence support?"} Choice --> Quadratic["Quadratic global interaction"] Causal --> Sequential["Sequential next-token generation"] Quadratic --> Judgment Sequential --> Judgment Judgment --> Conditional["Conditional architectural value"] Judgment --> Boundary["No universal conclusion"]

The model in the diagram is deliberately conditional. A task need can make global relationships relevant; attention can make those relationships directly available; the stack can preserve order and causal output control; and experiments can show whether the exchange helped under stated conditions. The same design carries costs that feed back into the judgment. A conclusion that omits either branch is not a synthesis of the Transformer’s proposal.

A different geometry, not an attention add-on

The Transformer’s central move is easy to misstate as “use attention to improve a recurrent network.” The paper proposes something stronger: dispense with recurrence and convolution in the sequence-processing architecture and use attention mechanisms to compute input and output representations. The encoder–decoder organization remains, and the decoder remains autoregressive, but the main route between positions changes.

In the recurrent geometry, a hidden state at one position depends on the state before it. Distance is therefore expressed as a chain of updates. In the Transformer’s unrestricted self-attention geometry, a position can compare with every other position in the same layer. The query-key compatibility operation creates a direct opportunity for a distant value to influence the current representation. Multi-head projections provide several learned comparison spaces, so the model need not compress every relational judgment into one averaged view.

That direct access is not the same as understanding. Attention calculates a weighted mixture; it does not guarantee that the highest-weight position is the complete reason for a prediction. The paper’s visual examples suggest that some heads follow long-distance dependencies or exhibit behavior related to syntax and semantics, but those are selected qualitative observations. The geometry makes relationships available and inspectable. The learned parameters still determine which relationships are used and how their information is combined.

The complete stack is what turns this operation into a sequence model. Token embeddings provide vectors for discrete symbols. Positional encodings provide the order that recurrence or convolution would otherwise help express. Encoder layers alternate cross-position attention with position-wise feed-forward transformation, using residual paths and normalization to compose the updates.