The Problem the Transformer Reframed
The subject of this series is the Transformer, the encoder–decoder architecture introduced in Attention Is All You Need. This opening focuses on the problem the architecture was designed to change: how a sequence model lets one position use information from another. By the end of it, you should be able to explain why the Transformer was consequential, distinguish its proposal from simply adding attention to a recurrent network, and recognize the costs that must remain in view when its speed or quality is discussed.
A sequence model does more than read a list of symbols. It must turn an ordered input into an ordered output while preserving relationships that may be separated by many positions. In translation, a word near the end of a sentence may depend on a subject introduced near the beginning. In parsing, the output must reflect a structure whose parts are not necessarily adjacent. The central engineering question is therefore not only how much a model can represent, but how information is allowed to travel through the computation.
The serial bottleneck in sequence models
Before the Transformer, recurrent neural networks were a standard way to model sequences. Their basic arrangement is easy to describe: read the input at position 1, update a hidden state, use that state at position 2, update it again, and continue. The state at position t is a function of the previous state and the input at t. That dependency gives the model a natural sense of order, but it also makes the next computation wait for the previous one.
This waiting is not just a matter of software style. Within one training example, the model cannot calculate all recurrent positions independently if each position consumes the state produced by the one before it. Longer sequences therefore bring more sequential steps. More hardware or larger batches can help in other dimensions, but they do not remove the dependency chain inside that example. The paper identifies this inherent sequential computation as a fundamental constraint.
The same chain creates a second problem. Suppose a model must relate two distant positions. A signal from the first position has to pass through the intervening recurrent states before reaching the second. The issue is not that the model has no route; it is that the route grows with sequence distance. The paper treats the length of such paths as important because learning long-range dependencies becomes harder when forward and backward signals must traverse more steps.
Convolutional sequence models offer a different compromise. They can compute many positions in parallel, but a narrow convolution sees only a local neighborhood at each layer. To connect distant positions, the model needs a stack of layers. Depending on the convolution, the path can grow linearly or logarithmically with distance. Parallelism improves, but global relationships are still reached through an indirect route.
The Transformer begins with this tension rather than with a new vocabulary. It asks whether positions can exchange information directly, without requiring a recurrent state to carry it from left to right and without requiring a deep convolutional route to widen the field of view.
The contrast in the diagram is about dependency geometry. The recurrent branch makes each position a prerequisite for the next. The Transformer branch makes the source positions available to a global attention operation, allowing the representation work to be organized with much more parallelism and much shorter paths between positions. The final loop is deliberate: the described Transformer decoder still produces an output sequence autoregressively, using previously generated symbols to predict the next one.
The Transformer changes the route between positions
The paper’s proposal is not that attention is useful. Attention mechanisms were already being used with recurrent systems to help a model focus on relevant parts of an input. The stronger proposal is to make attention the central mechanism for computing the representations of both input and output sequences, dispensing with recurrent and convolutional layers in the sequence-processing architecture.
At a high level, the Transformer retains an encoder–decoder organization. The encoder turns the input sequence into a sequence of continuous representations. The decoder uses those representations while generating the output one element at a time. What changes is the route by which positions interact inside those parts. Instead of passing context through a sequence-aligned hidden state, the model uses self-attention so a position can draw from all positions in the relevant sequence. The detailed operation—queries, keys, values, projections, and masking—comes later. For now, the important point is that the architecture makes relationships between positions an explicit, directly computed object.
That change has two consequences that are easy to conflate. First, the encoder can consider all source positions together in a layer, rather than waiting for a recurrent scan to finish. Known target positions can also be handled in parallel during training under a mask that prevents them from seeing future answers. Second, the model does not thereby become a fully parallel generator. At inference time, the next target token is not known until the previous one has been produced, so the decoder’s autoregressive loop remains.
This is why “attention-only” needs careful interpretation. It describes the replacement of sequence-aligned recurrence and convolution as the main route for position-to-position interaction. It does not mean that the model contains nothing but an attention formula. The Transformer also uses feed-forward transformations, residual connections, normalization, embeddings, positional information, and a softmax output. Nor does it mean that the order of a sequence disappears. Because recurrence and convolution no longer supply order implicitly, the model must receive positional information in another way. Those parts are not side notes; they are what make the new route usable as a sequence model.
The deeper shift is from a chain to a field of possible relationships. In a recurrent model, distance is built into the number of updates separating two positions. In the Transformer’s unrestricted self-attention, every position can be connected to every other position within one layer. That does not guarantee that the model will learn the right relationship, and it does not make computation free. It changes the problem the architecture presents to optimization and hardware: global interaction is available immediately, while the cost of forming all those interactions grows with the square of sequence length.
The paper evaluates this shift using more than one measure. It compares the amount of computation per layer, the minimum number of sequential operations, and the maximum path length between positions. This matters because a model can be parallel in one sense while still forcing information through long routes in another. The Transformer’s case rests on the combination: fewer sequential operations and short paths for global relationships, traded against the cost of broad pairwise attention.
A stronger claim than “attention added to an RNN”
The distinction between an attention-augmented recurrent model and the Transformer is central to understanding what changed. In the former, recurrence still owns the main sequence computation. Attention provides an additional way to retrieve or align information, but the hidden-state chain remains the organizing structure. In the Transformer, the architecture is reorganized around attention-centered encoder and decoder stacks. The paper presents this as a replacement of the recurrent layers ordinarily used in sequence transduction, not as a small attachment to them.
That replacement also changes what a designer must account for. A recurrent model gets order from its update rule, whereas the Transformer must add positional encodings. A recurrent decoder advances by its state transition, whereas the Transformer decoder uses masked self-attention to preserve the rule that a prediction may depend only on earlier known outputs. An encoder–decoder attention mechanism can still connect output positions to the encoded input, but the source representation itself is produced through self-attention rather than a recurrent sweep.
The practical significance appears in the paper’s experiments, but the result must be read under its conditions. On the WMT 2014 English–German translation task, the paper reports a score of 28.4 BLEU for its larger Transformer and says that it exceeds earlier reported models, including ensembles, by more than two BLEU points. It also reports successful English constituency parsing, suggesting that the design was not limited to one translation benchmark. These are important demonstrations of what the authors built and measured, not proof that every task or hardware environment will reward the same architecture.
Even the paper’s numbers illustrate why evidence and mechanism belong together. Its abstract and table list one English–French score, while the prose describing the result gives another. That inconsistency does not erase the broader evidence, but it does put a boundary around how confidently an exact figure should be repeated. A sound account of the Transformer therefore asks three questions at once: what relationship did the architecture make easier to compute, under what conditions did that help, and what costs or uncertainties remain attached to the claim?
The main cost is not hidden. The paper gives unrestricted self-attention a per-layer complexity that grows quadratically with sequence length. For ordinary sentence representations, the authors argue that the trade can favor self-attention over recurrence, but that comparison depends on the representation dimension and the task regime. For very long inputs, they identify restricted attention as a possible direction, while acknowledging that restricting the neighborhood lengthens the paths between distant positions. And for output generation, the causal loop remains sequential. The architecture changes the location and shape of the bottleneck; it does not abolish bottlenecks.
That is the problem the Transformer reframed. Sequence computation no longer has to be organized as a relay in which every position waits for the previous state. Positions can be related globally, and the resulting representation work can be organized for substantially more parallel execution. But global access carries a pairwise cost, order still needs to be represented, and generation still has a temporal boundary. The next step is to open the attention operation itself and see how a position decides which other information to use.