Overview
This series explains how the Transformer, introduced in Attention Is All You Need, reorganizes sequence computation around globally addressable self-attention rather than sequence-aligned recurrence or convolution. It moves from the serial bottleneck and attention’s query-key-value operation through the encoder-decoder stack, positional information, parallelism, quadratic interaction cost, and autoregressive generation. The paper’s WMT 2014 translation and English constituency-parsing experiments provide evidence under specified data, model, training, hardware, and evaluation conditions; they do not establish universal superiority, automatic interpretability, or a solution for very long inputs and sequential output. The series offers a mechanism-first way to judge when this exchange is worthwhile.
Learning outcomes
- The Transformer replaces recurrence and convolution as the main route for position-to-position interaction.
- Global self-attention offers short paths and parallelizable representation work, but its pairwise cost grows quadratically with sequence length.
- Positional encodings supply explicit order, while decoder masking preserves causality and autoregressive generation remains sequential.
- The paper supports a conditional architectural advantage under its reported tasks and settings, not universal superiority or automatic interpretability.
Keywords
Preview every chapter · Unlock full text, audio, and video from chapter 4
PlansFAQ
What did the Transformer change compared with recurrent sequence models?
The Transformer changed the main route between positions from recurrent state updates to attention-centered representations. This shortens global paths and enables more parallelizable known-position computation, but it does not remove quadratic attention cost or sequential decoding.
How does self-attention let one token use information from another?
Self-attention compares projected queries with keys, turns the scores into weights, and mixes the corresponding values. Multi-head attention performs this in several learned subspaces, while masks determine which positions may communicate.
When is the Transformer more efficient than recurrence?
In the paper’s comparison, self-attention is faster than recurrence when sequence length is smaller than representation dimension for the sentence representations considered. Actual wall-clock results still depend on hardware, implementation, memory movement, batching, and constants.
What are the main limitations of the Transformer?
The main limitations described are quadratic global-interaction cost, the need for explicit positional information, and sequential autoregressive generation. Restricted attention can lower interaction cost, but it lengthens paths between distant positions and was proposed rather than tested in the paper.