How Attention Turns Relationships into Computation
In the Transformer, attention is not an extra spotlight placed on a representation that some other mechanism has already built. It is the operation that builds each contextual representation by letting one position weigh information from other positions. This chapter explains that operation—from queries, keys, and values through scaling and multi-head projections—so you can follow the calculation and predict which information is allowed to cross each part of the model.
The useful question is not “which word is the model looking at?” in isolation. It is: given the current position and the sequence around it, how does the model calculate a mixture of information that is relevant to that position? The answer has a small number of precise steps. A position produces a query, candidate positions offer keys and values, compatibility scores become normalized weights, and those weights determine the information that is mixed into the output.
From a lookup question to a weighted representation
Start with one query and a collection of key-value pairs. The query represents the position that needs context. Each key is a representation against which that need can be compared, and each value carries the information that can be contributed if the corresponding key is a good match. These names are functional roles, not three different kinds of token. In a self-attention layer, they can all be derived from the same sequence representation; what differs is how the model projects and uses them.
Imagine a current position in a sentence whose interpretation depends on something farther away. The position does not copy the whole sentence into its output. It computes a compatibility score between its query and the key associated with each candidate position. A high score gives that candidate more influence; a low score gives it less. The output is a weighted sum of the candidate values. Attention therefore changes the representation at the current position by mixing information selected through learned compatibility, rather than by carrying one fixed state through every intervening position.
For a set of queries, keys, and values packed into matrices, the Transformer uses scaled dot-product attention:
Attention(Q, K, V) = softmax(QKᵀ / √dₖ)V
The notation describes a sequence, not a mysterious single operation. First, QKᵀ computes the dot product between every query and every key. The result is a matrix of compatibility scores: each row corresponds to a query, and each column corresponds to a candidate key. Next, every score is divided by the square root of the key dimension, dₖ. The softmax then converts the scaled scores in each row into weights. Finally, those weights are multiplied by V, producing a weighted combination of values for every query at once.
The matrix form matters because it makes the operation suitable for parallel computation. Instead of asking one position, then another, then another, the model can form the score matrix through matrix multiplication and process a collection of queries together. The direct access is also what changes the path between positions. A query can compare itself with a distant key in the same attention operation; it does not need a recurrent hidden state to carry the signal through all the positions in between.
The dot product is a practical choice as well as a conceptual one. The paper compares dot-product attention with additive attention, in which a small feed-forward network computes compatibility. The two have similar theoretical complexity, but the paper notes that dot products are faster and more space-efficient in practice because modern hardware provides highly optimized matrix-multiplication operations. That advantage belongs to the way the calculation is organized, not to the word “attention” by itself.
The scale factor has a more specific job. If the components of a query and a key are independent random variables with mean zero and variance one, the paper observes that their dot product has variance proportional to dₖ. As dₖ becomes larger, unscaled dot products can therefore grow in magnitude. The softmax may then become extremely peaked: one score receives almost all the weight while the others receive almost none. In that region, the gradients can become very small, making learning difficult. Dividing by √dₖ keeps the scores in a more manageable range under the paper’s stated variance intuition.
This explanation should be read with the same precision as the equation. The scaling factor is not a cosmetic normalization, and the paper gives a concrete reason for it. But the authors present the gradient argument as their suspicion about why unscaled dot products degrade for larger dimensions, supported by the variance calculation. It is not a complete theory of every optimization effect in every attention implementation.
The weighted sum also clarifies what attention does not do. It does not retrieve a single objectively correct token, and the largest attention weight is not automatically a complete explanation of the model’s prediction. The operation produces a learned mixture. Several positions may contribute, and the useful combination can differ with the query, the layer, and the learned projections. Attention is best understood here as a mechanism for constructing a context-dependent representation, not as a guaranteed human-readable reasoning trace.
Several relational views, three information boundaries
A single attention calculation gives each query one weighted mixture. The Transformer’s multi-head attention creates several such mixtures in parallel, but it does not simply repeat the same full-dimensional calculation eight times. It first projects the queries, keys, and values into different learned lower-dimensional spaces. Each projected set runs its own attention operation. The resulting outputs are concatenated and passed through another learned projection:
MultiHead(Q, K, V) = Concat(head₁, …, headₕ)Wᴼ
where each head is an attention operation on its own projected Q, K, and V. In the paper’s base configuration, there are eight heads, with key and value dimensions of 64 and a model dimension of 512. Because each head uses a smaller dimension, the total computation is similar to a single full-dimensional attention operation under that configuration.
The reason for this arrangement is representational. A single head must compress all of its relational judgments into one attention-weighted view. Averaging can blur distinctions that matter. Separate heads can learn to compare positions in different representation subspaces and can preserve different patterns of interaction before their outputs are combined. One head might be more useful for a local relationship in one context and another for a long-distance relationship or a different feature of the representations. The roles are learned; they are not assigned by a programmer in advance.
The paper’s component variation supports the value of choosing the number of heads carefully, but it does not turn eight into a universal rule. In its English–German development-set experiments, single-head attention was 0.9 BLEU below the best tested setting, while quality also fell when the number of heads became too large. Those results show that the division of representation space affects the tested model. They do not show that every task should use exactly eight heads or that a head has a stable, human-readable function in all settings.
The diagram separates two levels that are easy to collapse. The upper path is the calculation performed by one or more heads. The lower paths show where the calculation is placed in the Transformer and what information each placement can use. The same basic attention operation acquires a different meaning from the source of its queries, keys, and values and from the mask applied to its scores.
In encoder self-attention, the queries, keys, and values all come from the output of the previous encoder layer. Every source position can therefore compare its query with the keys for all source positions and mix their values. The model can build a representation of a position that includes context from the entire input sequence in that layer. Nothing in this operation requires a left-to-right order of calculation, although positional information must be supplied elsewhere so that the model can distinguish different arrangements of the same symbols.
Decoder self-attention uses a different information boundary. The queries, keys, and values come from the decoder sequence, but the attention scores for later target positions are masked before the softmax. At the point where the model predicts position i, it cannot use the target symbols that would be generated after i. The output embeddings are offset by one position, so the prediction at a position can depend only on known earlier outputs. The mask preserves the autoregressive rule while still allowing the known positions in a training example to be processed together.
The mask is not a minor implementation detail. Without it, a training calculation could look at the correct future target and produce an unrealistically easy prediction. That would break the information boundary that inference must obey, because those future symbols do not exist when the model is generating a new sequence. The mask makes the training computation parallelizable without allowing information to cross from the future into the prediction.
Encoder-decoder attention connects the two sequences. Here, the queries come from the previous decoder layer, while the keys and values come from the encoder output. Every decoder position can use the full encoded input sequence, which gives the output side a way to retrieve source information while it constructs a target representation. This is a different relationship from encoder self-attention: the encoder relates source positions to source positions, while cross-attention relates a target-side need to source-side representations.
These three placements can coexist in one decoder layer. A target position first develops context from the permitted target prefix through masked self-attention. It then uses that decoder-side state as a query against the encoder’s keys and values. The model does not have to choose between understanding the target prefix and consulting the input; the stack composes both relationships through separate sublayers. In the encoder, self-attention builds source context. In the decoder, masking controls target-side causality and cross-attention opens a source-to-target route.
The distinction also explains why “global attention” should not be paraphrased as “every position sees everything.” Encoder positions can see all source positions in self-attention. A decoder position can see all source positions through encoder-decoder attention, but it cannot see future target outputs through masked self-attention. The permitted connections depend on which sequence is being processed and on whether the relationship is source-to-source, target-to-target, or target-to-source.
A final boundary follows from the operation itself. Multi-head attention can create several relational views, but it does not automatically tell us what each view means. The paper’s visual examples suggest that some heads behave in ways related to syntax, semantics, long-distance dependencies, or anaphora resolution. That is useful evidence that the learned weights can exhibit structured behavior. It is not proof that every attention distribution is a faithful explanation of the model’s decision, nor that a human can assign one permanent role to each head.
Attention therefore supplies a precise answer to the question of how one position can use another position’s information: compare projected queries and keys, normalize the compatibilities, and mix the corresponding values. Multi-head attention allows several learned comparison spaces, while the three placements define who may communicate with whom. That operation changes the geometry of sequence computation, but it is not yet the whole model. The next architectural question is how repeated attention is combined with nonlinear transformation, residual paths, token interfaces, and explicit information about order.