Overview

Attention Is All You Need tested the Transformer under specified WMT 2014 translation conditions and in English constituency parsing, then examined component variations and selected attention maps. The larger English–German model reported 28.4 BLEU and the paper claimed more than two points over earlier reported systems; parsing reached 91.3 F1 on WSJ-only data and 92.7 F1 in a semi-supervised setting. These results support a strong architectural proposal and transfer to a selected task, not universal superiority, isolated causal credit, automatic interpretability, or a current leaderboard claim. The paper’s exact English–French score is internally inconsistent.

Learning outcomes

The WMT 2014 English–German result—28.4 BLEU for the larger model—belongs to the paper’s comparison and conditions.
Parsing results of 91.3 F1 and 92.7 F1 test different WSJ-only and semi-supervised data regimes.
Ablations and visualizations support component sensitivity and possible structured attention behavior, but do not isolate causation or prove interpretability.

Keywords

TranslationParsingBLEUF1AblationInterpretability

What the Paper Actually Demonstrated

The Transformer’s architecture makes a claim about how sequence computation should be organized, but an architecture becomes evidence only through a specified experiment. This chapter examines what Attention Is All You Need actually measured: the data and training conditions behind its translation results, the paper’s reported comparison with earlier systems, and the additional evidence from parsing, ablations, and attention visualizations. The aim is to separate a measured outcome from the broader conclusion someone might be tempted to draw from it.

That separation is especially important here because the paper combines several kinds of evidence. Translation benchmarks test output quality and estimated training cost under particular data and hardware conditions. Parsing tests whether the same broad design can work on a different sequence task. Ablations test whether selected components matter within one model family. Visualizations offer qualitative clues about what learned attention patterns may represent. These forms of evidence answer related questions, but they do not carry the same strength or scope.

Translation results belong to their conditions

The translation experiments use two WMT 2014 tasks with different data regimes. The English–German dataset contains about 4.5 million sentence pairs and uses byte-pair encoding with a shared source–target vocabulary of about 37,000 tokens. The English–French dataset is much larger, with 36 million sentences and a 32,000-word-piece vocabulary. This difference matters before any score is read: the two language directions do not provide the same amount of training data or exactly the same tokenization setup.

The paper groups sentence pairs by approximate sequence length and forms batches containing roughly 25,000 source tokens and 25,000 target tokens. The models are trained on one machine with eight NVIDIA P100 GPUs. The base models run for 100,000 steps, taking about 12 hours, while the larger models run for 300,000 steps, taking about 3.5 days. The training recipe uses Adam with the paper’s stated coefficients, a learning-rate warmup of 4,000 steps followed by an inverse-square-root schedule, dropout, and label smoothing. These are not incidental settings when the claim includes both quality and training time. They are part of the conditions that produced the reported outcome.

Evaluation also includes choices that shape the result. The authors average several recent checkpoints, use beam search, apply a length penalty, and set an output-length limit while allowing early termination. Those decisions do not invalidate the comparison; they identify what was compared. A claim such as “the Transformer trained faster” should therefore be read as shorthand for “the described configuration, trained with this regime and evaluated under these settings, achieved the reported result in the paper’s comparison.”

The clearest result is on English–German. The larger Transformer reports 28.4 BLEU and the paper says this exceeds the best previously reported models, including ensembles, by more than two BLEU points.

Published by DeepMox

FAQ

What did the Transformer achieve on WMT 2014 English–German translation?

In the paper’s WMT 2014 English–German comparison, the larger Transformer reported 28.4 BLEU and was said to exceed earlier reported models, including ensembles, by more than two points. The result is tied to the paper’s model, training, hardware, and evaluation conditions.

What did the Transformer parsing experiment show?

The Transformer reached 91.3 F1 in the WSJ-only setting and 92.7 F1 in the semi-supervised setting. These are different data regimes and show transfer to the selected English constituency-parsing task, not universal parsing superiority.

Can attention visualizations explain a Transformer’s predictions?

Attention visualizations cannot by themselves establish a faithful explanation of a Transformer’s prediction. The paper’s selected examples suggest structured behavior in some heads, but do not prove exclusive reliance or a fixed role across inputs.

Previous10 minTransformer Parallelism and Attention Cost

Next10 minTransformer Use Cases and Limits

Transformer Evidence: Results and Limits

What the Paper Actually Demonstrated

Translation results belong to their conditions

FAQ