How BERT Learns Before It Answers
BERT's bidirectional encoder is useful only if training teaches it how to use information from both sides of an input. The specific problem here is that a normal bidirectional language model would be allowed to see the token it is supposed to predict, while a left-to-right model cannot use future context at all. BERT's two pretraining objectives—masked language modeling and next sentence prediction—are designed to work around those constraints, and the paper's data and training setup show what that learning costs.
Masked prediction hides the shortcut
A conventional language model predicts the next token from the tokens that came before it. That rule is restrictive for understanding, but it prevents a trivial answer: when the model predicts the next token, the answer is not already in its input. If we simply allowed a bidirectional model to look at the entire sentence while asking it to predict every token, the target would be visible. The model could copy or recognize the answer rather than learn how surrounding context constrains it.
BERT changes the prediction task by hiding only some of the input tokens. It randomly selects 15% of WordPiece positions and asks the model to recover the original vocabulary item from the remaining context. The model does not reconstruct the entire sequence. It receives the text with selected positions altered and is penalized for the predictions at those positions. Every selected target is therefore a missing piece, while the unselected text can be read from both directions.
A short example makes the information flow concrete. Suppose the original text is “my dog is hairy.” If hairy is selected, BERT may receive “my dog is [MASK]” and must infer the original token. The word before the blank supplies grammatical and semantic context, but so does the complete structure of the sequence around the blank. In a longer passage, the useful evidence might appear many positions away. The target is hidden, yet the surrounding positions can interact through the bidirectional Transformer described earlier.
The masking procedure is more careful than replacing every selected position with the literal [MASK] symbol. Of the selected positions, the paper replaces 80% with [MASK], 10% with a random token, and leaves 10% unchanged. The model is still trained to predict the original token in all three cases.
Each part of that mixture addresses a different weakness. If every training target is visibly marked [MASK], the model can become dependent on a symbol that will not normally occur during fine-tuning. Random replacements make some inputs look like ordinary text while forcing the model to recover from a misleading token. Unchanged targets mean that the model sometimes has to predict a position that looks normal, discouraging it from treating the marker as the only signal that a prediction is required. The goal is to keep the representation useful for the observed input distribution while still creating a genuine prediction problem.
The procedure creates a trade-off. A left-to-right language model can calculate a prediction loss for every token in a sequence. BERT's masked language model calculates the prediction loss for only the selected 15%. The authors report that MLM converges somewhat more slowly than left-to-right training. That is a real efficiency cost, not a defect hidden by the terminology. At the same time, their ablation reports that the masked model begins to outperform the left-to-right model in absolute downstream accuracy almost immediately, and the paper treats the empirical improvement as worth the slower convergence.
This is the causal role of MLM in BERT's design: it removes the target from the input without removing the rest of the sequence. The encoder can use left and right context at every layer, but it must learn a relationship between the visible surroundings and the missing token. The resulting vectors are not just summaries of text copied from the input; they are shaped by a task that rewards contextual prediction.
The loss is applied only at selected positions, but the information path is wider than that list. A masked token's prediction depends on hidden states built from the sequence, and those hidden states are produced by shared layers. Unmasked tokens therefore provide context even when they are not themselves prediction targets. If several positions are selected in one example, each one is hidden or altered according to the masking procedure, so the model cannot assume that every surrounding position contains the original text. This makes the task closer to recovering meaning from a partially observed sequence than to filling a single blank with a memorized phrase.
The unchanged 10% is especially easy to misread. It does not mean the model is rewarded for copying a visible word without doing any work; the training label remains the original vocabulary item, and the representation must use the surrounding sequence to support the prediction. Likewise, a random replacement is not treated as a new truth. It is a corrupted observation whose original identity is still the target. In both cases, the model sees inputs that resemble ordinary text more closely than a sequence of obvious mask symbols would. That is why the masking rule belongs to the interface between pretraining and later use, not only to the data generator.
The paper's result is consequently a trade between signal density and contextual breadth. Predicting fewer positions gives each target access to a rich two-sided context, but it supplies fewer prediction losses per batch. BERT's authors report the slower convergence and then compare downstream accuracy rather than pretending the cost is absent. The design is persuasive only when both sides are kept in view: masked learning changes what the encoder can represent, and it also changes how much computation is required to reach a useful checkpoint.
The diagram shows two signals entering one parameter update, but the objectives ask different questions. Masked prediction asks what token belongs in a local or long-range context. Next sentence prediction asks whether one span has the relationship to another that the training example claims. The distinction matters because language understanding often depends on both what words mean in context and how pieces of text fit together.
Sentence pairs, documents, and the cost of the checkpoint
Masked token prediction does not directly require the model to distinguish a coherent continuation from an unrelated text span. BERT adds next sentence prediction, or NSP, to supply that missing text-pair signal. For each example, the first span is called A and the second span B. Half the time, B is the actual next span that follows A in the corpus and the example is labeled IsNext. Half the time, B is a random span and the example is labeled NotNext.
The final hidden representation of [CLS] feeds a binary classifier for this decision. The pretraining task is deliberately simple to generate from ordinary monolingual text, but it gives the encoder a reason to represent a relationship between two spans. That relationship is relevant to the downstream tasks BERT highlights: question answering combines a question with a passage, and natural-language inference combines a premise with a hypothesis.
NSP should not be mistaken for a complete theory of discourse. A random negative is a convenient contrast, not every way in which two sentences can be related. A pair can be topical but not adjacent, grammatically compatible but logically inconsistent, or adjacent without being useful evidence for a downstream judgment. The paper's claim is narrower: within its training and ablation setup, adding this objective improves selected downstream results. When the authors remove NSP from BERTBASE, the reported scores fall notably on QNLI, MNLI, and SQuAD 1.1.
The objectives depend on how text examples are assembled. BERT's pretraining corpus contains about 800 million words from BooksCorpus and 2.5 billion words from English Wikipedia. For Wikipedia, the authors keep text passages and discard lists, tables, and headers. More important than the names of the sources is the decision to preserve document-level continuity. The authors say a document-level corpus is critical because it allows the sampler to obtain long contiguous sequences; a corpus made from shuffled sentences would not supply the same kind of available context.
A training example consists of two spans whose combined length is at most 512 tokens. The spans are used for both objectives: token positions are selected for masking, while the pair receives the IsNext or NotNext label. This lets BERT practice two forms of contextual reasoning in one sequence. The segment markers introduced earlier identify A and B, but the encoder is still free to let information cross that boundary. NSP gives the [CLS] representation an explicit pretraining job at the sequence level, while MLM gives individual positions a prediction job.
There is an important qualification about what NSP teaches the [CLS] position. It trains that position to support the binary IsNext/NotNext decision during pretraining, but the paper explicitly warns that this does not automatically make [CLS] a generally meaningful sentence representation before fine-tuning. The vector becomes useful for a downstream classification task when the downstream objective reshapes it for that task. NSP supplies a relationship signal; it does not remove the need for task-specific learning.
The 50/50 construction also defines the strength and weakness of the signal. A true adjacent span and a random span are easy to generate, which makes the objective scalable over monolingual documents. But “not next” is a broad negative category. It does not tell the model whether a pair is contradictory, merely topical, or related at a nonadjacent point in a document. BERT's ablation can show that removing this particular objective hurts the paper's selected tasks; it cannot establish that this binary proxy captures every kind of discourse relation.
The scale of the reported pretraining explains why the paper's downstream simplicity should not be confused with a small overall system. The appendix describes batches of 256 sequences, one million training steps, and approximately 40 passes over the 3.3-billion-word corpus. To manage the cost of long attention spans, the authors use sequences of length 128 for 90% of the steps and length 512 for the remaining 10%. The longer sequences are disproportionately expensive because self-attention compares positions across the sequence, giving its cost a quadratic dependence on sequence length.
That schedule reflects a compromise rather than a minor tuning preference. Shorter sequences allow more examples to pass through the model during most of training. The final longer-sequence phase teaches the positional behavior needed for the maximum sequence length used in the experiments. The paper does not claim that this exact schedule is a universal recipe; it reports it as part of the setup that produced the studied checkpoints.
The hardware and runtime make the same point from another angle. In the authors' setup, BERTBASE used 16 TPU chips and BERTLARGE used 64 TPU chips, with each pretraining run taking four days. Those figures are not a universal bill for every BERT implementation, but they are important evidence about the system the paper actually evaluated. The representation that can later be reused across tasks is backed by a large corpus, many parameter updates, and substantial accelerator time.
The training process also has a termination condition that differs from downstream prediction. Pretraining ends when the shared parameters have been updated through the chosen schedule; it produces a checkpoint, not a label for one application. The checkpoint carries the contextual behavior learned from MLM and NSP. Fine-tuning later changes those parameters for a particular task, but the task is not what creates the general checkpoint in the first place.
Several visible mismatches remain. [MASK] is used during pretraining but does not appear in ordinary fine-tuning inputs. The 80/10/10 rule reduces that discrepancy, but it does not make the two distributions identical. Predicting only 15% of positions slows convergence relative to a left-to-right objective. Long sequences increase attention cost. And because the paper reports that larger models can be unstable on small GLUE datasets, more capacity can create an adaptation problem even after pretraining succeeds.
These boundaries sharpen rather than weaken BERT's central idea. The model learns a reusable representation by making the training input incomplete in a controlled way and by asking the [CLS] position to judge a relationship between spans. Document continuity supplies the context the objectives need, while the training schedule and hardware pay for processing it. The result is a checkpoint with contextual token and sequence representations. The next question is how that checkpoint becomes an answer span, a sentence label, or a token tag without rebuilding the encoder for each task.