Why BERT Made Context Transferable
This series studies BERT, the language model introduced in BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, and asks a practical question: how can one pretrained representation support so many different language-understanding tasks? The focus here is the problem BERT is designed to solve, not a general tour of neural networks. By the end of this opening, the useful distinction is clear: BERT is not merely a larger model, but a way to learn rich context once and reuse it through relatively small task-specific adaptations.
That distinction matters because language tasks do not all ask for the same kind of answer. Some classify an entire sentence or compare two sentences. Others must identify a particular token, entity, or answer span inside a passage. BERT's central claim is that a common bidirectional encoder can provide the representations beneath all of these decisions. To see why that claim was important, we first need to understand the design gap it addresses.
Why one-direction context creates a design gap
A language model learns by making predictions from text. In a left-to-right model, the representation at a position is constrained to the words that came before it. That arrangement is natural for generating the next word: the model must not read the future. It is less natural for understanding a completed piece of text. To decide what a word means in a passage, the words after it can be as informative as the words before it.
Consider a question-answering system. The answer may be a short span in a long passage, and its interpretation depends on the question as well as on the surrounding passage. A representation that can consult only the text to its left has an avoidable blind spot. Sentence-pair tasks create a related problem at a larger scale. Natural-language inference asks whether one sentence entails, contradicts, or is neutral with respect to another. Paraphrase detection asks whether two questions express the same meaning. The task is not just to understand each sentence separately; it is to model their relationship.
Before BERT, researchers had ways to make pretrained representations useful. ELMo combined features from independently trained left-to-right and right-to-left recurrent language models. OpenAI GPT used a left-to-right Transformer and fine-tuned it for downstream tasks. Other systems built task-specific machinery around learned representations, such as separate encoders followed by a cross-attention mechanism for two pieces of text. These approaches were productive, but they left a tension in place: the model used for pretraining and the structure needed for the final task were not always aligned.
BERT's proposed answer is deep bidirectionality. Its Transformer encoder allows each token to attend to the rest of the input in both directions, and it does so at every layer. This is more than placing two directional summaries side by side at the end. The contextual representation built at one layer can influence the next layer, so the interaction between left and right context is part of the representation's construction all the way through the encoder.
The word deep is important here. Suppose the phrase “the bank” appears in a passage. The word after it may help decide whether the passage concerns a riverbank or a financial institution. In BERT's design, that later evidence can participate in the representation of “bank,” while the representation is refined through multiple layers of self-attention. The result is intended for understanding an input that already exists, not for generating it one token at a time.
BERT also tries to reduce the amount of custom architecture required after pretraining. Rather than building a separate representational strategy for every task, it packs the relevant text into one sequence and lets the same encoder process it. The downstream system still needs to say what output it wants—a class label, a token label, or the start and end of an answer—but the central language representation is shared. That is the beginning of context becoming transferable.
The change BERT makes: learn once, adapt many ways
BERT's design divides the work into two stages. During pretraining, the model reads a large collection of unlabeled text and adjusts all of its parameters through self-supervised objectives. No person needs to annotate every sentence with a sentiment, entailment label, or answer span. The text itself supplies the learning signal. During fine-tuning, a copy of the pretrained model is initialized with those learned parameters and trained on labeled examples for one downstream task.
This division changes what “task-specific” means. A sentiment classifier may add a classification layer, while a question-answering system may add parameters that score possible start and end positions. The output interface differs because the decisions differ. But the difficult general-purpose work of building contextual token representations has already been performed by the shared encoder. Fine-tuning updates the pretrained parameters as well as the new output parameters, so the model can adjust its general knowledge to the task instead of treating the pretrained representation as permanently fixed.
BERT's input format helps make this reuse possible. A single text can be placed into a sequence beginning with a special classification token. A pair of text spans can be placed into the same kind of sequence, separated by a special separator token. Learned segment information indicates which span a token belongs to, while position information preserves order. At the output, the final representation of the classification token can summarize the sequence for a whole-input decision; the representation of each token remains available when the answer must be localized.
That common interface is the practical bridge between architecture and transfer. The encoder does not need to know in advance whether the sequence will become a premise-hypothesis pair, a question and a passage, or a single sentence for sentiment analysis. It produces contextual representations for the packed input. A small task-specific layer then interprets those representations in the form required by the labeled dataset.
The paper's question-answering example makes the division of labor especially visible. The question and passage are treated as one pair. BERT processes them together, so the passage token representations can incorporate information from the question. The downstream layer then scores where an answer begins and ends. The encoder supplies context; the task head supplies the decision rule. This is a narrower and more reusable boundary than designing a new reading-comprehension architecture from scratch.
The same principle applies at different levels of granularity. A sentence-level task can use one aggregate representation. A sentence-pair task can use that representation to classify a relationship. A token-level task can attach predictions to individual contextual vectors. The machinery that turns representations into labels is not identical across tasks, but the representation-producing part is deliberately kept stable.
This is why calling BERT “a bigger language model” misses the paper's main contribution. Scale matters—the authors compare a 110-million-parameter BERTBASE with a 340-million-parameter BERTLARGE—but scale is not the whole explanation. The design combines a bidirectional encoder, objectives intended to learn from unlabeled text, and a fine-tuning recipe that transfers the resulting parameters. The intended payoff is not only a stronger score on one benchmark. It is less reinvention at the boundary between general language knowledge and a particular task.
A strong paper claim, not a universal verdict
The paper reports unusually broad results for its time. It evaluates BERT on eleven natural-language-processing tasks and describes gains across sentence-level and token-level problems. Its headline abstract reports a GLUE score of 80.5, MultiNLI accuracy of 86.7%, SQuAD v1.1 test F1 of 93.2, and SQuAD v2.0 test F1 of 83.1 for the stated systems and evaluation setups. Those numbers matter because they connect the architectural idea to varied tasks: classification, inference, and question answering do not all reward the same narrow capability.
But the numbers need to remain attached to their conditions. The GLUE calculation in the paper excludes WNLI because the authors describe problems with that dataset. The strongest SQuAD v1.1 result involves an ensemble and, in one setup, additional TriviaQA pre-fine-tuning. Different tasks use different metrics, and comparisons include systems with different architectures and training regimes. The results support the claim that BERT's pretraining-and-adaptation recipe transferred broadly in the paper's 2018 evaluation; they do not turn one score into a universal measure of language understanding.
The same caution applies to the word simple. Fine-tuning can be relatively inexpensive once a checkpoint exists, and a task may need only a small newly initialized output layer. Pretraining is another matter. The reported model learned from BooksCorpus and English Wikipedia, and the appendix describes a large training run using substantial accelerator resources. BERT moves complexity away from many downstream architectures, but it does not remove complexity from the system. It concentrates more of it in data, pretraining, sequence length, and adaptation choices.
That trade is the central subject of this series. The coming explanation will examine how BERT's encoder represents a single text and a pair of texts, how masked language modeling makes bidirectional learning possible without letting a token simply reveal its own answer, and why next sentence prediction was added. It will then follow the resulting checkpoint into concrete tasks and test the paper's causal story with its ablations and limitations.
For now, the most important conclusion is modest but powerful: BERT makes context reusable by separating the learning of general bidirectional representations from the final decision a task needs. The promise depends on the structure of the encoder and the information it is given. So the next question is not whether BERT is large, but what it actually sees when it reads a sequence—and how one input format can preserve both a text and the relationship between two texts.