The Input BERT Learns to Read Both Ways
In BERT, “bidirectional” is not a decorative adjective; it describes the path by which information reaches each representation. This chapter focuses on that path and on the shared input format that lets BERT process a single sentence, a sentence pair, or a question and passage. Once those boundaries are clear, the [CLS] vector and the individual token vectors stop looking like special symbols to memorize and become the two outputs a downstream task can actually use.
Bidirectionality lives in every layer
BERT is built from a Transformer encoder. The important point for understanding this model is not the full mathematics of attention, but the direction of information flow. At a given layer, each token has a vector—a numerical representation—and self-attention lets that token draw information from other positions in the same input. BERT does not impose the left-only restriction used by a left-to-right language model. A token can therefore incorporate evidence from positions before and after it, and the resulting vector is passed into the next layer for further interaction.
That makes “both ways” a property of the whole representation-building process. Suppose an input contains the phrase “the bank was flooded.” The word flooded is to the right of bank, but it is useful evidence for interpreting bank. In a left-context-only model, the representation at bank cannot directly use that later word at the time it is built. In BERT, the relationship can be represented through self-attention, and later layers can refine it alongside information from the rest of the sequence. The model is not being asked to generate the sentence. It is being asked to understand the text that is already present.
This distinction explains why BERT's encoder is especially relevant to tasks that inspect an entire input. In question answering, a passage token can use information from the question when its representation is formed. In natural-language inference, the representation of a premise-hypothesis pair can reflect interactions between the two spans. The relevant context is not merely the previous word; it is whatever other position in the packed input helps resolve the task.
The paper contrasts this design with two nearby approaches. OpenAI GPT uses a left-to-right Transformer, so its attention is constrained to earlier positions. ELMo obtains contextual features by concatenating outputs from independently trained left-to-right and right-to-left recurrent language models. Both directions are present in ELMo's final feature, but they are not jointly conditioned on one another through every layer of one encoder. BERT's claim is narrower and more specific: the two-sided context is fused inside a deep stack of Transformer blocks.
The difference is not that concatenation is incapable of carrying useful information. ELMo's features were effective for several tasks. The difference is where the interaction happens and what the downstream model must do with it. With separately trained directional models, the final task architecture receives a combination of features. With BERT, a token's representation is built by repeated layers that can relate it to the rest of the same input. That gives the downstream task a more directly contextualized object to work with.
A useful way to think about the encoder is as a series of representational boundaries. At the input boundary, each position has information about a token, its segment, and its location. At each Transformer layer, self-attention allows information to cross between positions without a left-to-right barrier. At the output boundary, BERT exposes both a representation intended to summarize the sequence and representations that remain tied to individual positions. The encoder owns the contextual transformation; a later task-specific layer owns the final decision.
For a pair of texts, this boundary is also where BERT changes the order of operations. A question token and a passage token enter the same sequence before either one has been reduced to a separate summary. Self-attention can therefore pass information from the question into the passage representations and from the passage back into the question representations. The segment embeddings identify the source span, but they do not create a wall between the spans. The distinction is useful: the model preserves the difference between A and B while allowing their meanings to interact.
This is why BERT's design can replace a more elaborate two-stage arrangement for the tasks described in the paper. In an arrangement with independent encoders, each text might first receive its own representation and a later component would compare them. BERT instead concatenates the spans and lets the Transformer perform the interaction inside the shared encoding process. The output layer still decides what relationship or answer to predict, but it receives representations that have already had access to both sides of the input. The encoder is not merely storing two texts next to each other; it is the place where their contextual relationship can be formed.
The word “direction” also needs care. BERT does not run one reader from the left and another from the right and then merge their final states. Direction here refers to an attention constraint: a left-to-right model masks future positions, while BERT's encoder does not impose that causal mask during its contextual computation. Its token vectors are updated through layers of shared processing, with position information preserving order and attention determining which other positions can contribute. That is the mechanism behind the paper's distinction between a deep bidirectional model and a shallow combination of directional features.
This architecture also clarifies what BERT is not claiming. A bidirectional encoder is well suited to understanding a completed sequence, but the paper does not present it as a left-to-right text generator. Nor does the presence of a common encoder mean that every task has the same output problem. The encoder makes a reusable representation available; a task still determines whether it needs a class, a relationship judgment, a token label, or an answer span.
Capacity matters, but capacity is not the complete explanation. The paper reports two principal configurations. BERTBASE has 12 Transformer layers, a hidden size of 768, 12 attention heads, and about 110 million parameters. BERTLARGE has 24 layers, a hidden size of 1,024, 16 attention heads, and about 340 million parameters. These figures tell us that the experiments compare encoders of different depth and width, and later results show that the larger tested models perform better on the selected tasks. They should not be mistaken for the definition of bidirectionality. A larger left-to-right encoder would still face the direction constraint that BERT is designed to remove.
The same caution applies to the phrase “understands both sides.” BERT does not receive a human interpretation of a sentence. It computes vectors through learned transformations, and those vectors can be useful for a task when the task's labels and output layer guide fine-tuning. Bidirectional attention supplies an information route; it does not guarantee that the model has resolved every ambiguity or captured every relationship. That distinction will matter when we examine how BERT is trained.
One sequence, three kinds of information
BERT's encoder can only operate on the information placed into its input. To make one encoder useful across tasks, the paper constructs every example as a token sequence with several kinds of information combined at each position. The token says what piece of text is present. The position says where it occurs. For a pair of text spans, a segment embedding says which span supplied it. These signals are added together before the sequence enters the Transformer.
The token units are WordPiece tokens rather than necessarily whole words. A familiar word may remain one token, while a less common word can be split into subword pieces. The paper uses a 30,000-token WordPiece vocabulary. For a reader trying to understand the architecture, the crucial consequence is that BERT's positions are token positions, and a word's meaning may be distributed across one or more of them. The later token-level outputs correspond to those positions, not automatically to the words a person would count on the page.
Every sequence begins with [CLS], a special classification token. For a sentence pair, BERT places [SEP] between the two spans and uses another separator at the end. Learned segment embeddings distinguish tokens belonging to span A from tokens belonging to span B. Position embeddings distinguish the first position from the second, the second from the third, and so on. The representation at one position is therefore not just “the embedding for this token.” It is the sum of token, segment, and position embeddings before contextual processing begins.
Consider a simplified pair such as “my dog is cute” followed by “he likes playing.” BERT can represent it as a sequence beginning with [CLS], followed by the first span, [SEP], the second span, and a final separator. The segment signal marks the first span as A and the second as B. The position signal keeps the order intact. Once this sequence enters the encoder, self-attention can connect “he” in the second span with information from the first span, while the segment markers preserve the fact that the text came from different sides of the pair.
The same interface handles a single span. In that case, there is no second text to distinguish, but the sequence still has a beginning marker, token positions, and the information needed to preserve order. This common format is a system-level choice: the encoder does not receive one kind of object for classification and an entirely different kind for question answering. The task-specific difference is expressed mainly through what the output layer reads and what labels it is trained to predict.
At the output boundary, the first special token and the ordinary token positions serve different purposes. The final hidden vector for [CLS] is used as an aggregate representation for classification tasks. A classifier can map that vector to labels such as entailment, contradiction, or sentiment. The vector is not a magical summary that is equally appropriate for every problem; its usefulness depends on the pretraining and fine-tuning signals that shape it.
The other output is a contextual vector for each token position. These vectors can feed token-level predictions, such as named-entity labels, or support the start and end scores used in question answering. If the task asks “which part of the passage answers this question?”, a single whole-sequence label is too coarse. The task needs access to the representations attached to the passage positions. BERT's interface keeps those representations available rather than compressing the entire input into one vector too early.
This division also explains why the special tokens should not be understood as ordinary vocabulary words. [CLS] provides a stable location whose final representation can be read as the sequence-level input to a task head. [SEP] marks a boundary between spans. Segment and position embeddings provide additional identity and order information. None of these signals makes a prediction by itself. They give the encoder a structured input from which contextual representations can be computed.
There is a subtle but important consequence. Packing two spans together means that BERT's self-attention can provide interaction between them inside the encoder. A downstream system does not first have to obtain two isolated summaries and then invent a separate mechanism to compare them. The representations can already reflect cross-span context by the time the task-specific layer receives them. That is why the input format is part of BERT's transfer story, not just a formatting convention.
The chapter's structural model is now complete enough to ask the next question. BERT has positions that can exchange information in both directions, and it has markers that tell the encoder what the positions represent. But an untrained encoder would not know how to use those signals. The next step is to see how BERT learns from unlabeled text without allowing a token to simply read its own answer, and how it learns that two spans can stand in a meaningful relationship.