BERT: How Context Transfers

Overview

This series explains how BERT turns deep bidirectional context into a reusable representation for sentence labels, token tags, answer spans, and pairwise choices. It follows the path from shared input and masked pretraining to task-specific heads, then weighs benchmarks and ablations against costs in data, compute, sequence length, adaptation, and evaluation. The evidence supports broad transfer in the paper's 2018 setups, not a universal measure of language understanding.

Learning outcomes

BERT makes contextual representations reusable by separating shared encoder learning from task-specific decisions.
Masked language modeling hides selected tokens while preserving two-sided context, at the cost of slower convergence.
One encoder can support labels, token tags, answer spans, and choices because each task supplies its own output contract.
Reuse reduces downstream architecture, but moves complexity into pretraining data, compute, sequence length, and adaptation.
The paper shows broad transfer in its 2018 setups, but benchmark conditions and ablations do not prove universal causation.

Keywords

BERTTransformersBidirectionalityPretrainingFine-tuningSelf-attentionWordPieceQuestion answeringGLUESQuAD

Chapters

Why BERT Made Context Transferable

The design gap between one-way context and reusable representations

9 min · The design gap between one-way context and reusable representations

9 min

How BERT Reads Input Sequences

Bidirectional attention, segment markers, and output vectors

10 min · Bidirectional attention, segment markers, and output vectors

10 min

How BERT Learns Before It Answers

Masked tokens, sentence pairs, and the cost of the checkpoint

12 min · Masked tokens, sentence pairs, and the cost of the checkpoint

12 min

How BERT Adapts to Many Tasks

Task heads, answer spans, and fine-tuning versus frozen features

9 min · Task heads, answer spans, and fine-tuning versus frozen features

9 min

What BERT's Evidence Shows

Benchmarks, ablations, and the limits of causal claims

11 min · Benchmarks, ablations, and the limits of causal claims

11 min

The BERT Lesson: Reusable Context Has a Price

Where pretraining, compute, task fit, and evaluation complexity remain

11 min · Where pretraining, compute, task fit, and evaluation complexity remain

11 min

Preview every chapter · Unlock full text, audio, and video from chapter 4

Plans

Ch.1-3 FreeCh.4-6 Premium

FAQ

Why can BERT use context from both sides of a token?

BERT's encoder removes the left-to-right causal mask, so self-attention lets each layer incorporate information from positions before and after the token.

How does BERT pretrain a bidirectional encoder without revealing every target token?

BERT masks or alters 15% of WordPiece positions and predicts the originals, using 80% [MASK], 10% random replacements, and 10% unchanged inputs.

How does BERT adapt one checkpoint to different language tasks?

BERT packs each task's text into a shared input format, then fine-tunes the encoder with a task-specific head for labels, token positions, spans, or choices.

What do the BERT paper's benchmarks and ablations actually establish?

They support broad transfer and the value of masked bidirectional and sentence-pair training in the studied 2018 setups, but not universal language understanding or single-component causation.

What does BERT's reusable context cost?

BERT shifts complexity into large-scale pretraining data and compute; masked training converges more slowly, long sequences raise attention costs, and adaptation remains task-dependent.