Overview

BERT's benchmark tables show broad transfer across eleven NLP tasks, while internal ablations test whether masked pretraining, next sentence prediction, and deep bidirectionality help explain it. The results support broad transfer in the paper's 2018 setups and make those choices credible causal candidates, but cross-model confounds, mixed metrics, model variants, data conditions, and fine-tuning instability limit attribution. One feature-based NER result shows fixed activations can be competitive in that setup, not universally.

Learning outcomes

Benchmarks show that one BERT encoder transferred across varied task formats in the paper's selected 2018 evaluations.
Removing NSP or replacing MLM with left-to-right pretraining damaged the listed BERTBASE results, supporting—but not isolating—the design contribution.
BERTLARGE improved selected results when sufficiently pretrained, but fine-tuning could be unstable on small GLUE datasets.

Keywords

BERTBenchmarksAblationsGLUESQuADCausality

What the Evidence Actually Shows

For BERT, the benchmark tables answer one question—whether a shared pretrained representation transfers across tasks—while the ablations ask a harder question: which design choices might explain the transfer? This chapter reads the two kinds of evidence together. The goal is not to repeat the paper's headline scores, but to separate breadth of performance from causal attribution and to identify the costs that remain when task-specific architecture is reduced.

Broad transfer is the result, not the explanation

The paper evaluates BERT on eleven natural-language-processing tasks, covering sentence classification, sentence-pair inference, reading comprehension, token labeling, and commonsense choice. That range matters because a model can perform well on one narrow format by exploiting a specialized structure. BERT's claim is stronger: the same pretrained encoder, with task-shaped outputs, can support decisions at different levels of granularity.

The GLUE results provide the paper's broadest sentence-level comparison. On the included tasks, both BERTBASE and BERTLARGE outperform the listed systems in the table. BERTLARGE reaches 86.7 matched and 85.9 mismatched accuracy on MNLI, 92.7 on QNLI, and an average of 82.1 under the paper's calculation. The paper reports that this average excludes WNLI because of problems with that dataset's construction. The result is evidence of breadth within the stated benchmark setup, not a single neutral measurement of “language understanding.”

The reading-comprehension results test a different interface. On SQuAD v1.1, the paper reports 91.8 test F1 for a single BERTLARGE model and 93.2 for an ensemble that uses additional pre-fine-tuning on TriviaQA. On SQuAD v2.0, where a system must sometimes recognize that no answer is present, the reported single BERTLARGE model reaches 83.1 test F1. These results connect the shared encoder to the start/end span mechanism described earlier, but the model variant and data setup are part of what each number means.

SWAG supplies another kind of relationship judgment. BERTLARGE reaches 86.3 test accuracy in the paper's table, compared with 59.2 for the cited ESIM+ELMo baseline. The task asks the model to choose the most plausible continuation among four candidates. This is not question answering and not ordinary sentence classification, yet the input can still be represented as a pair and scored through [CLS]. Together, these benchmarks make a credible case that the transfer interface is not limited to one task family.

The right interpretation is therefore structural. BERT can carry contextual information from pretraining into several downstream output contracts: a whole-sequence label, a relationship label, an answer span, or a multiple-choice score. The results do not say that all of those tasks are the same. They show that one representation-producing system can be adapted to them without replacing its central encoder.

Published by DeepMox

FAQ

How do BERT benchmarks differ from ablations?

A benchmark records what the complete recipe achieves under a task's setup, whereas an ablation changes a design component to examine how the result moves.

Why aren't BERT's benchmark scores directly comparable?

They use different tasks and metrics, and some results involve different model variants or data conditions, so the printed scores are not on one common scale.

Can frozen BERT features match fine-tuning?

They were close in the reported CoNLL-2003 setup: fixed BERTBASE features reached 96.1 development F1 versus 96.4 for BERTBASE fine-tuning and 96.6 for BERTLARGE. That result is not universal.

Previous9 minHow BERT Adapts to Many Tasks

Next11 minThe BERT Lesson: Reusable Context Has a Price

What BERT's Evidence Shows

What the Evidence Actually Shows

Broad transfer is the result, not the explanation

FAQ