The Two Questions TwHIN-BERT Asks of a Tweet
For TwHIN-BERT, the decisive design question is what a Tweet representation should be asked to preserve. This chapter separates the two signals in the model—masked language prediction and engagement-derived social comparison—so readers can see what each contributes, where the social loss enters the encoder, and why their combination is more precise than simply calling all surrounding information “context.”
What masked language modeling gives the encoder
The first signal is familiar from BERT: masked language modeling, usually shortened to MLM. During pre-training, the model receives a sequence in which some tokens have been hidden or altered. Its task is to recover the original token at each selected position. The answer must be inferred from the rest of the sequence, so the model learns regularities in vocabulary, syntax, spelling, and meaning without requiring a human to label every example.
The masking is what lets BERT use context from both directions. In a left-to-right language model, a token can attend only to material that comes before it. If the missing word is near the end of a sentence, the model cannot use what follows it. BERT instead masks the target before prediction. The encoder can attend to tokens on both sides of that position, because the target itself is unavailable as a direct shortcut. A phrase such as “the team won by a narrow ___” can be completed using the words around the gap, not only the prefix.
This is not merely a trick for filling blanks. Each prediction forces the encoder to construct a representation of the visible sequence that contains enough information to identify the hidden token. Repeating the task across a large corpus gives the model pressure to represent relationships among words and subwords. The original BERT procedure selected 15% of WordPiece tokens for prediction and used a mixture of masking, random replacement, and leaving the token unchanged. Those details belong to the original BERT method; TwHIN-BERT's relevant statement is that it retains a masked-language-model objective alongside its social objective.
BERT's input format also provides a useful vocabulary for TwHIN-BERT. A special [CLS] token appears at the beginning of an input, and a [SEP] token can separate text segments. Each input position combines token, segment, and position information. After the Transformer has processed the sequence, the hidden state associated with [CLS] can serve as an aggregate representation of the whole input. A small task-specific layer can use it for classification, while all of the pre-trained parameters can be fine-tuned for the downstream task.
For a Tweet, this pooled representation is valuable because it compresses a short, noisy sequence into a vector that later tasks can use. A hashtag classifier can attach a prediction layer to it. A sentiment or topic classifier can do the same. Alternatively, the representation can be kept fixed and passed to a simpler model. In either case, MLM supplies the text-side learning pressure: represent the content well enough that missing pieces can be recovered and the resulting vector can transfer to another task.
That transfer path creates a boundary between what pre-training knows and what a downstream task teaches. During pre-training, the model sees unlabeled text and learns general patterns that can be reused. During fine-tuning, a task-specific output layer supplies labels and the model's parameters can adjust to the task. With fixed features, that adjustment does not happen: a simpler classifier must work with the representation already produced. The distinction matters for TwHIN-BERT because a representation that is useful for social comparison is not automatically a complete classifier for engagement, sentiment, or topic. Those later tasks still define their own targets and evaluation rules.
It also explains why a pooled vector is a meaningful interface. The individual tokens carry local evidence, but a downstream classifier needs a compact summary of the Tweet. The [CLS] state is trained inside the Transformer and can collect information from the entire sequence. TwHIN-BERT can then ask an additional question of that summary through its social projection head. The model is not required to give every token a separate, human-readable social label; it is required to make the representation useful under the training objectives that act on it.
There is an important distinction between this mechanism and the original BERT paper's second pre-training task, next-sentence prediction. BERT used NSP to train on relationships between sentence pairs. TwHIN-BERT's stated addition is different. Its pair is not chosen because one sentence follows another in a document. It is chosen because two Tweets are judged socially similar from engagement-derived representations. The question is therefore not whether one text is next to another in a corpus, but whether two separate posts occupy a compatible position in the social behavior represented by Twitter.
That distinction keeps the baseline honest. MLM already gives TwHIN-BERT a way to learn from both sides of a Tweet's text. The social objective must justify itself by supplying information that token prediction does not directly ask for. It is an additional constraint on the representation, not a more fashionable name for bidirectional language modeling.
What social contrast adds—and what it cannot mean
The second signal begins with the social-similarity pairs produced upstream from the engagement network. TwHIN-BERT encodes each Tweet with its Transformer and takes the hidden state associated with [CLS] as the Tweet-level representation. It then sends that representation through a multilayer perceptron, or MLP, projection head. The projected vectors are the objects compared by the social loss.
This placement matters. The model is not inserting a user's identity or a raw engagement count into every token position. The graph stage has already converted observed user–Tweet relationships into pairs of Tweets that are treated as socially similar. The language model receives the text of those Tweets and is trained to make their projected representations compatible. The social signal reaches the encoder through the learning objective, not as a claim that each word has a fixed social meaning.
The loss is contrastive. “Contrastive” means that the model learns by comparing a desired match with alternatives. For each pair, the paired Tweet is the positive example. The other Tweets in the batch that are not its partner serve as in-batch negatives. The model increases the cosine similarity of the positive projected vectors relative to those alternatives. Cosine similarity compares the direction of two vectors, which makes the training signal about their position in the learned space rather than about a raw magnitude alone.
A small example shows the logic. Suppose the upstream graph process supplies two Tweets as a socially similar pair. When their text is encoded in the same batch, the model should give their projected vectors a high relative similarity. It should give a lower similarity to the other Tweets in that batch, unless they are the designated partner. The result is not a binary label attached permanently to either Tweet. It is a pressure on the geometry of the representation: the positive relationship should be easier to recover than the alternatives under the current comparison set.
That last qualification is essential. In-batch negatives are convenient and efficient, but they do not form a complete list of all socially unrelated Tweets. Another Tweet in the batch may in fact appeal to some of the same users; it simply is not the selected partner for this training example. The contrastive loss therefore teaches a relative ordering under sampled pairs. It does not provide a universal social taxonomy in which every nearby vector is meaningful and every other vector is irrelevant.
The comparison set also determines the kind of update the encoder receives. The model is not asked to answer an absolute question such as “Is this Tweet socially relevant?” It is asked to make the designated partner more similar than the alternatives presented at that moment. That design makes large batches useful because they offer more contrasts, but it also means that the meaning of an update depends on how pairs are sampled and which other Tweets appear beside them. A pair-generation mistake can therefore become training pressure, and a possible social relationship can be treated as a negative simply because it was not the selected positive.
This does not make the objective unusable. It makes its interpretation conditional. The graph-derived pair is evidence about a relationship in the platform's observed engagement structure; the contrastive loss is a way to encode that evidence into the text model. Neither step turns an observational relationship into a human judgment. Readers should ask what the graph counted as similar, how reliably the retrieval found that similarity, and whether the downstream task needs precisely that kind of relation.
Nor does “socially similar” mean “semantically identical.” Two posts can attract related users because they concern the same topic, use a shared style, appear in the same conversation, or serve a similar role in the platform. They can also be related in ways that are not obvious from their wording. The source supports the narrower interpretation that they appeal to similar users under the learned engagement structure. It does not support treating the pair as a human-verified paraphrase, a factual agreement, or a causal influence relationship.
This is why the social comparison uses a projection head. The model can learn a space that is useful for comparing social relationships without requiring every downstream representation to be reduced to that one relation. The projection is the interface where the social objective makes its demand. The encoder still has to support the text objective, and the resulting model can later be evaluated on tasks that do not ask directly about the original pair.
TwHIN-BERT combines the two pressures as a joint loss:
[ L = L_{social} + \lambda L_{MLM} ]
The notation expresses a practical balance. The social term encourages compatible representations for Tweets connected by engagement patterns. The MLM term continues to reward representations that support masked-token prediction. If the social term were used alone, the model would have less direct reason to preserve the textual distinctions that matter for language tasks. If only MLM were used, the model would have no explicit objective to align Tweets according to the platform's engagement structure. The combined design asks the encoder to satisfy both conditions.
The equation is not a guarantee that the two signals will always agree. A social pair can be only loosely related in language, and a textually similar pair may not share the same audience. The weighting parameter and the quality of the mined pairs determine how strongly each pressure is felt. The paper reports a particular temperature and loss weight for its released configurations, but those values are settings for the described experiments, not universal constants. More generally, the model can preserve useful text behavior only to the extent that the joint training process balances these competing sources of evidence.
This also clarifies what success would look like. A social gain should appear in a task where user-conditioned engagement matters, because that is the relation the added objective was designed to represent. On semantic tasks, the joint model should at least retain useful language representations if the MLM signal remains effective. A good result in one category cannot be silently substituted for a result in the other: predicting which Tweet a user will engage with is not the same as deciding whether a Tweet is true or understanding every implication of its wording.
TwHIN-BERT's second question is therefore carefully bounded: given an engagement-derived positive pair, can the text encoder learn a representation in which that relationship is recoverable, while still performing ordinary language modeling? To answer that question, the system first needs a defensible way to create the pairs. The next step is to examine how heterogeneous engagements become a graph, how the graph gives Tweets a searchable geometry, and how approximation at billion-Tweet scale shapes the social signal that the contrastive loss receives.