Teaching a Model to Live With Ternary Weights
BitNet b1.58’s ternary linear layer creates a learning problem as well as a storage format: a model must acquire useful representations while its most important transformations have far fewer weight values available. This chapter examines how the reported BitNet designs address that problem. It separates the durable BitLinear idea from the surrounding Transformer choices, then follows BitNet b1.58 2B4T from broad pre-training through instruction tuning and preference optimization. The aim is to distinguish a documented training recipe from a universal law about low-bit models.
The BitLinear core and its changing surroundings
The stable architectural decision is to replace the ordinary full-precision linear layer with BitLinear. That is the point at which the ternary-weight, low-precision computation enters the Transformer’s repeated transformations. The model is still recognizably a Transformer: it has layers that combine representations, normalize them, add positional information, and pass them through feed-forward components. BitNet does not discard the whole architecture and replace it with a new kind of sequence model.
This continuity is practical. The 2024 report describes a LLaMA-alike design using RMSNorm, SwiGLU, rotary position embeddings, and no biases. Those choices make the model easier to connect to open-source language-model tooling because readers and systems can recognize the surrounding interfaces. The low-bit change is concentrated in the operation that dominates much of the numerical work, while the rest of the model retains familiar Transformer responsibilities.
The 2025 2B4T report shows why “BitNet architecture” should not be read as a frozen list of components. It retains BitLinear, rotary position embeddings, and bias removal, but reports subln normalization and squared ReLU in the feed-forward network rather than the earlier report’s RMSNorm and SwiGLU description. It also uses the LLaMA 3 byte-level BPE tokenizer with a 128,256-token vocabulary. These differences are not a contradiction that needs to be smoothed away. They describe different generations of the design.
The distinction matters because only some choices are directly tied to the low-bit thesis. BitLinear is the structural identity that makes the model native to ternary weights. Subln is reported as a way to improve training stability in the 2B4T quantized regime. Squared ReLU is motivated by possible sparsity and computational advantages. RoPE supplies positional information, and the tokenizer determines how text and code become the token sequence that the Transformer processes. Each component participates in the model, but the supplied reports do not isolate the causal contribution of each one.
That prevents an easy but misleading story. It would be wrong to say that every reported gain comes from ternary weights alone, because the model is a package of architectural and training decisions. It would be equally wrong to say that the low-bit change is incidental, because BitLinear changes the central matrix operation and the 2025 report’s inference path is built around it. The accurate description is layered: a recognizable Transformer frame contains a native low-bit linear core, and the surrounding choices are tuned for particular model generations.
The surrounding frame also determines what crosses the low-bit boundary. Tokenization decides the units that enter the network; positional embeddings tell the model where those units occur; normalization keeps intermediate representations in a usable range; the feed-forward path transforms them after attention and before the next layer. BitLinear owns the restricted matrix operation, but it does not own the entire state of the Transformer. This division of responsibility is why the model can remain compatible with familiar Transformer concepts while still requiring a specialized runtime for its central layers.
Training from scratch follows directly from that architecture. A post-training conversion can take a finished full-precision checkpoint and search for a lower-precision approximation. BitNet starts with the low-bit operation as part of the model it intends to run. The parameters must be learned in a setting where the forward computation repeatedly uses the constrained representation. The point is not that the model never has any higher-precision training state; the point is that the model is not first taught one computational regime and only later asked to survive another.
This changes what “success” means during training. It is not enough for a quantized checkpoint to resemble a full-precision checkpoint numerically. The learned representations must remain useful when they pass through many ternary-weight transformations. The report’s results suggest that this can work at the tested scales, but the architecture itself does not guarantee it. It creates the conditions under which the training process can attempt the trade-off.
From pre-training to alignment
BitNet b1.58 2B4T’s training lifecycle has three broad states: large-scale pre-training, supervised fine-tuning, and direct preference optimization. The states have different purposes and different data relationships. Pre-training builds general language and world knowledge; supervised fine-tuning teaches the model to respond in instruction and conversation formats; preference optimization adjusts the style and behavior of those responses toward selected human judgments. The low-bit constraint persists through the lifecycle, while the data and objective change around it.
Pre-training itself is divided into two stages. The first uses a relatively high peak learning rate with a cosine decay schedule. The report connects that choice to an observation that 1-bit models can be more stable than full-precision counterparts, allowing more aggressive initial updates. The aim is to establish broad capability across a large mixture of public text, code, and related data.
Approximately midway through the planned token count, the schedule enters a cooldown. The learning rate drops abruptly and then follows a lower-valued cosine schedule. At the same time, the data mix places more emphasis on higher-quality curated material and synthetic mathematical data. Weight decay follows a related transition: it is used during the first stage, reaching a reported peak of 0.1, and is effectively disabled in the second. The model is therefore not just trained longer with the same settings. The optimization environment changes as the model moves from broad acquisition toward refinement.
The reason this staging matters is that numerical constraint and data quality interact. Early training needs enough varied material to build a wide foundation, while the cooldown gives later tokens greater influence over the final adjustment of the model. A lower learning rate reduces the size of those later changes, and removing weight decay lets the parameters settle under the curated signal. The report presents this as a recipe that worked for 2B4T; it does not show that every low-bit model must use the same transition point or the same regularization schedule.
A transition in this lifecycle changes the conditions around the same model state; it does not create a new model from nothing. Knowledge acquired during the first stage remains in the parameters as the learning rate falls, but the later data and smaller updates can change which patterns are emphasized. The boundary between stages is therefore a control decision: the training process changes what new evidence can do to the model. Because the report does not publish an ablation over alternative transition points or data mixtures, the reader can explain the intended mechanism without treating the chosen schedule as a proven optimum.
The amount of training also matters to the identity of the model. BitNet b1.58 2B4T is described as having learned from 4 trillion tokens, not as a small conversion of an existing model. That large training run is part of the evidence for native low-bit scaling, but it also makes comparisons difficult to interpret casually. Capability reflects the representation, the architecture, the data, the optimization schedule, and the amount of training together. A reader cannot attribute the whole outcome to the ternary alphabet while ignoring the rest of the lifecycle.
After pre-training, supervised fine-tuning changes the model’s immediate task. It uses instruction-following and conversational data, supplemented with synthetic material aimed at reasoning and complex instruction adherence. The model is no longer being asked only to continue broad text distributions. It is being shaped to recognize a conversational format, follow a user’s request, and produce an answer in the expected role.
The report highlights two optimization observations for this stage. First, it sums cross-entropy losses across tokens within a batch rather than taking a mean reduction. Second, the low-bit model benefits from a relatively larger learning rate and a longer fine-tuning duration than comparable full-precision models in the authors’ experiments. These details illustrate a larger point: a recipe that works for a native ternary model may not be a drop-in copy of a conventional fine-tuning recipe. The restricted representation can change how quickly useful behavior emerges and how long the model needs to settle.
Direct preference optimization follows SFT. DPO uses preferred and less-preferred responses to adjust the model directly, avoiding the need to train a separate reward model. In the 2B4T report, preference data comes from collections intended to capture varied judgments about model outputs. The stated purpose is to refine helpfulness, safety, and conversational style without significantly damaging the capabilities built earlier. The report gives a two-epoch DPO run, a learning rate of 2 × 10−7, and a beta value of 0.1, and notes the use of optimized training kernels.
The transition from SFT to DPO is not a replacement of the model’s knowledge. It is a change in what the optimization rewards. Pre-training supplies the broad representational base; SFT establishes a usable interaction pattern; DPO shifts the model toward responses that the preference data treats as better. Each stage inherits what the previous stage created, so a failure in an earlier state can constrain what later alignment can repair. Preference optimization can refine behavior, but it is not evidence that the underlying low-bit representation has become equivalent to full precision in every capability.
This also explains why the stages should not be judged by one shared metric. Pre-training is responsible for general language behavior and knowledge, SFT for following an explicit interaction format, and DPO for relative response preferences. A model can improve in one state while exposing a weakness from an earlier state. The report’s qualitative statement that DPO steered responses without significant degradation is useful evidence about the reported run, but it does not replace task-level evaluation of the final checkpoint.
This lifecycle also clarifies what the reports do not establish. The authors describe empirical observations about learning rates, loss aggregation, longer fine-tuning, data staging, and DPO. The supplied material does not include ablations that independently remove each choice and measure the change. It does not show that squared ReLU rather than SwiGLU caused the 2B4T results, that the exact cooldown point is optimal, or that the same recipe transfers unchanged to other languages and model sizes. Those are open experimental questions, not missing details to be filled in by intuition.
BitNet’s architectural and training story therefore has a precise shape. The BitLinear layer makes ternary computation part of the model’s native path. The surrounding Transformer components can change as the design evolves. Pre-training, cooldown, SFT, and DPO alter the data and objective while preserving the low-bit constraint. The result is a model that has been taught to operate under a numerical limitation, not a full-precision model wearing a smaller storage format. The next step is to connect this mechanism and lifecycle to a fair evaluation question: what would the reported measurements have to show before we could say that the native constraint changes the efficiency frontier rather than merely changing the representation?