What 1.58 Bits Actually Changes
For BitNet b1.58, “1.58-bit” names a particular set of weight values, not a strange fractional number stored in every memory cell. This chapter focuses on the numerical path behind that name: why the model uses three weight states, how those states are produced, and why the activations use a different low-precision format. By the end, the reader can trace what happens inside a BitLinear layer and understand why the representation changes the model’s deployment bargain without, by itself, determining the whole system’s memory or speed.
Three values, not a fractional number
A conventional language model gives each weight a value drawn from a wide range of floating-point numbers. A weight might be positive or negative, large or small, and the precise value participates in the learned transformation. BitNet b1.58 replaces that freedom in its quantized linear path with only three possibilities:
a weight can be −1, 0, or +1.
The label 1.58 comes from the number of states. A choice among three equally possible symbols contains log2(3) bits of information, which is approximately 1.58. The phrase is therefore an information-theoretic description of a ternary alphabet. It does not mean that an implementation has discovered a physical object containing 1.58 indivisible bits, nor that every byte in a deployed model has exactly that density.
The distinction from a binary model lies in the zero. With only −1 and +1, every connection remains active with one of two signs. The ternary option lets the model suppress a connection altogether. The 2024 report describes this as explicit feature filtering: a zero can say that, for this learned transformation, a particular contribution should not be added or subtracted. That extra state costs a little more information than a strict binary choice, but it gives the model a way to represent absence as well as direction.
This is a modeling choice, not merely a packing trick. Consider a linear transformation as a collection of weighted contributions. A positive weight pushes the output in the direction of an input feature; a negative weight pushes against it; a zero removes that feature’s direct contribution at that layer. The actual model is far more complicated than this three-part picture, because many layers compose their transformations and each weight has a learned scale. But the picture captures why the third state can improve the expressive behavior of a low-bit layer: the model does not have to turn every connection into either an equal-strength positive or an equal-strength negative vote.
The ternary values are derived from a full-valued weight matrix during the quantized operation. The 2024 report describes an absolute-mean, or absmean, procedure. First, the model calculates the average absolute magnitude of the weights in the matrix. Call that scale γ. Each weight is divided by that scale, with a small stabilizing term in the denominator, and the result is rounded to the nearest integer and clipped to the interval from −1 to +1. In simplified form, the path is:
a full-valued matrix → divide by its average absolute magnitude → round → clip to −1, 0, or +1.
The scale matters because it preserves a shared sense of magnitude while the individual values lose their fine-grained precision. Without scaling, the same threshold would not mean the same thing for a matrix whose weights are generally small and one whose weights are generally large. Absmean quantization gives the ternary representation a local reference before it discards detail.
The operation is easier to understand with a simple threshold picture. After division by the matrix scale, a normalized value close to zero rounds to zero; a sufficiently positive value rounds to +1; a sufficiently negative value rounds to −1. Values outside the allowed interval are clipped, so the result cannot acquire a fourth state simply because one original weight was unusually large. The original magnitude is not preserved separately for every connection. Instead, the matrix shares a scale, and the ternary pattern records the direction, suppression, and rough strength category of each contribution. That is the price of replacing many floating-point choices with three states.
The scale also shows why this is a learned-computation issue rather than a cosmetic encoding. A different matrix has a different distribution of weight magnitudes, so its quantization scale changes the boundary at which contributions become positive, negative, or zero. The low-bit operation is consequently tied to the layer that owns the weights. It is not enough to declare that a checkpoint is “1.58-bit” and leave every numerical decision to a generic file converter; the model’s matrices and the computation that consumes them have to agree about the representation.
It is useful to separate two objects that can otherwise be confused. During training, the model has parameters and optimization state that allow it to learn. During the forward computation, the weight values used by the low-bit layer are quantized into the ternary form. The report’s central claim is that the model is trained from scratch for this native low-bit regime; it is not simply an ordinary finished model whose file has been rewritten. The quantized path is therefore part of the model’s intended behavior, not an afterthought applied only when it is time to deploy.
That does not imply that the ternary representation preserves every detail of a full-precision matrix. It deliberately throws detail away. The question is whether the surrounding architecture and training process can learn useful distributed representations despite that loss. The later results matter because the representation alone cannot answer that question.
How the low-bit linear path works
The weights are not the only values in a linear layer. An input token arrives as an activation vector, and the layer combines that vector with its weights to produce an output that will feed the next part of the Transformer. BitNet b1.58 uses a mixed-precision path known as W1.58A8: roughly, 1.58-bit ternary weights and 8-bit activations. The asymmetry is important. Persistent weights receive the most aggressive restriction, while activations retain eight-bit integer values rather than being forced into the same three-state alphabet.
The activation procedure described in the reports scales the values per token to a signed 8-bit range. The 2025 report calls this an absmax strategy, while the earlier report explains that activations are scaled per token without a zero point. A token’s activation vector can therefore have its own range before it is represented as integers. The purpose is not to make activations identical to weights. It is to reduce the precision and data movement of the runtime input while retaining more numerical gradation than a ternary signal would provide.
The two paths can be described side by side. For weights, the layer estimates an absolute-mean scale and maps each normalized value to −1, 0, or +1. For activations, it estimates a per-token range and maps the input to signed 8-bit integers. The layer then performs its low-bit transformation with the ternary weight pattern and the integer activation values. Keeping the paths distinct prevents a common misunderstanding: “1.58-bit model” does not mean that every intermediate quantity is ternary.
The per-token part matters because activations are temporary signals whose ranges can differ from one token to the next. A single global activation scale would have to accommodate unusually large values and could use too little of the available integer range for ordinary tokens. Scaling each token separately lets the representation adapt locally before the values enter the linear operation. The reports present this as a convenient and system-friendly implementation choice, while also making clear that the activations remain 8-bit rather than ternary.
The asymmetry should not be mistaken for an accident or for a claim that every possible low-bit design must use exactly this pairing. The supplied reports do not provide an ablation that proves ternary weights and 8-bit activations are universally optimal. They document the W1.58A8 design used for BitNet b1.58 and 2B4T. Its significance is practical: the weights, which persist across all requests, receive the strongest compression, while the short-lived input to each operation keeps more levels of numerical information. That division creates a distinct interface between the stored model and the live computation.
The architectural change is concentrated where it matters most for cost: the standard full-precision linear layer becomes BitLinear. In a Transformer, these matrix transformations account for a large share of the computation. When the weights can only be negative one, zero, or positive one, the operation can be organized around integer addition, subtraction, and skipping rather than a conventional multiply for every full-precision pair. The 2024 report describes this as a computation paradigm with almost no multiplication operations in the matrix multiplication. That description explains the potential; the actual benefit still depends on a kernel that can use the representation directly.
At the level of one contribution, the rule is intuitive. A +1 weight adds the corresponding activation, a −1 weight subtracts it, and a zero weight contributes nothing. A whole matrix is a large pattern of those three choices, combined with the scales used by the quantization procedure. This is why the reports describe the matrix multiplication as relying mainly on integer addition: the expensive question is no longer “what floating-point product should be formed for every pair?” but “which signed activation values should be accumulated, and which should be skipped?” The phrase “almost no multiplication” is more accurate than “no numerical work.” Scales, accumulation, normalization, and movement still have to be handled.
The change does not turn the entire Transformer into a ternary object. The reports retain a recognizable Transformer frame and discuss separate choices for normalization, feed-forward activation, positional embeddings, biases, tokenization, and other components. The 2024 model uses LLaMA-alike components to ease integration with open-source tooling. The 2025 2B4T report describes its own surrounding choices, including subln normalization, squared ReLU, RoPE, bias removal, and the LLaMA 3 tokenizer. These details belong to the model generation in which they are reported; they should not be mistaken for consequences that follow automatically from the number 1.58.
That boundary is also why nominal precision cannot be converted directly into a complete deployment forecast. A real model contains embeddings, and the 2025 report’s headline memory comparison is explicitly non-embedding. The runtime must store activation values, scales, and whatever metadata the chosen representation requires. A GPU implementation may pack several ternary values into a storage unit, but it still needs to load and unpack them in the right place. A CPU implementation may use a different organization altogether. The representation creates an opportunity to reduce memory and arithmetic costs; packing, memory hierarchy, and kernels determine how much of the opportunity becomes observable performance.
There is a second boundary around quality. A full-precision layer can express small distinctions in each individual weight. A ternary layer cannot. BitNet’s answer is not to deny that loss, but to train the whole model under the constraint and rely on many layers of learned transformations. The 2024 report found that, in its matched experiments, the performance gap narrowed with model size and that a 3B BitNet model could match the reported full-precision baseline on its measures. The result is evidence that the restriction can be workable at scale; it is not a proof that every model size, task, or architecture will absorb the same loss equally well.
The most important mental model is therefore a chain of distinctions. The three values explain the 1.58 label. The absmean step explains how weights enter that alphabet. The per-token 8-bit path explains why activations are low precision without being ternary. BitLinear explains where the model architecture changes. Packing and kernels explain why the change must continue into inference. None of those distinctions alone proves a useful language model, but together they define the technical bet that the training process must validate.
A numerical representation becomes meaningful only when a model can learn with it. The next question is consequently not how to compress the weights further, but how BitNet’s architecture and training lifecycle make a ternary-weight model workable: which parts of the Transformer remain familiar, which choices change between model generations, and what the 2B4T training process does to accommodate the constraint.