The Missing Half: Kernels, Memory, and Hardware

BitNet b1.58 2B4T’s reported resource advantage belongs to more than its checkpoint. This chapter follows the runtime path that turns ternary weights and 8-bit activations into an output: how packed weights move through memory, how a GPU kernel reconstructs them for computation, and how the CPU reference path takes a different approach. The practical judgment is whether a low-bit model can preserve its advantage at the interface between representation and hardware, where packing saves bandwidth but unpacking and unsupported instructions can introduce new costs.

Why the representation needs a custom path

A standard deep-learning library is usually optimized around familiar numerical formats such as FP16, BF16, INT8, or INT4. BitNet’s W1.58A8 operation does not fit neatly into those categories. Its weights are ternary, its activations are 8-bit, and the useful computation depends on recognizing that a weight means negative one, zero, or positive one rather than treating it as an ordinary integer with a broad range of possible values.

The 2025 report’s GPU implementation addresses this mismatch with a custom CUDA kernel. It packs four ternary values into one 8-bit storage unit for movement through high-bandwidth memory. The packed form is efficient to store and transfer, but it is not the form in which the arithmetic is most naturally described. The kernel loads a tile of packed weights from high-bandwidth memory into the GPU’s faster on-chip shared memory, unpacks the values, reconstructs the usable ternary representation, and computes with the 8-bit activations.

That sequence can be summarized as pack, store, load, unpack, and compute. Each step owns a different problem. Packing reduces the number of bytes that must travel through the memory interface. High-bandwidth memory provides capacity for the model’s stored weights. Shared memory gives the kernel a faster local working area. Unpacking converts the storage code into values that the operation can accumulate. The computation then uses the ternary pattern and the activation integers to produce the next tensor.

The int8 storage unit is an implementation container, not a claim that four ternary values have become ordinary four-bit integers. Each ternary value has three possible states, and the kernel must know how the packed bits encode those states. The useful question is not whether the container’s name sounds familiar, but whether the representation can be loaded, decoded, and accumulated without expanding every value into a costly full-precision object. That is the interface the custom kernel owns.

For a matrix tile, the kernel repeats this process across many weight units and activation values. A packed block is fetched, brought close to the arithmetic units, decoded into signed choices, and combined with the corresponding activation entries. Positive choices contribute additions, negative choices contribute subtractions, and zero choices can be omitted. The partial sums are then combined into the output tensor. This organization gives the runtime a way to trade generality for a path tailored to the one model format.