Training with Less Numerical Headroom

Nemotron 3 Super’s architecture is designed to save memory movement and decoding work, but those savings begin before inference. The model was pretrained for 25 trillion tokens with an NVFP4-based recipe, and the training run exposed a concrete question: how much precision can be removed before optimization loses useful information? This chapter examines that question through the model’s mixed-precision assignments, the observed underflow pattern, and the experiments that tested whether higher precision could repair it. It then follows the base model through data shaping, checkpoint selection, and long-context continuation.

The important distinction is between using low precision deliberately and treating low precision as a blanket switch. Nemotron’s report does not describe every tensor as equally safe at four bits. It describes a numerical budget in which most linear work uses NVFP4 while selected layers remain in BF16 or MXFP8. The evidence also shows why the budget must be monitored: a loss curve can improve while downstream capability fails to improve, and a long context window requires more than a large nominal limit.

NVFP4 is a selective training recipe

A training format has to represent more than the forward-pass weights. During pretraining, the system also represents activations and gradients and must carry enough numerical detail through the backward pass for the model to keep learning. Smaller formats reduce storage and can accelerate matrix operations, but their limited range and precision make small values vulnerable to underflow or large values vulnerable to overflow. The practical question is not whether a format is “good” in isolation; it is whether the format, scaling method, kernels, and layer assignment preserve a useful optimization path.

Nemotron 3 Super uses NVFP4 for ordinary linear layers across the full 25-trillion-token horizon, but the report lists several exceptions. The final 15% of the network remains in BF16 to promote stability at scale. Latent projections, MTP layers, attention projections, and embeddings also remain in BF16. Mamba output projections use MXFP8 because smaller models showed a high incidence of underflow when that layer was quantized to NVFP4. These choices reveal the design logic: put the most aggressive format where its step-time and memory benefits matter, and protect operations whose numerical behavior is more sensitive or whose cost is small enough that a higher-precision assignment is acceptable.

The recipe also uses multiple levels of scaling. Weights use two-dimensional block scaling so the same quantized weights remain consistent between forward and backward computation. Activations and gradients use one-dimensional blocks along the GEMM reduction axis. Random Hadamard transforms are applied to inputs for weight-gradient computation, and gradient tensors use stochastic rounding. At the element level, the format uses E2M1 values in 16-element micro-blocks, E4M3 scaling factors for those micro-blocks, and a second FP32 global scale. These details matter because the useful precision comes from the combination of small values and their scales, not from the element format alone.

The result should be described as mixed-precision NVFP4 training, not blanket four-bit training.