The Last Mile: Compression Without Losing Behavior

For Nemotron 3 Super, deployment quantization is not a final cosmetic conversion from a large model to a smaller file. It is a choice about which numerical errors the target hardware can tolerate and which model states must remain precise enough to preserve behavior. This chapter compares the FP8 and NVFP4 checkpoints, explains how sensitivity-based layer promotion allocates precision, and follows one failure case—quantizing the recurrent Mamba state—where accuracy alone missed a visible change in verbosity.

The practical judgment is simple but demanding: a compressed checkpoint should be evaluated as an operating system for an agent, not only as a table of benchmark scores. A format that saves memory but produces longer, less controlled trajectories may reduce the useful throughput of a coding agent. Nemotron’s deployment report is valuable because it documents both the successful mixed-precision recipe and the failure surface that shaped the final cache choice.

Two formats, two hardware targets

The report produces an FP8 W8A8 checkpoint for Hopper-class hardware and an NVFP4 W4A4 checkpoint for Blackwell. W8A8 means that weights and activations use eight-bit floating-point representations in the targeted operations; W4A4 uses four-bit representations more aggressively. The formats are not interchangeable labels for the same deployment. They are selected with the capabilities and bottlenecks of different GPU generations in mind.

The FP8 checkpoint quantizes the routed and shared MoE GEMMs and the Mamba linear layers. It keeps the embedding, selected attention operations, output layers, and router at higher precision. The attention key-value cache is stored in FP8, while the Mamba state cache is stored in FP16 and the recurrent arithmetic remains in FP32. This assignment is already mixed precision: the goal is not to force every operator into the cheapest format, but to lower the cost of the operations that dominate the target workload while protecting operations where error has a larger consequence.

The NVFP4 checkpoint is more aggressive. It is especially attractive for prefill-heavy workloads such as coding-agent deployments, where linear and MoE GEMMs can dominate the work before long decoding begins. Its element values use signed E2M1 numbers, with per-block scaling over groups of 16 along the final dimension. The scale values themselves use a higher-precision representation and a statically calibrated global scale. The block structure gives the format more usable range than a single scale for an entire tensor, but it still cannot represent every layer equally well.

The report therefore changes the scaling rule for different objects. For weights, it searches for per-block scales that minimize weight mean-squared error. That search can happen offline, so it can spend more computation finding a good scale without slowing the serving path. For activations, it keeps max-based per-block scaling because activation scales must be computed efficiently at runtime. The asymmetry is deliberate: offline calibration can be expensive; runtime quantization must be predictable and cheap.

Some operations are promoted from NVFP4 to FP8 or BF16 when their sensitivity justifies the cost. AutoQuantize treats that choice as a constrained optimization problem.