Overview

DeepSeek-V3 uses FP8 selectively rather than converting its entire training state. Dense matrix multiplications and selected activations use FP8, while sensitive operators, master weights, and gradients retain higher precision. This chapter explains 1×128 activation tiles, 128×128 weight blocks, periodic FP32 accumulation, E4M3, and FP8 MoE dispatch, then separates numerical validation from speed claims. In comparisons at two smaller supporting scales after approximately one trillion tokens, relative loss error stayed below 0.25% versus BF16; the recipe remains hardware- and kernel-dependent.

Learning outcomes

DeepSeek-V3's FP8 path covers dense matrix multiplications and selected activations, not the entire model state or every operator.
Fine-grained 1×128 activation and 128×128 weight scaling limits outlier damage, while periodic FP32 accumulation recovers precision on H800 hardware.
At two smaller model scales, relative loss error stayed below 0.25% versus BF16 after about one trillion tokens; this does not prove an end-to-end speedup.

Keywords

DeepSeek-V3FP8quantizationaccumulationBF16FP32

Making FP8 Work at Scale

DeepSeek-V3's runtime reduces waiting caused by expert communication, but moving and multiplying all those activations still consumes memory and compute. The question here is how the model uses FP8—a lower-precision number format—without allowing rounding error to destabilize training. The answer is not to convert every tensor and hope for a speedup. DeepSeek-V3 assigns precision by operation, scales values in small groups, periodically restores accumulation precision, and keeps selected weights and gradients in safer formats. That makes FP8 a systems design with explicit failure conditions.

Lower precision is a constrained budget

A matrix multiplication combines many products and sums. In a large language model, these operations account for much of the compute, so a format that uses fewer bits can increase compute density and reduce the amount of data that must be stored or moved. FP8 is attractive for that reason. But the same reduction in numerical range and detail that makes it efficient can cause trouble during training. Activations, weights, and gradients can contain outliers. A single unusually large value can force a shared scale that makes most other values too coarse, while an accumulation of many small products can lose information before the result is written down.

DeepSeek-V3's framework treats precision as a budget. The three dense linear operations associated with a matrix layer—forward propagation, the backward pass for activation gradients, and the backward pass for weight gradients—are executed in FP8. Their outputs may be produced in BF16 or FP32. This is where the dense compute benefit is concentrated. The report says this arrangement theoretically doubles computational speed compared with the original BF16 method, but that is a statement about the operation's potential, not an end-to-end measurement of training time.

Other operations remain in higher precision. The embedding module, output head, MoE gating modules, normalization operators, and attention operators use BF16 or FP32. The master weights and weight gradients remain in FP32, and the optimizer's moment states use BF16 rather than FP32. Keeping these pieces out of the lowest-precision path protects the parts of the optimization process that are more sensitive or inexpensive enough that their higher-precision cost is acceptable.

This division is easy to misunderstand. “FP8 training” does not mean that the model's entire state is eight-bit. It means that the most compute-dense matrix multiplications and selected stored activations use FP8, while numerically sensitive operations and recovery state retain more precision. A format switch applied everywhere could save memory in a narrow accounting while damaging the update signal that determines whether the model learns. Selective precision is therefore not a compromise added after the optimization; it is the mechanism that makes the optimization viable.

Activation storage is part of the same budget. Because the weight-gradient matrix multiplication consumes activations during back-propagation, DeepSeek-V3 caches many of them in FP8 rather than keeping them in BF16. In the MoE path, activations before the expert up-projections are quantized to FP8 before dispatch. Their smaller representation reduces memory use and communication volume as tokens travel to experts.

Published by DeepMox

FAQ

Why doesn't DeepSeek-V3 use FP8 for everything?

It keeps embeddings, output heads, gating, normalization, attention, master weights, and weight gradients in BF16 or FP32 because these paths are more numerically sensitive or retain optimization state.

How does DeepSeek-V3 control FP8 rounding error?

It uses small-group scales, periodically promotes partial sums to FP32, and overlaps recovery work with matrix operations; higher-precision exceptions protect sensitive interfaces.

What does DeepSeek-V3's FP8 evidence actually show?

At two smaller model scales trained for about one trillion tokens, FP8 models had relative loss error below 0.25% versus BF16; this validates numerical behavior in those comparisons, not a universal training-speed multiplier.

Previous10 minDeepSeek-V3: DualPipe and Topology-Aware Routing

Next12 minDeepSeek-V3: From Pretraining to Usable Behavior

DeepSeek-V3: How Selective FP8 Stays Stable

Making FP8 Work at Scale

Lower precision is a constrained budget

FAQ