Mixtral and Hunyuan-Large: The Serving Bill for Memory and Context

Mixtral 8x7B and Hunyuan-Large turn conditional computation into two different serving problems: the system must make the selected experts available, and it must retain enough attention state to process the request. This chapter follows an inference request across weight memory, routing, expert placement, communication, and context length. The practical goal is to distinguish compute sparsity from storage and data-movement sparsity, then see why a maximum context window is a systems capability rather than a single architecture setting.

What active computation leaves behind

For one Mixtral token, the router selects two experts from eight. The other experts do not calculate a feed-forward output for that token, but their weights still belong to the 47B-parameter model. A serving system must store those weights, place them on devices or make them available through a memory hierarchy, and execute the selected path. The token therefore saves some expert arithmetic without making the larger parameter pool disappear.

Mixtral's report states this distinction directly. It treats active parameters as related to inference compute, while saying that memory costs are proportional to the sparse parameter count. It also calls out routing overhead and increased memory loads when more than one expert runs on a device. A statement such as “Mixtral is a 13B model” would hide the interface between the router and the hardware. The more accurate statement is that Mixtral has about 13B active parameters per token under its routing design while carrying a 47B total pool.

Expert parallelism makes that interface visible. The model can distribute experts across GPUs. When the router assigns a token to an expert, the token's representation is sent to the device holding that expert; after the expert computes, the result must return to the token's original position. The selected computation is therefore coupled to communication and placement. If both selected experts are local, the route may cross fewer device boundaries. If they are remote or heavily requested, the same active count can lead to a different execution pattern.

The system must also combine work from many tokens. A batch can contain tokens that select different experts, so the serving engine has to group, dispatch, and restore those token positions. This is one reason Mixtral's report says sparse MoE models are more suitable for batched workloads with sufficient arithmetic intensity. Batching can create enough expert work to amortize dispatch and memory movement. At low batch sizes, the fixed coordination cost becomes more prominent relative to the expert computation saved.

The router's assignment patterns complicate the picture in both directions. Mixtral reports that consecutive tokens often reuse the same experts in later layers. That locality may allow an implementation to keep an expert's weights or intermediate data useful across neighboring tokens. But the same pattern can over-subscribe one expert or GPU. A route that looks favorable from the perspective of data reuse can be unfavorable from the perspective of load balance.

Hunyuan-Large begins from a different expert path but faces the same boundary. Every token uses one shared expert and one selected specialized expert from sixteen.