Mixture-of-Experts: Large to Store, Sparse to Compute

This chapter uses Mixture-of-Experts models to test the central decision in llmfit: what must a machine store, and what work must it perform for each generated token? An MoE model can contain many expert networks while routing each token through only a subset of them. Readers need to keep total parameters, active parameters, VRAM placement, system-RAM placement, and per-token traffic separate; otherwise an MoE recommendation can look either impossibly large or misleadingly small.

The important correction is simple but easy to lose. Total parameters describe the model that exists and the weights that may need to be stored. Active parameters describe the part of that model used for a token, which is often more relevant to decode speed and the base quality tier. These numbers are related, but they do not answer the same question.

The two parameter counts that matter

A dense model applies the same network structure to every token. In an MoE model, a router selects a subset of experts for each token. Each expert is a learned feed-forward network, and the model also has components shared across tokens: attention, routing, embeddings, output heads, and sometimes a shared expert. The router’s decision makes the per-token calculation sparse, but it does not make the unused experts disappear from the model’s storage requirements.

Mixtral-8x7B makes the distinction concrete. The catalog records about 46.7 billion total parameters and roughly 12.9 billion active parameters. The first number tells us that the model’s complete set of expert and shared weights is substantial. The second tells us that the per-token path uses a much smaller active set. If we used 12.9 billion for every memory question, we would understate the storage pressure. If we used 46.7 billion for every decode-speed question, we would overstate the amount of expert work performed for each token.

llmfit assigns those numbers to different parts of its analysis. Fit and weight-memory pressure use the full model size. Decode speed uses active parameters when the record identifies the model as MoE, because the estimator is trying to model the work read or computed for one token. The base quality tier also uses active parameters, reflecting the project’s judgment that the effective per-token capacity is closer to the active model than to the sum of every dormant expert. This does not declare an MoE equivalent to a dense model of the same active size. The inactive experts still contribute to storage, routing behavior, and the model’s learned capability.

The catalog contains the same distinction in newer and much larger designs. Qwen3-30B-A3B is recorded at roughly 30.5 billion total parameters and about 3.3 billion active parameters. DeepSeek-V3 is recorded at about 684.5 billion total and 37 billion active. Those ratios show why the shortcut “parameter count equals speed” breaks down for sparse architectures. They also show why “active parameters equal memory requirement” is just as wrong.