Mixtral and Hunyuan-Large: How a Token Chooses Its Path
This chapter follows one token through the sparse Mixture-of-Experts layers used by Mixtral 8x7B and Hunyuan-Large. The focus is the router: what it scores, how it selects computation, and why the selected path—not the entire parameter pool—determines the direct work for that token. Understanding that path gives us a precise way to read “active parameters” without confusing it with the model's total memory or eventual serving cost.
From a token representation to an expert choice
A Transformer repeatedly turns each token into a numerical representation, often called a hidden state. At a particular layer, that representation normally enters a feed-forward block: a learned transformation that expands and then compresses the information before the next part of the network processes it. Mixtral replaces that one block with a sparse MoE layer containing eight separate feed-forward blocks, or experts.
The router is a small learned function that looks at the current representation and produces one score for each available expert. Those scores are not a permanent label attached to the token. They are a decision made at that layer for that current state. The router keeps the highest-scoring choices and suppresses the rest. In Mixtral, the retained set is the top two of the eight experts.
The next step is not simply “send the token to expert one and expert two.” Mixtral applies a softmax to the selected scores, turning them into weights that sum to one, and uses those weights to combine the two expert outputs. If the current representation is x, the router's selected weights determine how much each selected expert contributes to the layer output. The result returns to the normal Transformer stream and can be transformed again at the next layer.
This sequence has three important consequences. First, the router owns the selection decision, while the experts own the transformations that follow it. Second, the selected experts can change as the representation changes. Third, the other six experts remain part of the model but do not perform their feed-forward calculation for that token at that layer. Sparsity is therefore a property of the execution path, not a claim that the unused parameters have vanished.
The diagram shows two related routes rather than one universal MoE procedure. Mixtral chooses two members of one eight-expert pool and combines their outputs according to the router. Hunyuan-Large has a shared route that every token uses and a separate choice among sixteen specialized experts. The two designs can both be called sparse, but they assign different responsibilities to the always-present and conditional parts of the layer.
It is also useful to distinguish a routing score from an expert's output. The score is a control signal: it decides which computation is eligible and, for Mixtral, how strongly each selected output contributes. The expert does not decide whether it has been selected, and an expert's selection does not prove that it owns a semantic category. The meaning of a route is learned through the interaction of the router, the expert weights, the rest of the Transformer, and the training distribution.
The decision is local to a layer and a token. Across a long sequence, different tokens may use different subsets, and across successive layers the same token may change route. The union of those choices can involve much more of the model than any one token activates. That is why “active per token” is the right boundary for describing the conditional path, while it is the wrong boundary for describing how many weights the serving system must retain or how many expert devices may receive traffic.
Why top-2 routing changes the parameter arithmetic
Mixtral's report gives the model 47 billion total parameters and about 13 billion active parameters per token during inference. The figures are easiest to understand by separating the question “How much does the model contain?” from “How much expert computation does this token invoke?” The first question counts the complete parameter pool, including all eight alternatives in the MoE layers. The second counts the portion used on the selected route, including the two expert blocks that process the token.
If every token were processed by all eight experts, the model would lose the main computational reason for using a sparse layer. By holding the number of selected experts at two, Mixtral can add alternatives to the pool without asking each token to evaluate all of them. The extra alternatives increase the model's available capacity across a sequence, while the top-2 rule limits the number of expert evaluations for any one token.
That does not mean the model has a single fixed set of 13 billion weights that is independent of context. Token A and token B can select different experts. The same token can also encounter different choices at different layers because its hidden representation changes as it moves through the Transformer. The active path is therefore a sequence of local decisions, not a permanent submodel selected once at the beginning of a request.
This is the source of the conditional-computation bargain. A dense 47B model would apply its feed-forward capacity broadly to every token. Mixtral instead makes a larger collection of alternatives available and asks the router to choose two at a time. The report's comparison with Llama 2 70B uses this distinction to emphasize capability with roughly one-fifth as many active parameters. That is a comparison of the reported active computation, not proof that all of the model's costs are one-fifth as large.
The router's weights also matter. They determine not only which experts are used but how their outputs are blended. A strong first choice can dominate the combined result; a second choice can still contribute a useful correction or complementary transformation. The source specifies the softmax over the top-two logits, but it does not say that one selected expert corresponds to a human-readable subject. The router is selecting learned functions for the current representation, not consulting a directory of named specialties.
This local decision explains both the promise and the limit of the 13B figure. For one token, the system avoids evaluating the other expert blocks. For a request containing many tokens, the router can distribute work across several experts, and the selected set can vary from position to position. The serving engine still has to coordinate those assignments and keep the broader parameter pool available. The report therefore uses active parameters as a useful proxy for inference compute, while explicitly warning that memory cost and device utilization require separate analysis.
The arithmetic also should not be confused with a guarantee of quality. Adding experts creates more possible transformations, but the router must learn when each one is useful, and the experts must learn from enough data to make those choices valuable. If the route is poorly balanced, capacity can sit unused in one place while another place becomes a bottleneck. If the training distribution does not support a useful distinction, a larger pool does not automatically become a larger range of reliable behavior.
Hunyuan-Large adds a shared path
Hunyuan-Large uses the same broad idea—only part of a larger expert system is activated—but changes the shape of the choice. Its architecture contains one shared expert and sixteen specialized experts. Each token uses the shared expert and the top-scoring specialized expert. The shared expert is intended to capture knowledge common to all tokens, while the specialized pool provides a conditional choice.
That arrangement changes what the route means. In Mixtral, both selected experts come from the same eight-way pool, and the router weights their contributions. In Hunyuan-Large, one part of the computation is deliberately common and one part is selected. The route is therefore not just “choose the best two specialists.” It is “retain a common transformation, then add one selected conditional transformation.” This is a design answer to the question of how much computation should be shared before specialization begins.
It also changes how to read the 52-billion activated-parameter figure. The number belongs to Hunyuan-Large's particular accounting of one shared expert plus one activated specialized expert and the rest of the architecture. It cannot be placed beside Mixtral's 13B as though both numbers counted the same kind of expert path. Mixtral's top-2 route and Hunyuan-Large's shared-plus-top-1 route have different widths and different meanings even before total model size is considered.
The word “specialized” should be handled carefully. Hunyuan-Large uses it for the role of the sixteen routable experts, and the design is intended to learn domain-specific knowledge. But the supplied report does not establish that each expert becomes a stable, human-interpretable domain module. Mixtral's own router analysis is a useful warning: across several data domains, its expert assignment distributions do not show obvious topic-based specialization, while token patterns and positional locality are more visible in some layers. A route can be structured without mapping cleanly onto a subject a person could name.
Dynamic paths, persistent costs
Because routing is performed on representations at successive layers, expert choice can vary across a sequence. This is important for both capability and systems behavior. A token's initial representation may favor one transformation, while later representations—after attention and earlier feed-forward operations—may favor another. The model can therefore use its expert pool flexibly, but the hardware must handle a changing assignment pattern rather than a single static placement decision.
The changing path does not remove the need to keep the whole model available. Mixtral's report says its inference compute is related to active parameters, but serving memory is tied to the 47B sparse parameter count. It also describes expert parallelism, in which tokens are sent to the device holding the selected expert and the result is returned to the token's original location. That movement creates a boundary between the router's decision and the expert's execution. If many tokens select the same expert, one device can become overloaded while other devices have less work.
This is why an inactive expert has two different statuses. For the current token, it is inactive computationally: no feed-forward output is calculated for that token. For the model as a whole, it remains active infrastructurally: its weights may need to be stored, loaded, placed, or made reachable by the serving system. The distinction becomes especially important at low batch sizes, where there may not be enough parallel work to offset routing and memory movement overhead. Mixtral's report explicitly notes that sparse models are more suitable for batched workloads where arithmetic intensity is high.
Hunyuan-Large's shared path gives its router a different operating boundary, but it does not make that boundary free either. Every token uses the shared expert, and one specialized expert is selected from sixteen. The report's recycle-routing discussion also shows that the training system must cope with capacity limits: when a specialized expert is overloaded, tokens that would otherwise be discarded can be reassigned to another expert with available capacity. That is a training strategy, not evidence that all specialized experts are interchangeable at inference, but it makes clear that routing decisions have consequences beyond the formula for the output.
The path through an MoE layer is therefore a compact mechanism with a large systems shadow. The router scores a current hidden state; the selected experts transform it; their output returns to the Transformer; and the same kind of decision can happen again later. The total pool supplies capacity, while the active route controls direct per-token computation. The next comparison is about the choices inside that route: whether to select several peers from one pool or preserve a shared path alongside a specialized one, and how each choice responds when demand is uneven.