Mixtral and Hunyuan-Large: From Data and Training to Capability

For Mixtral 8x7B and Hunyuan-Large, routing creates available capacity but does not decide what that capacity knows. This chapter follows the training lifecycle that fills the expert pool, chooses its scale, extends its context, and turns a base model into an instruction-following model. The practical judgment is to separate an architectural effect from a data or optimization effect: a benchmark result belongs to the complete training path, not automatically to sparse routing.

Data is part of the model design

The two reports describe different answers to the question of what the expert pool should learn. Mixtral's account is relatively compact. It says the model is pretrained on multilingual data and that multilingual material was upsampled compared with Mistral 7B. The report connects this extra capacity and data mixture to strong results in French, German, Spanish, and Italian. The important relationship is not simply “more experts produce multilingual ability.” It is that a conditional architecture was paired with a training distribution intended to make multilingual capacity useful.

Hunyuan-Large describes a much more elaborate data program. It reports approximately 7 trillion pretraining tokens, including nearly 1.5 trillion tokens of synthetic data. The natural and synthetic portions are not treated as interchangeable. The team says it filters data for writing quality, educational value, toxicity, and privacy-sensitive content, then uses category labels to adjust the mix. Synthetic data is emphasized for mathematics, coding, low-resource areas, and high-educational-value material—fields the report says can supplement what naturally distributed text provides.

The synthetic-data process has four stages. First, high-quality webpages, question-answering data, code repositories, books, and other resources provide seeds for generating varied instructions. Second, an instruction-evolution process improves clarity and informativeness, expands low-resource coverage, and increases difficulty. Third, specialized models generate responses. Fourth, a critique model and self-consistency checks filter the resulting instruction-response pairs, removing low-quality or inconsistent examples.

This process matters because the model is not merely memorizing a fixed collection of natural documents. The training team is trying to shape which kinds of reasoning, coding, and educational responses appear often enough and clearly enough for the model to learn them. The report attributes capability gains partly to this data quality and synthesis, but the supplied evidence does not isolate the effect of each stage in a controlled ablation. The pipeline is a design explanation, not proof that one particular filter caused one particular benchmark improvement.

Tokenization is another point where data and systems meet. Hunyuan-Large uses a 128K-token vocabulary: 100K entries from the tiktoken tokenizer and 28K added to improve Chinese-language support. The report says its compression rate improves from 2.78 to 3.13 characters per token compared with the Llama 3.1 tokenizer. Better compression can reduce the number of tokens needed to represent some text, which matters for training and context budgets. But the reported statistic is a tokenizer comparison, not a guarantee of lower end-to-end cost for every language, prompt, or hardware path.

The contrast with Mixtral is instructive. Mixtral's report foregrounds multilingual upsampling; Hunyuan-Large foregrounds a much larger synthetic-data and filtering pipeline plus a tokenizer explicitly adjusted for Chinese. Neither approach is a generic substitute for the other.