From Samples to Supported Types: How Magika Earns Coverage
Magika's coverage is not a property that appears automatically when a neural network is attached to a file reader. It is the outcome of a data lifecycle: samples are associated with canonical labels, checked, divided into different experimental roles, and eventually turned into a versioned model and output contract. This chapter follows that lifecycle so readers can distinguish what Magika has been trained to recognize from what the project merely tracks, and can read an accuracy claim without confusing a balanced experiment with their own file population.
A content type begins as a data-quality decision
The initial Magika study began with a practical question: which formats are common and important enough to support first? The researchers did not attempt to enumerate every format in existence. They selected a diverse set of 113 canonical content types, covering text, source code, documents, media, archives, executables, and other files. The paper describes a final collection of about 26.5 million samples, with approximately 24 million used for training and about 1.2 million each for validation and testing.
Those numbers matter, but the sources matter just as much. GitHub and VirusTotal see different parts of the file world. GitHub supplies abundant text-heavy material: programming languages, configuration files, and documentation. VirusTotal contributes more binary-oriented samples, including archives, executables, and office documents. Using both gives the dataset a wider base than either source would provide alone. It does not make the dataset neutral. Each platform has its own population, and the mixture is still a research choice rather than a measured copy of one production environment.
The researchers built a stratified dataset in which each canonical type had equal representation. That decision makes comparisons legible. If one class dominates the test set, a system can appear accurate while performing poorly on rare classes. Equal representation lets the study ask how well each type is supported on its own terms. But it also changes the meaning of the average. A reader whose workload is mostly PDFs, images, or source files should not assume that the reported mean is the probability of a correct answer for that workload. Their estimate needs to weight the per-type behavior by the frequencies they actually see.
Before a sample entered the dataset, it passed several simple checks. The researchers required a minimum file size of 16 bytes for the study samples. For binary files, magic bytes supplied necessary—but deliberately not sufficient—evidence for an extension claim. A sample labeled as a Windows executable, for example, was expected to begin with MZ, but the presence of MZ alone did not prove that it was a valid executable. For text files, the check was whether the content could be decoded as text. VirusTotal samples also had to avoid antivirus flags, because a malicious or heavily obfuscated sample carries a higher risk of being mislabeled or atypical in ways that would distort the intended ground truth.
Notice what is absent from this validation process: the researchers did not simply require existing content-type tools to agree. Doing that would make the new detector imitate the consensus of the old detectors, including their shared errors. Trusting one incumbent above the others would have a similar problem. The heuristics were intentionally straightforward. They helped remove obvious mismatches without making the benchmark a disguised vote by the systems it was supposed to evaluate.
The dataset also needed examples that did not belong to any named class. For UNKNOWN, the study generated random byte sequences of arbitrary length. For TXT, it generated random strings of text characters. Those examples teach a classifier that the correct answer is sometimes not one of the distinctive formats in its catalogue. In a multi-class system, this is a crucial escape hatch: without an unknown-like class, an unfamiliar binary must be forced into whichever supported label happens to receive the highest score.
The synthetic classes did not dramatically improve the measured accuracy on the study's ordinary dataset—the paper reports an effect smaller than 0.05 percent—but that is not the only reason to include them. They establish a meaningful behavior for out-of-scope-looking input. A model that can say “generic text” or “unknown binary” has a way to express weak evidence instead of manufacturing specificity. That statistical choice later becomes a tool-level fallback, but it starts here as a dataset-design choice.
The split between training, validation, and testing protects the meaning of the final number. Training examples update the model's parameters. Validation examples help choose hyperparameters and tune thresholds. Test examples are held out from those decisions and are used once as an unbiased measure within the study's design. If the same samples guide training, threshold selection, and final reporting, the resulting score can describe familiarity with the benchmark rather than performance on unseen examples. Magika's paper explicitly keeps the 1.2-million-sample test set out of model training and tuning.
The lifecycle in the diagram is not a promise that every relationship is equally strong. A validation rule can reject an otherwise interesting sample; a canonical label can simplify several file extensions into one class; and a model version can expose a tool label that is not a raw neural class. The point is ownership. Data curation determines what the model is asked to learn, the split determines what evidence can support a metric, and model configuration determines what users can actually receive.
This design also explains why the research paper's dataset should not be read as a timeless inventory. It is a carefully constructed first benchmark, not a census of all valid files. The paper acknowledges both the incomplete list of formats and the possibility of source bias or mislabeled samples. Its size reduces random noise in many measurements, but size alone cannot remove a blind spot that never entered the collection.
A new model version changes the coverage contract
The current repository shows Magika continuing beyond that first experiment. The project README describes a later corpus of approximately 100 million samples across more than 200 content types, and the current standard_v3_3 model documentation lists 216 possible outputs for Magika the tool. These are current project summaries, not a replacement for the paper's detailed 113-type test protocol. The safe conclusion is that the supported surface has expanded substantially; the safe uncertainty is that the selected sources do not reproduce a complete current v3.3 evaluation methodology with the same level of detail.
The difference between a model version and a broad knowledge base is central. A repository may track many content types for research, metadata, or future work. A particular model can be trained on only a subset, and the tool can add structural or generic outputs such as empty, directory, symlink, txt, or unknown. The presence of a label in a generated enumeration is therefore not enough to establish that the default model can return it. The model-specific documentation and output APIs are the authoritative places to check the current contract.
Version changes can target confusing boundaries rather than simply add more names. The standard_v3_3 changelog reports a better balance between JavaScript and TypeScript examples, associated with an increase in reported TypeScript accuracy from 85 percent to 95 percent in that model update. It also records synthetic datasets containing non-ASCII characters for simple text and JSON, along with more threshold tuning. These changes address the evidence the model sees and the confidence with which the tool interprets it. They show why “support” is not just a count of rows in a table.
A new class can require a fresh output layer and a new round of training. The paper describes adding a content type as a process of sourcing enough representative examples and retraining, with the training pipeline automated even though a run can take several days. Its experiments found that binary classes could reach robust performance with around 10,000 samples per type, while textual classes needed more examples to raise their average F1. The figures are observations from the initial study, not a universal recipe. They nevertheless reveal the maintenance exchange: the work moves away from authoring a carefully ordered signature and toward collecting representative data, protecting label quality, calibrating thresholds, and keeping clients compatible with the new model.
That exchange can improve coverage without making it free. A signature author can often add one rule and inspect whether it collides with existing rules. A learned model must absorb variation in examples, preserve neighboring distinctions, and be evaluated against cases that are easy to confuse. The JavaScript/TypeScript update is a concrete illustration. The formats are related enough that their training balance affects the boundary between them; more examples and revised balance can change one class's results without changing the entire product's purpose.
The cost is also uneven across text and binary content. Binary formats often have distinctive boundary bytes, while textual formats share characters, syntax, and conventions. That does not mean text is inherently impossible to classify. It means the model needs enough varied examples to learn which patterns distinguish a language or document type from generic text and from its close neighbors. The paper's training observations and the current changelog's targeted text updates point in the same direction: data diversity is part of the mechanism, not merely a larger number on a training dashboard.
The model's output space must evolve with these decisions. If a new class is trained but the tool's metadata, thresholds, or client contract is not updated, the neural output and user-facing result can diverge. Conversely, a tool may expose a generic or structural label that the neural model never predicts because some inputs bypass inference. A useful version check therefore asks three separate questions: what the model can emit, what the tool can return after post-processing, and what the selected client actually bundles by default.
This is why the model changelog is more informative than a single coverage total. The v3.3 entry reports 216 tool outputs, approximate 99 percent average accuracy, and approximately 2 milliseconds of inference on the stated benchmark CPU. Earlier entries describe similar accuracy with changing speed and coverage, including the major expansion from the first model to the 200-plus-type generation. Such summaries show direction and intent. They do not prove that each type has the same error rate, that every client has identical startup behavior, or that a new workload will reproduce the benchmark.
A reader evaluating a model update should therefore look for changed boundaries. Did the version add types, split a previously broad class, rebalance confusing categories, improve non-ASCII text, or retune thresholds? What is the current output list, and how does it differ from the previous one? Which files in the reader's environment are common enough to dominate the practical error rate? The repository provides useful clues, but the answers for a high-stakes deployment still require testing against representative local samples.
The deeper lesson is that coverage is a contract between data and use. A model can be excellent within a carefully selected set and still be incomplete for a particular organization. A broad model can be valuable because it reduces the number of files that fall into a generic bucket, yet a generic result remains the correct response for unsupported or ambiguous input. Treating the fallback as a defect to be eliminated at all costs would recreate the original problem in a more sophisticated form: a system that must always name something, even when its evidence does not justify the name.
Magika's data lifecycle thus leads to the same uncertainty-aware principle as its byte extractor. The detector does not earn trust by claiming to know everything. It earns useful coverage by making the scope of what it learned visible, testing that scope against held-out evidence, and retaining a path for “none of the above.” With that model in place, the next task is to connect sampled bytes, trained scope, and raw scores into one complete picture before deciding how a final label should be trusted.