What the Numbers Really Tell Us
For arXiv source archives, the important question is no longer whether non-rendered material exists; the two supplied studies make that clear. The harder question is what their headline numbers actually measure. This chapter separates the full-corpus estimate of “interesting” hidden information from the sampled estimate of explicitly sensitive disclosures, so readers can use the evidence without turning different denominators into a false ranking.
Two studies, two denominators
The first study, Hidden Secrets in the arXiv, examines the available source files for 2.7 million arXiv submissions through the end of 2025. Its goal is broad: measure three kinds of hidden information across the file, metadata, and content/text surfaces. It is not limited to confirmed passwords or private documents.
To reduce the influence of standard templates, the researchers define “interesting” content as a dangling file, metadata value, or comment that occurs in no more than two submissions. That choice is an attempt to distinguish material likely added by a particular author from material inherited across thousands of papers. It does not turn every surviving item into a secret. A unique comment may be harmless; a repeated value can still be sensitive. The filter changes the question from “how much material is present?” to “how widespread is unusual material that may have originated in an individual submission?”
At the raw scale, the study reports 12 million dangling files, 144 million metadata key-value pairs, and 644 million comments. After filtering for uniqueness, it finds that 75% of the considered submissions contain interesting comments, 56% contain unique metadata, and 25% contain unique dangling files. The dimensions overlap: comments and metadata occur together most often, and 18% of submissions contain all three. Overall, 88% contain at least one kind of interesting hidden information.
That 88% is therefore a broad prevalence estimate for a defined class of non-rendered or unusual content. It includes low-impact and ambiguous cases, such as template remnants, software information, or a bibliography note, alongside material with clear confidentiality implications. It should not be translated into “88% of papers leaked secrets.” The study itself calls the result a reliable lower bound for the hidden information its method can detect.
The second study, You Have Been LaTeXpOsEd, asks a narrower question with a different design. It analyzes a uniformly random sample of 100,000 arXiv submissions, totaling more than 1.2 terabytes of data in its description of the audit. Its pipeline focuses on sensitive information in comments and unreferenced auxiliary material, using pattern matching, logical filtering, and LLM-based entity extraction.
The sample also has a different usable-data boundary. About 7.7% of downloaded submissions had no source archive and were available only as PDFs; 6.1% had a source archive but no LaTeX files; and 4.6% had LaTeX but no usable comments. The remaining 81.6% formed the principal comment-analysis set. From more than 12 million comments, preprocessing reduced the corpus to about 6.6 million comments, both to remove boilerplate and to make contextual analysis feasible. This is not a trivial technical detail: the rate is conditional on what could be extracted and processed.
On that basis, Qwen-2.5 72B flagged 9,926 papers, or 9.9% of the 100,000-paper sample, producing 13,806 detections because one paper could have several categories. The summarized categories include 6,984 PII findings, 3,386 peer-review-related findings, 3,283 coauthor-conflict findings, 106 network-identifier findings, and 47 explicit-credential findings. These are counts of classified findings under the study’s labels, not counts of people harmed, accounts accessed, or credentials proven active.
The relationship between the two results is best understood as two views of one system. The full-corpus study asks how common unusual hidden material is across three surfaces. The sampled audit asks how often its pipeline can identify content that appears sensitive. The second question is stricter in one sense and narrower in another. It excludes some hidden content that is not classified as sensitive, while its sample and preprocessing leave uncertainty about the entire arXiv population.
Version handling adds another denominator choice. Hidden Secrets treats a paper as affected if an issue appears in any publicly accessible version, rather than counting only the latest version. That is a defensible way to measure exposure over the life of a record, but it can produce a different rate from a design that samples submissions as individual observations. It also cannot reveal what happened in versions whose sources were deliberately removed. The full-corpus result is therefore broad in both subject matter and lifecycle, while the sampled audit is narrower in content and tied to the material its pipeline could process.
The 13,806 LaTeXpOsEd detections are not 13,806 separate papers. One paper can receive more than one label—for example, a comment may contain both personal information and a peer-review discussion. The category totals are consequently useful for understanding the mix of exposure types, but not for estimating the number of affected authors or independent incidents. A sound interpretation keeps the paper-level rate, the detection count, and the category counts separate.
The two lanes should not be added together. Nor should 9.9% be treated as the percentage of the 88% that is dangerous. The studies do not establish that nesting relationship. Their figures answer different questions, use different filters, and attach different meanings to a positive result.
From observation to conclusion
A good way to read these studies is to follow the pipeline from an observed artifact to the conclusion it can support. The unit of observation comes first. A file can be dangling or metadata-bearing. A comment can contain a private conversation. A submission can contain one or many such items. A detection is an output of a method, not automatically an independent incident. A link can be a public reference or an access-enabling route. Confusing these units produces inflated or understated interpretations before any statistical argument begins.
The Hidden Secrets study combines several forms of evidence. It uses build behavior to distinguish required and dangling files, metadata extraction to inspect properties, an abstract syntax tree to locate irrelevant LaTeX content, regular expressions to find structured indicators, and an LLM to classify the meaning of unique comments. The components serve different purposes. A build recorder can show that a file was not read; it cannot say whether an author meant to publish it. A regular expression can find a URL; it cannot determine whether the destination is private. An LLM can recognize a coauthor dispute in prose; it can also misclassify an innocent note. Manual validation is what turns selected candidates into more defensible findings.
The full-corpus method deliberately favors a conservative interpretation of source structure. The authors report that their AST approach avoids labeling content irrelevant when it is needed to compile. They compare recorder output with file reads across 2.32 million compilable papers and report no discrepancies in that validation. At the same time, they acknowledge that parsing errors affected 681,000 submissions, with an average of 9,829 bytes of content disregarded, and that some analyses ended early or could not compile. Rare custom ways of hiding text may also escape detection. The resulting 88% should be read as “at least this much under the defined and validated surface,” not as an exhaustive ceiling.
LaTeXpOsEd makes a different trade-off. It uses a manageable sample and aggressively reduces boilerplate before asking models to interpret comments. This is necessary for cost and processing time, but it means that the 9.9% rate applies to a prepared analytical corpus, not to every byte ever submitted to arXiv. Its authors also identify false positives and false negatives in both heuristics and LLMs. Encoded, fragmented, or unusual secrets may be missed; benign placeholders may be over-extracted. A sampled result can be useful evidence of recurring exposure modes without being a precise census of all possible leaks.
The detection methods themselves explain why the studies find different kinds of things. Pattern matching is strong when the structure is recognizable. It can locate an IP address, email address, URL, or known credential format across a large corpus. Hidden Secrets reports 127 million unique URLs across 905,000 submissions and validates selected API tokens, private keys, and passwords. Those numbers show the reach of the search, not that every URL or token was sensitive. The researchers explicitly use filtering and manual review because a pattern match is a candidate.
Contextual analysis adds a different capability. In LaTeXpOsEd’s benchmark, models were asked to identify PII, credentials, network identifiers, peer-review material, coauthor conflict, and other sensitive content in synthetic comment snippets. Exact-match accuracy required every relevant label to be correct; an at-least-one score counted a result if it found any relevant category. GPT-5 reached 82.7% exact match, while Qwen-2.5 72B reached 80.0% exact match and 91.0% on the at-least-one metric. The distinction matters operationally: a model may be good enough to trigger a human review while still being poor at describing every sensitive element accurately.
That benchmark also shows why model selection is not just an accuracy contest. The authors chose Qwen for the large run because its performance was close to the strongest models on their test while its reported input cost was much lower and it could be used as an open-weight option. Their cost estimates are time-dependent and based on a synthetic benchmark, so they do not guarantee production performance. They do demonstrate that contextual triage can be economically feasible at scale, provided the data-handling model does not create a new disclosure by sending private source material to an uncontrolled service.
Severity adds another interpretive layer. In the sampled audit, 47 explicit credential findings are more urgent than thousands of low-impact identifiers, even though the credential count is smaller. A private document, a GPS field, a peer-review exchange, and a network identifier create different harms and require different validation. The studies classify some access-enabling material as critical, while treating many correlation clues as low or medium severity. Those labels are risk judgments, not proof that every finding caused harm. Current permissions, accuracy, identity linkage, and whether a value is still active all matter.
The evidence is strong enough to support a practical conclusion even with these limitations. Both studies independently show that the public source archive contains more than the rendered paper and that the extra material is common. Both identify mechanisms that a PDF-only workflow cannot see. The exact prevalence of confirmed sensitive disclosure remains less certain because the methods, samples, filters, and validation scopes differ. What would change the conclusion is not a cleaner-looking headline number; it would be evidence that source archives are not actually public, that the detected items were systematically false positives after appropriate validation, or that the methods had examined the wrong population. The supplied evidence does not support those reversals.
The disciplined reading is therefore neither “almost every paper exposes a secret” nor “only one in ten papers has a problem.” It is this: hidden source material is widespread under a broad definition, and a meaningful, repeatedly observed subset carries sensitive content. The size of that subset depends on what is counted and how it is validated. That is precisely why the next engineering question is not whether cleaning tools remove something, but whether they cover the different failure surfaces that produced the findings.