When extracted text lies: fonts, encodings, and quality signals
In pdf-inspector, the difficult question is not where a glyph was painted; the previous chapter established how the parser can find that out. The harder question is whether the character attached to that position is the character the page intended. This chapter diagnoses that gap by following font-local codes through Unicode recovery and then through the quality signals that decide whether native text may be published or should be routed to OCR.
A character code is not automatically a character
A PDF font often uses its own numbering system. The bytes in a text-showing operation identify entries in that font, not necessarily letters in Unicode. A font's character identifier, or CID, may point to a glyph without saying whether that glyph is an “A,” a Chinese ideograph, a mathematical symbol, or something else. A glyph identifier, or GID, can be even more implementation-specific: it identifies a shape in a font program, not the language-level character a reader should see.
A renderer can still draw the correct page from these codes because it knows how the embedded font maps codes to shapes. A text extractor needs a second kind of information: a mapping from source codes to Unicode characters. That is the job of a ToUnicode CMap when the PDF supplies one. The word “CMap” is short for character map, but the important point is its direction: it translates the codes emitted by the page into text that other software can search, index, or summarize.
This creates a failure that is easy to miss. If the mapping is absent or wrong, the parser may know the exact x/y position and width of a run while returning the wrong characters. Geometry can be accurate and semantics can still be false. An empty result is obvious; a printable but incorrectly decoded result can pass through an ingestion pipeline as if it were authoritative.
pdf-inspector treats this as a decoding problem before treating it as a formatting problem. It handles direct character mappings and ranges, determines whether source codes use one or two bytes, and knows that Type0 and Identity-H/V fonts need different treatment from simple fonts. It can use built-in CMaps, encoding differences, and TrueType information as fallbacks. The parser also records whether a page encountered fonts whose GID-encoded text could not be resolved.
The distinction between a primary map and a fallback matters. A fallback can recover useful text when a producer omitted a convenient mapping, but it can also be less authoritative than the font's actual embedded information. A sparse or suspect map may be compared with another source of evidence; a valid semantic mapping should not be discarded merely because a different route returns more entries. The goal is not to choose the most complete-looking table. It is to choose the mapping that best explains the actual font and its painted codes.
Recovering Unicode without overtrusting fallbacks
The CMap parser in pdf-inspector reads the declared code space and the mapping sections that associate source codes with Unicode output. A direct mapping says, in effect, “this code produces this character.” A range mapping describes a sequence of codes whose Unicode values advance together. The decoder supports both one-byte and two-byte source codes, because assuming the wrong width changes how every subsequent byte is grouped.
That detail becomes important in Type0 fonts. A two-byte code is not two independent Latin characters; it is one font-level identifier. If its mapping is missing, treating the bytes as ordinary text can create convincing nonsense, especially for PDFs containing CJK or specialized fonts. pdf-inspector avoids presenting unmapped two-byte CIDs as ordinary CJK text by default. A last-resort passthrough is possible only under a specific condition in which a producer used Unicode values as CIDs and stripped the map. Even then, control characters are not accepted as ordinary output.
TrueType fallback information supplies another form of evidence. Subset fonts can carry opaque names and unusual numbering. A sequential remapping may look plausible while scrambling the relationship between real glyph IDs and characters. When the embedded TrueType map provides stronger coverage than that sequential hypothesis, pdf-inspector can prefer the font's actual mapping. This is a useful pattern for any recovery system: a fallback should be judged by how directly it describes the artifact being parsed, not just by how many entries it contains.
The parser also has to handle imperfect but not entirely useless results. If too many source codes remain unmapped, a CMap decode can return no usable text so that another decoding method can be tried. If a map contains a replacement character, that is evidence of a failed conversion rather than a character the application should silently index. These choices keep the boundary between “we recovered something” and “we recovered the intended text” visible.
The result is still provisional. A successful lookup does not prove that the source producer assigned the right Unicode meaning, and a fallback does not prove that every glyph on the page was covered. That is why pdf-inspector carries font and quality evidence forward instead of collapsing all mapping paths into one indistinguishable string.
The fingerprints of a broken text layer
A broken mapping does not always announce itself with an empty page. pdf-inspector uses several layers of signals because different failures leave different traces.
The clearest symptom is a replacement character: the familiar placeholder inserted when a decoder cannot represent the source. A repeated or dense run of replacements can mark a page as needing OCR. The project also looks for private-use characters and C1 control characters, which often appear when font-internal identifiers have leaked into Unicode output. A string dominated by punctuation and symbols can be garbage even when it contains no replacement character at all.
Another symptom is a dollar sign behaving like a space. A broken CMap can turn a sequence that should read as separate words into a pattern such as Word$Word$Word. The symbol is printable, so a simple “is this valid Unicode?” check will not catch it. pdf-inspector looks at how often dollar signs occur between letters and whether the pattern is repeated well beyond what ordinary financial text would explain.
The project also includes a statistical check for substitution-like corruption. Some broken maps shift characters into other printable ASCII letters. The output then has word-shaped tokens, no control characters, and no obvious replacement symbols. Yet its letter frequencies may retain the shape of natural language while the letters occupy the wrong positions, and case can flip strangely inside words. pdf-inspector uses those properties only when the sample is large enough and predominantly Latin. This protects short labels, non-Latin pages, and legitimate technical text from being treated as proof of a decoding failure.
These signals illustrate the difference between symptom and diagnosis. A high proportion of punctuation is a symptom; the possible causes include a failed mapping, a decorative page, or a specialized notation. A replacement run points more directly to decoding failure, but a small isolated marker may be harmless in an otherwise sound page. The implementation therefore has both span-level checks and page-level accumulation. A localized suspicious span can contribute evidence without a single unusual token automatically condemning an entire text-heavy page.
The diagnosis is not limited to character statistics. A page can contain real native text and still need visual recovery because the main words are vector outlines rather than text-showing operations. It can contain a full-page image with a small native overlay, making the overlay look clean while the page remains visually incomplete. It can contain a GID-encoded font that returns a non-empty string but cannot be trusted. These are different causes—encoding, representation, and coverage—but they share a consequence: native text alone does not support a complete result.
Different causes, one routing consequence
The page-level result in pdf-inspector is designed to preserve that distinction. A page can carry an OCR reason for suspected garbling, scanning, or vector text, rather than a single unexplained “failed” flag. The reason does not claim to identify every missing character. It tells the next stage what kind of evidence was insufficient and why additional visual work is reasonable.
A full-page scan with a small text overlay is a useful example. The overlay may pass the character-quality checks, because those characters really are decodable. The visual background still matters, so the page-level extractor consults the large-image signal as well. This prevents the native path from silently treating a few labels as the whole document. The cause is not a bad CMap; it is incomplete coverage.
Vectorized text is different again. The page may show letters as outlines or paths rather than as text operators. A text extractor cannot recover those letters through the same route it uses for Tj or TJ operations, even though the page can look perfectly sharp. The detector can classify such a page as needing visual recovery. The route is shared because the native representation is insufficient, not because the underlying defect is identical.
This is why “text-based PDF” should not be used as a synonym for “safe to index.” Classification and quality analysis answer related but different questions. Classification looks at the page's structural ingredients and overall type. Quality analysis asks whether the extracted result is meaningful enough for the intended output. A page can be text-bearing but low quality, image-bearing but still have valuable native fragments, or mixed in a way that requires a page-specific decision.
Region extraction makes the same principle more precise. If a layout system supplies bounding boxes, pdf-inspector can inspect the native text inside each region and return a needs_ocr flag when that local result is empty or unreliable. One page may therefore keep its native paragraphs while sending an image-backed table or a garbled region to OCR. This is a more selective boundary than declaring the entire page unusable, but it depends on the caller providing meaningful regions and on the coordinate frame matching the rendered page.
Why suppression can be safer than a partial answer
When native text fails its quality checks, the public native path suppresses the unreliable Markdown instead of presenting it as if it were a normal result. That choice can look conservative until the alternative is considered. A blank page is visible to an operator or downstream rule. A plausible-looking substitution cipher can be searched, summarized, and cited before anyone notices that its characters do not match the page.
The OCR-aware path has a different responsibility. It may retain a clean native fragment privately so that a later OCR result can be compared with it. This is useful when the native layer contains a trustworthy heading or paragraph but misses an image-backed body. The fragment is not automatically returned as complete content; it becomes one candidate in an evidence comparison. The public result can then preserve the stronger native text, replace it with OCR, or combine both when OCR adds material coverage.
That separation also prevents an overly aggressive fallback. Routing every suspicious page to OCR may recover more pixels, but it can displace exact native text with recognition errors and adds runtime work. Keeping the native candidate available lets the later stage ask whether OCR is weak, duplicative, or genuinely complementary. The decision belongs after diagnosis because the cause determines what kind of recovery is useful.
The quality signals have limits. They are heuristics, and an unflagged page is not proven correct against its image. Large statistical checks have better evidence than a short label; non-Latin and highly specialized material may not fit the same signals; and a page that passes decoding can still be visually incomplete. needs_ocr therefore means “the implementation judged native recovery unreliable or insufficient under its signals,” not “OCR will certainly solve the page.”
The practical lesson is simple: decoding is part of extraction, not a cleanup step after extraction. pdf-inspector first tries to connect font-local codes to Unicode through evidence about the font. It then tests the resulting text for symptoms that the mapping or page representation is misleading. Only after those causes are separated does it decide whether native Markdown should be exposed, whether a page should enter selective OCR, or whether the result should remain open to a stronger hosted path. The next checkpoint will combine this trust judgment with classification, geometry, and page-level output so that the native boundary can be evaluated as a whole.