Teaching a Model to Read Documents
Granite Vision's document ability depends on more than giving a model many page images. This chapter examines how IBM Research turns document pages into structured training evidence, then uses synthetic questions, public datasets, and preprocessing to teach different visual-language behaviors. The practical goal is to understand why DocFM and its derived tasks are central to Granite Vision's specialization, while keeping dataset scale separate from proof that every learned answer will be correct.
From document pages to a training foundation
A raw image contains pixels, but a training pipeline needs ways to select, describe, compare, and question what those pixels contain. Granite Vision's data program begins with DocFM, an IBM effort built from approximately 85 million document pages extracted from unique PDF documents sourced from Common Crawl, Wikipedia, and ESG reports. The paper describes DocFM as a foundation for generating later datasets, not as a claim that every one of those pages becomes a direct instruction example for the final model.
DocFM was created with Docling, an open-source document conversion toolkit. Docling enriches a page in several ways that matter for visual understanding. It extracts text together with positional metadata, identifies layout elements such as headings, paragraphs, lists, captions, form elements, pictures, formulas, and footnotes, and estimates the order in which a reader should encounter those elements. It also detects the page's primary language and classifies visual figures such as charts, images, and diagrams.
These signals change what can be done with a page. Text alone might tell a question generator that a report mentions revenue, but position can indicate whether the word belongs to a title, a table cell, or a footnote. Reading order can distinguish a two-column page from a single linear stream. Figure classification can route a page toward a chart question or an image-captioning task. A form element can become the focus of a field-extraction example rather than being treated as ordinary prose. The representation is useful because it describes the page's structure while remaining connected to the original visual content.
The pipeline also applies selection before training. The Common Crawl portion was assembled by identifying trusted domains likely to contain relevant PDFs, searching multiple archive indexes for PDF media types, and attempting to download the files. The paper reports millions of discovered and converted documents and pages, but the important design point is the filtering sequence: source discovery, successful retrieval, duplicate handling, conversion, and then selection of subsets for particular visual tasks. A large corpus is not a single undifferentiated pool. Each stage changes what evidence is available and what quality assumptions can be made about it.
Granite Vision's document mixture therefore covers several kinds of relationship rather than one broad label called “OCR.” It includes questions about document content, tables, charts, infographics, diagrams and flowcharts, scene text, grounding and text localization, captions and reasoning, user-interface screens and code, and structural parsing. General-image collections are added as a separate complement. The purpose of this breadth is not to make the model memorize a catalog of datasets. It is to expose the language model to different ways that visual arrangement can determine the answer: a field may need extraction, a chart may need comparison, a diagram may need following, and a screen may need structural interpretation.
That diversity also creates a balancing problem. A model trained mostly on one task may become good at that task while developing a narrower idea of what a visual question looks like. Mixing document QA, chart extraction, grounding, and general image conversations can improve coverage, but the mixture must still connect each instruction to the right visual evidence. The paper gives an overview of categories and counts, yet it does not provide a complete ablation showing how each category changes the final model. The data program supports the specialization thesis; it does not make every category equally influential by default.
Synthetic questions make visual structure teachable
The most distinctive part of the data program is its attempt to generate questions that refer to what a document looks like, not only to text that can be copied from it. The DocFM-VQA pipeline selects 2.4 million pages from DocFM and uses Mixtral 8×22B to generate 20 million question-answer pairs. Instead of passing the raw page to the question-generating language model, the pipeline builds a verbalized representation: extracted text is supplemented with descriptions of visual elements. Charts are converted to tables with DePlot, images receive captions from BLIP2, and tables are represented in Markdown through TableFormer.
This procedure creates an interface between visual structure and language-based question generation. A chart-to-table conversion gives the generator access to values and organization. An image caption supplies a description of what a figure depicts. A Markdown table makes rows and columns explicit in a form a language model can process. The generated question can then be paired with the original visual material so that the multimodal model is trained to answer from the page rather than from a text-only report.
The design addresses a specific weakness in ordinary synthetic QA. If a language model sees only a table of numbers, it can ask about a value but has no knowledge of how that value was drawn. It may not ask about a colored line, a marker, a label placed to the left of a figure, or a relationship that is visible only in the layout. By adding visual descriptions to the generation context, the pipeline can prompt questions about elements that would otherwise disappear during text extraction. It does not solve visual understanding automatically; it makes more relevant questions available for training.
Charts receive an additional treatment because they combine appearance, structure, and arithmetic. A common synthetic method is to give a language model tabular data, generate simple questions, and render a chart afterward. The Granite Vision paper identifies two problems with that approach. The question generator does not know the chart's visual properties, such as marker shapes or colors, and it tends to produce easy questions about one data point. Such data can improve scores on similarly synthetic tests without transferring as well to human-written questions.
DocFM-VQA-Charts changes the input to the question generator before rendering. The pipeline adds a random marker and color for each data series, so the generated question can refer to visible appearance. It also adds calculated rows and columns containing operations such as maximum, minimum, sum, average, or a randomly selected addition or subtraction between rows or columns. Those derived values encourage questions that require comparison or arithmetic and provide more reliable answers for operations that a language model may calculate inconsistently. The resulting dataset contains about 5.5 million questions on roughly 500,000 synthetic charts.
The important mechanism is not the randomization by itself. It is the attempt to make the supervision contain the same kinds of cues that the model will later need to use. A question about the red series requires the training example to preserve color as a meaningful variable. A question about the largest value requires the example to connect a visual mark to a numerical comparison. A question about a derived operation requires the answer to be grounded in an explicit calculation rather than in a vague description of the trend. Data design is being used to define the relationship between an image and a correct response.
Flowcharts require a different kind of structure. Their meaning is distributed across nodes and directed edges, so a model may need to follow a process rather than locate a single word. Granite Vision's synthetic flowchart pipeline uses Mixtral 8×7B to identify business domains and industries, generate process types, and create JSON descriptions containing nodes and edges. Those descriptions are rendered as visual flowcharts and then used to generate question-answer pairs. The paper reports 17,000 JSON representations and rendered flowcharts producing 76,000 QA pairs. This is a smaller collection than the document and chart programs, but it targets a distinct relationship: how visual connectivity represents a sequence or decision path.
The same principle appears in chart extraction data. The paper describes 550,000 chart images generated from tables taken from DocFM and public stock-market data, with varied chart types and requested output formats such as Markdown, HTML, or JSON. Some charts are placed back into their source-document context. That context matters because a chart can be understandable in isolation but ambiguous when its title, surrounding paragraph, or neighboring table is part of the question. The dataset therefore gives the model examples of both the visual object and the document around it.
Synthetic generation has a clear cost-quality trade-off. Granite Vision's authors explored using leading visual-language models to filter potentially hallucinated question-answer pairs in DocFM-VQA, but did not include that filtering in the current version because the observed performance benefit did not justify the computational cost. This decision makes the dataset cheaper to construct at the chosen scale, but it also means that synthetic supervision is not automatically validated by a second model. The correct conclusion is not that the generated answers are unusable; it is that their quality is a design boundary the paper explicitly leaves visible.
Public datasets broaden the same training interface. The paper adds data for scene text and OCR, table and chart QA, diagrams, grounding, UI screens and code, structure parsing, and general image questions. These sources can broaden the mixture beyond the particular synthetic generators described above. But public availability does not remove the need for governance. The preprocessing pipeline reports legal review for potentially problematic or personally identifiable data, removal of data with legal or unclear licensing concerns, CSAM removal using NSFW detection, face blurring to obscure visual PII, and deduplication through exact pixel matching or perceptual hashing. Records with identical text and similar images were removed.
Those safeguards are part of the data system's operating boundary, not a certification of the final model. The paper does not give complete retention and removal statistics for each dataset, detection error rates, or an independent compliance audit. A face-blurred training image may still be unsuitable for a particular use, and a deduplicated corpus can still reflect the biases or omissions of its source material. Similarly, a synthetic chart can teach a useful relation while failing to represent the typography, noise, or unusual layout of charts encountered in a real organization.
The result is a more useful way to read Granite Vision's data scale. Millions of pages and questions matter because they provide room to vary document forms and task instructions. They do not substitute for coverage of the documents an organization actually receives, nor do they prove that every visual relationship has equal quality. The data program is strongest when understood as an attempt to make supervision match the structure of the work: preserve the page, expose its layout, generate questions about visual evidence, and add tasks that require extraction, comparison, localization, or process understanding.
The visual architecture described earlier supplies the capacity to preserve page detail. This data program supplies reasons for the model to use that detail. The remaining question is how those two ingredients are aligned in the weights: which part learns the visual-language connection first, which part learns task behavior later, and what is retained or reset along the way. That is where the compactness thesis becomes a training curriculum rather than a description of a dataset.