What the Benchmarks Measure—and What They Leave Out
Granite Vision's evaluation asks whether its coordinated visual and training design produces measurable document behavior, not merely fluent descriptions of images. This chapter focuses on how to read the paper's benchmark protocol, its reported DocVQA and ChartQA results, and its table and chart extraction tests. The practical value is a calibrated judgment: these measurements provide evidence about selected capabilities, but they do not by themselves establish reliability, cost, or correctness in an enterprise workflow.
The benchmark is a measurement protocol, not a single score
A benchmark is a controlled question about a model, not a complete description of the model. Granite Vision's paper uses the standardized lmms-eval package and concentrates on document-related tasks while also reporting several natural-image results. The authors state that they did not use test-time prompt tuning or chain-of-thought optimization. That detail matters because a score can reflect not only the learned model but also extra prompting or inference procedures. Here, the reported setup is intended to measure the model without those added test-time techniques.
The document benchmarks probe different parts of the page-reading problem. DocVQA asks a model to understand and extract textual information from document images. ChartQA requires questions about charts that combine visual interpretation with logical reasoning. TextVQA focuses on reading and reasoning about text within images. AI2D uses science diagrams, while InfoVQA combines text, graphics, layout, and data visualizations in infographics. WebSRC uses screenshots and structural information from web pages. These tasks overlap, but they are not interchangeable. A model can be good at finding a printed word and still struggle to follow a diagram or relate a chart value to the correct label.
LiveXiv adds a different kind of test. It uses recently published arXiv papers and asks questions about figures and tables in scientific manuscripts. A constantly refreshed source makes it harder for a model trained on older web data to have encountered the exact test material. That helps reduce one form of static-test contamination. It does not make the benchmark contamination-proof, and it does not turn a scientific-paper workload into a complete test of invoices, forms, or internal business documents. Its value is more specific: it checks whether multimodal performance holds on newer visual content that is less likely to have been memorized from a fixed benchmark.
The paper's Table 1 reports Granite Vision at 0.88 on DocVQA and 0.86 on ChartQA in the small-model comparison. The authors describe these as leading results on key document benchmarks and emphasize that they are competitive with much larger proprietary models. That is meaningful evidence for the paper's central thesis. A model in the compact size category is not being evaluated only on easy image descriptions; it is performing strongly on tasks that require reading and reasoning over documents and charts.
The wording of that conclusion matters. “Leading in this comparison” is not the same as “best on every document task.” The table groups models by size and reports multiple benchmarks, with missing values for some systems. The paper also marks some rows with an asterisk. For those models, the reported numbers came from their original publications or publicly reported benchmark results rather than from the same controlled lmms-eval run. The Granite Vision result may be directly measured in the paper while a competitor's result may have been produced under a different prompt, preprocessing pipeline, or evaluation version.