What Counts as a Better Global Answer?

GraphRAG is meant to improve answers to global sensemaking questions, but “better” has no single obvious meaning when a question asks for themes, perspectives, or trends across a corpus. The focus here is the paper’s evaluation design: how it generates realistic corpus-wide questions, compares answers across several qualities, and uses extracted claims as a second measurement lens. Understanding those choices is necessary before treating any later win rate as evidence of general superiority.

A benchmark built from activities, not answer keys

For a narrow factual question, evaluation can sometimes compare a system’s answer with a known answer or check whether it retrieved the right passage. A global question is different. There may be several reasonable summaries of a collection, and no short answer key can list every acceptable theme or relationship. The paper therefore treats evaluation as a question-design problem first: it constructs questions that require broad understanding of a particular corpus, then compares how different systems respond.

The procedure begins with a short description of a dataset and its purposes. An LLM uses that description to imagine potential users of the collection. For each user, it proposes tasks that the person might perform. For each user-and-task combination, it generates high-level questions that require understanding of the entire corpus and do not depend on retrieving one low-level fact. In the experiments, the authors use five users, five tasks per user, and five questions per combination, producing 125 questions for each dataset.

This activity framing makes the target more concrete than a generic prompt such as “summarize the data.” A question is connected to a reason someone might consult the collection. In the podcast-transcript example, a technology journalist is framed as wanting to understand how technology leaders view policy and regulation. The resulting questions ask how guests perceive privacy laws, whether they discuss ethical trade-offs, what policy changes they suggest, and whether technology companies collaborate with governments. These questions require the answer to gather viewpoints across episodes rather than quote one speaker.

For the news-article dataset, the example user is an educator incorporating current affairs into a curriculum. The task concerns health and wellness, and the questions ask what topics can support health education, how articles address preventive medicine, whether articles contradict one another, what public-health priorities appear in coverage, and how the collection can support health literacy. A response that names one article may be relevant, but it does not fully address the activity. The question asks for patterns across the collection and, in some cases, relationships among conflicting perspectives.

The procedure also tries to reduce a particular form of leakage. The authors do not generate the questions directly from the same corpus that is later indexed and used to answer them. Instead, the questions are generated from a description of the corpus and its use cases. The intention is to avoid allowing the question generator to copy themes that the evaluated systems will later retrieve or summarize.