From Model Output to a Ranked Review Queue
The BIS procedure’s next problem appears after the first comparison has been made. A large language model can identify a plausible difference between a rule and a prospectus, but identical inputs can produce different outputs. This chapter explains how the procedure uses independent repetition and recurrence to order candidate divergences, then asks the harder operational question: whether that order improves expert review without hiding material issues. The answer requires treating the queue as a measured interface between a variable model and accountable supervisory expertise.
Five independent views of the same comparison
The basic comparison is already structured: the model receives the applicable regulatory materials, the prospectus, and a prompt that asks it to identify corresponding provisions and possible changes in scope, conditions, or consequences. But structure does not eliminate stochasticity. The same model can emphasise one passage in one response and notice another in a later response. A single run therefore gives a snapshot of candidate generation, not a stable map of every difference worth reviewing.
The BIS procedure addresses that instability by repeating each regulatory-package–prospectus comparison five times. Each run uses the same texts and the same prompt, and no run has access to the outputs of the other runs. That independence matters. If one response were shown to the next, later answers could simply reproduce an earlier candidate, making apparent agreement partly an artefact of the process. Separate runs instead provide five opportunities for the model to notice, describe, or overlook a possible divergence under the same input conditions.
The outputs then pass through two transformations. First, equivalent findings are consolidated. Five responses may describe the same underlying difference with different wording, and counting each description separately would exaggerate the number of issues. Consolidation tries to treat those responses as one candidate while retaining the information that it recurred. Second, the procedure ranks the consolidated candidates by recurrence: how many of the independent runs returned an equivalent finding.
The reported figures show why these transformations are necessary. Across the exercise, the procedure returned 576 individual divergences. After equivalent findings within and across runs were merged, 193 distinct candidate divergences remained. Of those, 79 appeared in at least four of the five runs and were retained as recurrent candidates for the reported queue. The sequence is not a claim that the model found 79 breaches. It is a way of reducing a large, variable set of model observations to a more manageable set of repeated signals.
The denominator is important at every step. The 79 candidates are not 79 out of all relevant provisions, nor are they 79 cases independently established by experts. They are 79 of the 193 distinct candidates produced by this procedure that crossed a recurrence threshold of at least four runs. The 193 themselves came from merging 576 individual outputs. Each number describes a stage in the screening pipeline, not a different estimate of legal risk.
Recurrence is best understood as a prioritisation signal. If equivalent candidates appear repeatedly under the same supplied texts and prompt, a reviewer may reasonably choose to inspect them earlier than a candidate that appeared once. Repetition suggests that the model’s comparison process is sensitive to that textual relationship under those conditions. It does not show that the difference is important in law, that it is impermissible, or that the model has captured every important issue.
A candidate that appears in only one run may still matter. It may depend on a subtle cross-reference, an unusual fact pattern, or a provision whose significance is difficult to articulate consistently. Conversely, a high-recurrence candidate may be repeated because the wording strongly attracts the model’s attention while turning out to be authorised or immaterial when an expert reads the broader framework. The ranking creates an order for inquiry; it does not create an order of legal seriousness.
The procedure’s reported model also matters to how the result should be read. The BIS reports using GPT-5.5 Pro and notes that the prompt could be used with another state-of-the-art LLM, but that results will vary with the model’s capabilities. Recurrence is therefore conditional on more than the documents. It reflects the interaction of the supplied texts, the prompt, the model, and the consolidation method. A change to any of those elements can change the queue.
This is a useful form of uncertainty management because it makes variability visible rather than pretending that one response is definitive. It is not a statistical confidence interval. Five runs do not establish a probability that a candidate is correct, and the four-of-five threshold does not have a general meaning outside the procedure and evidence used to calibrate it. The count tells the reviewer how consistently the model returned an equivalent candidate in this particular setup.
There are two reasons consistency can mislead. The model may repeatedly focus on an obvious phrase even when a lawyer would find the phrase authorised or immaterial. Or the model may notice a subtle issue only once because the relevant relationship is difficult to express consistently, even though the issue deserves attention. The consolidation step adds its own challenge: responses that describe the same underlying difference should be merged, but descriptions that sound alike may concern different requirements. A queue is therefore shaped by generation and by interpretation of the generated responses. Both stages need to remain inspectable.
A useful way to read a recurrence count is as a question about the model’s behaviour, not as an answer about the document. “Returned in four runs” means that the supplied setup repeatedly produced an equivalent candidate. It invites an expert to ask why the wording attracted attention and whether the reason survives broader reading. It does not answer whether the rule was breached. “Returned once” means that the candidate was less stable under that setup. It does not answer whether the issue can safely be ignored. The ranking is valuable when it tells experts where to begin while preserving reasons to look beyond the beginning.
A queue is an interface between model and expertise
The practical value of recurrence appears in the work it changes. Without a queue, experts begin with a large body of rules and prospectuses and must both locate possible differences and interpret them. With a queue, they can start from recurrent clause pairs linked to the source texts. The model performs the initial comparison; the queue organises attention; the supervisor or legal specialist decides what the difference means for the task at hand. The interface is useful only if each part keeps its boundary.
The first boundary concerns evidence. A ranked candidate should remain attached to the regulatory and prospectus passages that produced it, along with its divergence type, recurrence, and explanation. This lets the reviewer inspect the model’s match instead of relying on a summary. They can check whether the passages address the same requirement, read the surrounding definitions and exceptions, and determine whether the apparent difference persists in the full documents. The queue should shorten the path to relevant evidence, not replace the evidence with a score.
The second boundary concerns disposition. Once a reviewer examines a candidate, they may conclude that the variation is permitted, that it needs clarification, that it is material, or that it warrants escalation or action. Those are different judgments. Recurrence can decide which item is read first, but it cannot decide which authority applies, whether a later rule changed the position, or what consequence follows. The queue is an interface precisely because the expert can accept, reject, widen, or escalate what it presents.
The third boundary concerns what the queue might hide. A ranking system creates at least two distinct failure surfaces. It can put a candidate near the top that turns out not to be material, consuming expert time. It can also fail to put a material difference near the top—or fail to return it at all—leaving the remaining review exposed. The BIS authors identify both questions directly: routine use would require measuring how often highly ranked candidates are judged material and how often material differences fail to enter the queue.
Those two measurements are not interchangeable. A queue can contain many genuine textual differences and still be poor at prioritisation if the top of the list is crowded with items that do not matter. It can also look efficient because it returns a short list while missing rare but consequential clauses. A useful evaluation must therefore examine the ordering of candidates and the omissions it produces, not simply count how many outputs sound plausible.
That evaluation needs document pairs independently assessed by supervisory and legal experts. The experts’ assessments provide a comparator for the model’s ranking: which candidates are material, which are permitted, which require escalation, and which apparent issues are false alarms. The comparator need not turn every legal question into a single mechanical label. In fact, unresolved disagreement may be part of the correct record. But the institution needs a disciplined way to determine whether the queue is directing attention toward issues that experts actually regard as important and whether it is missing issues those experts would have wanted to see.
The evaluation also has to preserve the difference between screening and adjudication. An expert may agree that two passages differ while disagreeing about the legal consequence. That is not a defective label; it may be the central fact about the matter. The assessment record can therefore distinguish a clear textual relationship from a disputed judgment about permission or materiality. Doing so prevents the benchmark from rewarding the model merely for sounding decisive. It also gives supervisors a better question to ask of the system: does it reliably bring the right evidence to the people who can resolve, escalate, or document the uncertainty?
A serious test should vary the kinds of cases that produce the queue rather than rely only on conspicuous wording differences. It should include candidates that experts consider useful, candidates that are repeated but ultimately permitted, and material issues that appear infrequently or are absent from the model’s list. It should examine whether the paired passages are actually the right passages and whether the model preserves the relevant condition, sequence, or consequence. The BIS source does not report such a complete benchmark; it identifies this kind of independent expert testing as work required before routine operationalisation.
The recurrence threshold then becomes a governance choice informed by evidence rather than a magic number. A threshold that admits only highly repeated candidates may make the queue easier to handle, but it could leave more of the document outside the first-pass list. A lower threshold may bring more potentially useful issues forward while increasing the burden on experts. The appropriate balance depends on the task, the consequences of missing a difference, the quality of the remaining full-document review, and the resources available for follow-up. The BIS source does not report a universally validated threshold; it says that further testing should establish the threshold, the amount of full-document review that remains necessary, and the tasks for which screening is suitable.
This last point changes how efficiency should be measured. It is not enough to say that the model reduced 576 observations to 79 recurrent candidates. That reduction is valuable only if the 79 help experts make better use of their time and the documents outside the queue remain subject to an adequate review strategy. The relevant question is not “How short is the list?” but “What quality of attention does the list enable, and what evidence supports the unattended remainder?”
The same reasoning limits what the reported exercise can establish. The BIS Bulletin presents a promising procedure, a scale application, and historical validation cases, but it does not report a complete independently labelled benchmark, a production false-negative rate, or a comparative study across models. The 79-candidate result is exact for the reported exercise; it is not a general performance guarantee. Nor does the evidence show that the recurrence threshold would transfer unchanged to another jurisdiction, document set, language, or model.
A queue becomes more trustworthy when its uncertainty is operationally visible. Reviewers should know how the candidates were generated, how equivalent findings were merged, how often each one recurred, and what the procedure is known to miss. They should also be able to inspect lower-ranked material or run a separate review when the stakes warrant it. That is not an argument against automation. It is the condition that prevents prioritisation from becoming silent exclusion.
The core lesson is therefore narrower than “repetition makes the model reliable.” Repetition makes one aspect of the model’s behaviour measurable: whether it returns equivalent candidates under repeated, independent runs. Consolidation makes those observations usable as a queue. Expert assessment determines whether the queue is decision-relevant, and continuing full-document review protects against the queue’s blind spots. The resulting system can focus supervision without pretending that a frequency count is a regulatory judgment.