Making the Model Reviewable

The subject here is not whether a supervisory LLM can produce a persuasive explanation. It is whether an expert can inspect and test the evidence behind a ranked candidate, understand the system’s limitations, and intervene when the result is not good enough. This chapter distinguishes explainability from transparency, then applies performance and human-oversight principles to the BIS screening queue. The practical goal is a review posture that uses the model’s strengths without turning fluency, recurrence, or a convenient interface into false assurance.

Transparency is not the same as explainability

The Financial Stability Board treats explainability as the extent to which people can understand how an AI model or system produces outputs from inputs: the factors, logic, or processes involved, the relative importance of inputs, and how those relationships vary across contexts. It treats transparency as the availability of appropriate information about the model or system to relevant stakeholders. The two concepts reinforce one another, but they are not interchangeable. A provider can disclose a model’s capabilities, limits, or development process while the institution still cannot fully understand how a particular output was generated.

That distinction is especially important for an LLM comparing a regulatory package with a prospectus. Transparency can tell an expert which model and prompt were used, what documents were supplied, what the system is intended to do, and what it is not approved to do. It can preserve the paired passages, the divergence category, and the recurrence signal. Those records are essential for accountability. They do not prove that the model’s interpretation of the passages is conceptually sound or that its explanation faithfully describes the process that produced the candidate.

The FSB describes explainability as a continuum rather than a binary property. A simpler model may expose more of its decision logic, while a more complex model may perform better on some tasks but be harder to understand and more costly to operate. The relevant choice is not “explainable or unexplained” in the abstract. It is whether the degree of understanding is sufficient for the use case, given its materiality, risk, autonomy, and potential effect on people or institutions.

For the BIS queue, the most valuable form of local review may not be a claim about the model’s hidden reasoning. It is a traceable account of the specific comparison: the applicable rule passage, the corresponding prospectus passage, the possible direction of the difference, and the reason the candidate was placed in the queue. An expert can then inspect the relationship directly. A global view remains useful too: reviewers need to know how the system behaves across a body of comparisons, what kinds of provisions it tends to surface, and whether its rankings remain stable under changed conditions.

The distinction becomes sharper with generative AI. The FSB’s consultation example describes an insurer that uses retrieval-augmented generation to ground an LLM in an existing information corpus and also uses a reasoning feature. The insurer remains mindful that a model’s stated reasoning may not accurately represent the steps it actually uses. The example therefore combines controlled deployment, qualified human oversight, stability testing, prompt guidance, and restrictions on sharing outputs directly with customers. The lesson for supervisory screening is not to reject every generated explanation.