The Supervisory Job Comes Before the Model
This chapter examines how a financial supervisory authority turns a proposed AI project into a specific supervisory question. The practical test is not whether the authority can find a promising model, but whether it can define the risk or decision to improve, show how success will be measured, and identify reasons to narrow, redesign, defer, or stop the work. That discipline matters because a vague technology proposal can look innovative while leaving the authority unable to tell whether it has improved supervision at all.
Start with the supervisory decision, not the technology
A financial supervisory authority does not need an AI system in the abstract. It needs to carry out a mandate: authorize institutions, assess prudential risks, monitor conduct, detect possible money-laundering activity, or understand how a market is changing. Risk-based supervision directs attention toward the risks most relevant to that mandate. An AI project is therefore well framed only when its intended contribution to one of those activities is clear.
“Use AI to improve supervision” fails that test. It names a means without identifying the problem, the affected decision, or the standard by which the means will be judged. A better starting question is narrower: Which supervisory task is too slow, too narrow, or too difficult to perform with the current process, and what change would count as an improvement without weakening the quality of judgment?
The IMF paper offers a concrete example involving fit-and-proper assessments. An authority might want to improve the efficiency and accuracy of reviewing questionnaires used in authorization. The proposed system could automate machine reading and textual analysis, flag issues for staff, and leave complex cases for closer human review. That description is more useful than “apply natural-language processing,” because it identifies the work being changed and the role the output would play. It also creates measurable expectations: compare the time taken, the accuracy of assessments, the number of issues correctly identified, and oversight errors before and after the change.
The example is a project illustration, not evidence that every authority should automate this task or that the proposed approach will work. Its value is diagnostic. It reveals the questions a serious proposal must answer:
- What decision or supervisory activity is changing?
- What does the present process do, and where is its bottleneck or blind spot?
- Will the system recommend, prioritize, summarize, or make a decision?
- Which cases remain subject to human review?
- What will count as a better result, and what must not get worse?
- What is explicitly outside the project’s scope?
The last question is easy to overlook. A project that reviews fit-and-proper questionnaires may flag relevant passages; it may not determine whether a person is suitable, resolve every governance concern, or replace the authority’s legal and supervisory responsibility. Stating those boundaries is not a sign of weak ambition. It prevents a successful narrow tool from being credited with solving a broader problem it was never designed to address.
The IMF paper warns against vague formulations such as doing something with all available data or simply implementing AI. It recommends a modest scope so an authority can build a learning curve and gain trust before attempting a larger implementation. A small scope is not automatically low risk—an authorization decision can be material even if the dataset is small—but it makes the intended use, failure surface, and evidence needed for a decision easier to see.
This framing also separates competing explanations for a weak supervisory process. The visible symptom may be a long review queue, but the cause could be repeated manual transcription, poor document structure, unclear escalation rules, missing data, inadequate staff capacity, or a legal requirement for a form of review that software cannot replace. AI may address one contributing condition while leaving the root problem untouched. If the authority has not identified which cause it is testing, a faster workflow could merely accelerate an unreliable process.
Four feasibility questions before investment
Once the supervisory problem is specific, the authority can apply the D.A.T.A. framework described in the IMF paper. The framework asks four sequential questions: Does the authority have access to data that is valuable and rare? Can employees use that data to create solutions? Can the technology provide the proposed solution? Is the solution consistent with law and ethics? The acronym refers to Data, Autonomy, Technology, and Accountability.
Here, “Autonomy” does not mean that the model should operate without people. It concerns whether the organization can turn its data into a workable solution: whether staff have the skills, authority, and practical ability to use what is available. “Accountability” asks whether the proposed use can be governed within legal and ethical boundaries. The framework is useful because it forces a proposal to confront organizational and institutional conditions instead of treating them as implementation details to solve later.
The first question is more demanding than asking whether data exists. Data must be relevant to the supervisory problem, accessible under the authority’s mandate, and sufficiently valuable for the intended use. A large archive of questionnaires does not by itself show that the records are consistent enough to train or test a system. A collection of complaints may reveal consumer harm, but it may not represent people who use different channels or do not complain. A set of regulatory reports may be abundant while still containing missing fields, inconsistent definitions, or duplicated submissions. The question is whether the data can support the particular judgment, not whether the authority possesses a great deal of it.
The second question tests the organization. Can supervisors explain the problem and review the output? Can data specialists prepare and assess the information? Can engineers connect a system to existing tools? Can legal or ethics functions identify obligations and unacceptable uses? If the answer is no, purchasing a model does not remove the gap. It may make the gap less visible by placing a polished interface over an unresolved decision.
The third question resists technology-first reasoning. A model may classify documents or detect patterns, but the authority must still determine whether that technical capability matches the task. A solution designed for text retrieval is not automatically suitable for forecasting liquidity stress. A system that flags possible issues may support triage but cannot be assumed to establish a violation. The use case, data, and technology have to fit together.
The fourth question concerns more than formal compliance. The authority must consider privacy, discrimination, explainability, cybersecurity, and who is accountable for the outcome. A use that affects a high-materiality supervisory decision may require a different governance structure and level of scrutiny from a low-impact administrative aid. An authority also needs to know whether it can explain the system’s role to the people who rely on it and challenge the result when it appears wrong.
The paper describes a simple scoring version of D.A.T.A.: one point for each yes. Four points indicate a high likelihood of success; three suggest critical adjustments; two indicate that success is unlikely; one makes success very unlikely; and zero means the project should be terminated immediately. These thresholds are a screening aid, not a validated probability model for supervisory AI. They should prompt questions and decisions, not create false precision.
The same caution applies to the paper’s citation of a representative study claiming that as many as 85 percent of big-data initiatives fail because executives cannot accurately assess project risks in advance. That figure concerns big-data initiatives in the cited source, not a measured failure rate for financial supervisory authorities or AI projects in this paper. It supports the importance of early risk assessment; it does not prove that a particular authority has an 85 percent chance of failure.
The diagram is a decision path, not a claim that projects move through four independent technical gates in a fixed order. A failed question can require new evidence or a revised problem statement. Its purpose is to keep the project’s feasibility conditions visible before an authority commits to model development.
Turn the problem into a project that can be judged
The IMF paper calls the first lifecycle phase Project Foundation. Its result should be a plan that defines the business problem, the criteria for achieving the objectives, technical and functional requirements, the implementation challenges, and the resources required. This phase converts a general ambition into a claim that can be tested: if the authority changes this part of the process in this way, a specified supervisory result should improve without unacceptable new risk.
That claim needs two kinds of measures. A business or supervisory key performance indicator (KPI) asks whether the authority’s work improved. In the fit-and-proper example, the relevant questions include whether review time falls, whether staff can process more files, and whether the quality of issue detection is maintained or improved. A technical indicator asks how the model behaves, using measures such as accuracy, precision, recall, or error measures when they fit the task. Technical performance is evidence about the tool; it is not the same as evidence that the authority achieved its purpose.
The two kinds of measures must still connect. A model with high accuracy on a test set may be of little value if the test cases are unlike current questionnaires, if the most consequential errors are hidden by an average, or if staff cannot use the result in the authorization workflow. A modest reduction in review time may not be an improvement if important issues are missed. The authority needs a baseline or comparison that makes the change visible and a rule for deciding which losses are unacceptable.
A well-formed project also identifies the questions it will not answer. This protects the evaluation from expanding after the system appears promising. If the initial use is document triage, the authority should not quietly treat it as a complete assessment of governance quality. If the use is market surveillance, a flagged pattern may justify investigation rather than establish misconduct. Follow-up work can be valuable, but it should be recognized as a separate need with its own evidence and resources.
Feasibility includes conditions outside the model. The IMF paper calls for assessment of human resources, information-technology infrastructure, cybersecurity, data and model governance, funding, and organizational culture. Public budgets and procurement processes often require clear plans and transparent commitments, while AI work may require iteration and changes as evidence accumulates. That tension does not make agile work impossible, but it means resource estimates and decision points must be credible enough for the authority to govern change.
A small pilot is most useful when it is designed to teach the authority something specific. It might test whether a defined document set can be classified with acceptable error, whether staff can understand and challenge the flags, or whether a data-quality problem is larger than expected. A pilot that has no success criterion, no owner, and no stop condition can produce activity without producing knowledge. Conversely, a pilot that meets its technical target but exposes weak coverage or unacceptable accountability has still generated a valuable result: the authority has learned not to scale that design unchanged.
The decision at the end of Project Foundation is therefore conditional. Proceed when the problem, data, capability, technology, accountability, measures, and resources line up sufficiently for a bounded test. Adjust when one of those elements is promising but incomplete. Defer when a prerequisite such as reporting quality or infrastructure must be repaired first. Stop when the use cannot be justified within the authority’s legal, ethical, or supervisory purpose. None of these decisions says that AI is universally good or bad. They say whether this proposal is credible for this job.
That last distinction leads directly to the next constraint. Even a carefully framed project can fail if its data does not represent the problem, cannot be traced, or cannot be used lawfully. Before the authority chooses a model, it must establish what its supervisory data can actually support.