How Agencies Learn—or Repeat—Their Mistakes
A public-sector AI acquisition does not become accountable merely because it appears in an inventory or passes a checklist. The harder question is whether the agency can see what happened, explain who decided, preserve the evidence, and carry a useful lesson into the next acquisition. This chapter examines two obstacles to that kind of learning: systems that escape the intended review point, and systems whose performance or failure is not documented well enough to guide anyone later. The cases matter not as stories of individual blame, but as evidence about how institutional memory is either built or lost.
When the review point misses the system
Procurement-based oversight works only if it reaches the capability that creates the public consequence. That sounds obvious, but administrative categories can obscure the object being governed. A low price does not necessarily mean low impact. An in-house system does not necessarily avoid the risks of a commercial one. A procurement that begins as ordinary software can acquire an AI component after the initial decision. And a risk-based framework can leave officials uncertain about whether the system crosses the threshold for additional scrutiny.
The checklist research identifies several versions of this problem. Some jurisdictions connect mandatory review to a value threshold, even though a small pilot can operate in a sensitive setting and affect a large or vulnerable population. The researchers describe a city parking study that used cameras and AI analysis in a historically over-policed neighborhood. Because it covered only one city square, the project fell below the value threshold for the full procurement process. The point is not that every small pilot needs the same review as a national system. It is that price and scale are imperfect proxies for public impact.
A similar gap appears when AI is hidden inside a larger procurement. The researchers describe a welfare case-management system intended to assist a human caseworker. The system also used an AI-driven triage function to determine the order in which claims were handled. That component could affect adjudication even though AI had not been contemplated when the procurement began. A linear procurement process can fail to revisit its assumptions when the proposed product changes. The agency may have followed the original procedure and still have no AI-specific requirements for a function that has become central to the workflow.
In-house development creates a different version of the same boundary. A technically capable government unit may build a risk-assessment tool more cheaply or with a better understanding of its own data than an outside vendor. That can reduce the mismatch between the agency and supplier. It can also remove procurement as a point at which requirements, rights, expertise, and public deliberation are made explicit. The concern is not that in-house work is inherently less trustworthy. It is that the location of development should not determine whether the system is inventoried, risk-screened, tested, and monitored.
Risk classification has a similar dependency. If only “high-risk” applications receive mandatory attention, someone has to decide what high risk means before the system is fully understood. The checklist researchers report that uncertainty about this classification was a frequent question for officials.