How Evidence Becomes a Score

This chapter follows an iFixAi result from individual evidence items to an inspection score, then to a five-pillar grade and a final verdict. The practical goal is to read the number as a controlled aggregation with gates and missing-evidence rules, not as a mysterious estimate of an agent's overall safety.

The arithmetic matters because the audit combines different evidence paths. A typed authorization decision, a judge-scored conversation, and an atomic claim assessment do not enter the score in exactly the same way. The scorecard tries to preserve their boundaries while still giving a deployment team a compact signal about where governance is holding and where it is not.

From evidence item to inspection result

An inspection begins with evidence items. Each item has an expected outcome and an actual result, but it can also carry the method that produced it: structural, judge-based, atomic-claim, or another explicitly defined path. For a structural authorization check, the item may record whether the tool was authorized, whether it executed, and whether a policy rule was returned. For a conversational check, it may preserve the prompt, the response, the rubric verdict, and any extraction status.

The inspection then applies its own scoring method. Some checks use a simple fraction of passing evidence items. Others use a rubric-weighted score in which dimensions inside a judge verdict have different importance. Some deliberately override the generic average because their published metric is binary blocking, denial, or escalation correctness. This is why two inspections with the same number of green items may contribute differently: their evidence items answer different questions and their specifications define different score semantics.

Before an inspection can be called pass or fail, it must clear its evidence floor. The floor is not a quality score; it is a minimum amount of scoreable evidence needed for the result to be meaningful. If a provider has no relevant surface, if the fixture does not supply enough cases, or if judge and provider errors remove too many items, the inspection may become inconclusive instead of being scored as a behavioral failure.

Extraction errors are part of this boundary. A judge that cannot return a conforming verdict has not necessarily observed a bad answer from the agent. Where the inspection permits it, the affected item is excluded from the scoreable denominator and the report carries a warning. A configuration error is different: it means the inspection could not execute correctly and remains visible as an error. The score is therefore built not only from pass and fail, but also from a decision about which observations are gradeable.

flowchart LR A["Evidence item"] --> B["Inspection score"] B --> C{"Evidence floor met?"} C -->|"No"| D["INCONCLUSIVE with warning"] C -->|"Yes"| E["Category score"] E --> F["Five-pillar weighted grade"] F --> G{"Mandatory gates pass?"} G -->|"Yes"| H["PASS or FAIL by threshold"] G -->|"No"| I["Score capped at 0.60"] G -->|"Not run"| J["No gradeable verdict"]

The floor sits before aggregation. That placement prevents a tiny or unavailable sample from quietly becoming a zero, a pass, or a misleading contribution to a broader category.