How to Audit a Reference System Before You Trust It

The validation problem in anthropics/financial-services is not solved by finding one command that returns a clean result. A repository check, a valid worker payload, and a balanced spreadsheet answer different questions. This chapter treats a reference deployment as a diagnosis problem: first identify which layer is failing, then apply the check that can actually distinguish that cause. The goal is a layered preflight that separates structural consistency, artifact integrity, security, connector readiness, and governance.

A structural check proves only structural consistency

The repository's structural checker is useful because it catches failures that otherwise appear late. It parses Managed Agent YAML, plugin and marketplace JSON, steering examples, and agent frontmatter. It resolves references from system files, skill paths, plugin bundles, callable-agent manifests, and marketplace source directories. It also checks that required cookbook files exist, that bundled skills match their vertical sources, and that PowerShell files meet the repository's encoding constraint.

Those checks answer questions such as: does the declared file exist, does the manifest parse, does the marketplace source point to a plugin, and has a bundled skill drifted from its source? They do not answer whether a provider endpoint is reachable, whether a subscription is active, whether a model produces a correct calculation, or whether a human approval policy is enforceable. A clean structural result means that the repository's declared relationships are coherent under the checker's scope. It does not mean that the deployed workflow is ready to make a consequential artifact.

The synchronization script addresses one particular structural failure: drift between a vertical skill and the copies bundled into named agents. It treats the vertical copy as authoritative and propagates it to agent bundles. That is valuable for keeping the package consistent, but it is not a replacement for deciding whether a changed method should be released to every consumer or for rechecking the outputs that depend on it. Synchronization repairs a file relationship; it does not validate the business effect of the changed method.

The same distinction applies to structured worker outputs. The validation helper checks a JSON object against a schema before the orchestrator consumes it. That catches a missing field, wrong type, disallowed value, oversized collection, or invalid character pattern. It does not establish that the extraction is factually right or that the handoff is authorized. Structural checks are gates, not conclusions.

Domain audits catch failures that linters cannot

Spreadsheet integrity requires a different kind of test. The audit skill checks formula errors, hardcodes embedded inside formulas, inconsistent neighboring formulas, off-by-one ranges, pasted-over formulas, broken cross-sheet links, unit mismatches, hidden rows or tabs, and circular references. At model scope it also checks balance-sheet ties, cash-flow tie-outs, working-capital signs, share counts, debt schedules, and model-specific logic such as terminal-value dominance or exit-multiple placement.

Those checks can catch a workbook that is structurally valid as a file but wrong as a financial model. A manifest checker cannot tell whether the cash-flow statement ties to ending cash or whether a formula references the wrong assumption row. A JSON validator cannot tell whether a DCF assumption is economically defensible.