The Credential Path: How a Secret Gets Past the Safety Net

A secret-shaped value in an agent-assisted coding pull request is not yet a confirmed leak. The useful question is what happens between the first automated flag and the point where a credential could authorize a later action. This chapter follows that path through independent validation, authorship, review visibility, and removal or integration. The result is a more precise judgment than “the model wrote a secret”: the evidence points to a shared workflow in which detection, human handling, and integration controls can each fail differently.

A credential deserves special treatment because it is not merely configuration. A live key, token, private key, or password-bearing connection string can act as identity material. If it enters a public repository or a build path, its significance depends on whether it is real, what service accepts it, which permissions it carries, and how quickly it can be revoked. The empirical study does not establish all of those downstream conditions. It does, however, show how a broad automated signal narrows to a smaller set of cases that deserve a different kind of intervention.

From a secret-shaped flag to a genuine credential

The starting point was the secrets_identity category, which contained 294 flagged secret smells. That number came from the automated judge’s classification. The researchers did not treat the label as proof. Instead, they removed the judge’s conclusion from the next stage and manually examined each candidate using the repository, the pull-request discussion, and the commit history that followed the introducing change. This separation matters: the people validating a suspected secret had to assess the evidence for themselves rather than merely repeat the model’s decision.

The manual process could label 272 of the 294 candidates. Twenty-two cases could not be labeled because the relevant pull requests no longer existed. That is a state of missing evidence, not a clean result. A deleted or unavailable request cannot be inspected for the value, its context, or what happened after it was proposed. Treating those cases as harmless would quietly turn loss of observability into evidence of safety.

Of the 272 candidates that could be labeled, the coders confirmed 74 as genuine credentials, a rate of 27.2%. The remaining labeled cases were not confirmed as genuine credentials; they could include placeholders, examples, or values that the manual evidence did not support as live identity material. The transition from 294 flags to 74 genuine cases is the essential evidentiary lesson. Automated screening is useful because it creates a manageable set of candidates. It is not a substitute for checking whether the value has the properties that make it an actual credential.

The lifecycle is easier to reason about when its states remain separate:

The transitions have different owners and different meanings.