Why Plausible False Diagnoses Beat Overt Prompt Injection

This chapter examines why a telemetry-fed AIOps agent can reject an obvious instruction yet accept a false diagnosis that looks like useful incident evidence. The focus is the difference between direct prompt injection, traditional log injection, and the paper’s adversarial reward-hacking technique. That comparison supports a practical judgment: defenses must distinguish not only malformed data from well-formed data, but also evidence that describes an incident from content that quietly steers the remedy.

The distinction matters because the failure chain does not depend on a malicious sentence appearing in a model’s direct prompt. The attacker’s influence can arrive inside a record the agent has been trained or configured to treat as operational context. If that record supplies a plausible cause and a related fix, the agent may follow its normal objective—resolve the incident—while moving toward the attacker’s desired outcome.

Three ways input can manipulate an agent

Direct prompt injection tries to replace an agent’s intended task with an adversarially chosen one. Its language is often recognizable: ignore the current instructions, follow a different rule, reveal something, or take an unrelated action. The attacker is addressing the model’s instruction-following behavior directly, even if the input arrives through an external resource rather than a user’s visible prompt.

Traditional log injection attacks the integrity of the record or the way downstream systems parse it. Unsanitized input can introduce line breaks, delimiters, or other control characters that forge entries, truncate records, or alter their structure. The objective may be to conceal activity, corrupt an audit trail, disrupt incident response, or exploit a logging component. This is a serious problem even when no language model is present because it changes what investigators and detection systems believe the log contains.

Adversarial reward-hacking, as named by the paper, uses a different route. It does not need to override the agent’s task or break the log format. It inserts a plausible explanation for the incident and pairs it with a remediation that appears to follow from that explanation. The content is designed to make the agent’s normal goal—find the cause and fix the problem—work in the attacker’s favor.

These approaches differ along three useful dimensions. First is the control point: the model’s instruction context, the telemetry parser or record structure, or the semantic content of an otherwise ordinary telemetry field. Second is appearance: an explicit command, malformed or forged data, or a technically plausible incident narrative. Third is the mechanism of influence: task replacement, record corruption, or objective-aligned misdiagnosis. A single system may face all three, but a control that addresses one does not automatically address the others.

The lead and the body of a plausible false diagnosis

The paper describes the adversarial reward-hacking payload as having two related parts. The lead offers a plausible explanation for the observed incident.