When Observability Becomes Control
LLM-driven AIOps is the subject of this series: systems that use logs, metrics, traces, and alerts to detect anomalies, investigate incidents, and recommend or carry out remediation. This opening chapter focuses on a simple but consequential question: what changes when the telemetry that describes an operating system also becomes an input to an agent that can change it? The answer gives readers a practical way to judge the security of autonomous AIOps—not by treating “AI” as a mysterious new risk, but by tracing how evidence becomes action and where trust can fail along the way.
AIOps promises to reduce the time between noticing a problem and responding to it. That promise is attractive because modern systems produce more operational signals than a human team can examine one by one. But speed changes the meaning of an error in the evidence chain. A misleading record that a human would question may become a premise in an agent’s diagnosis. If the agent can also invoke tools or submit a remediation without a separate approval step, the same mistake can move quickly from interpretation to infrastructure change.
The central idea is therefore not that telemetry is unimportant or inherently dangerous. It is that telemetry has acquired a new role. It remains a record of events, but in an agentic workflow it is also part of a control loop. Its provenance, structure, and limits become security properties of the AIOps system itself.
From monitoring records to an operational loop
Traditional observability gives an organization a view of system behavior. Tools collect telemetry, store it, make it available for queries, and generate alerts when rules or anomaly detectors identify something unusual. Logs, metrics, traces, and alerts are different forms of evidence, but they serve a shared purpose: helping people understand an internal state that cannot be observed directly from a single component.
AIOps adds an agent to this picture. The agent is not merely displaying a dashboard or sorting a queue. It uses telemetry to perform tasks that would otherwise require an operator, including anomaly detection, root-cause analysis, and remediation. Root-cause analysis matters because the visible symptom is often not the underlying fault. An alert may say that a service is failing, while the useful question is which dependency, configuration, or resource condition caused the failure. The agent tries to connect those clues and propose a response.
The basic operating pattern has three stages. First, something activates the agent: an alert, a ticket, a chat request, or a scheduled scan. Second, the agent analyzes the incident by querying the observability stack and gathering relevant logs, metrics, and traces. Third, it submits a solution. In a human-in-the-loop design, that solution is a recommendation for an on-call engineer. In a fully autonomous design, it may be passed to tools that carry out the remediation end to end.
This sequence is easy to describe and easy to underestimate. The agent’s action is downstream of telemetry, but the agent’s view of the incident is largely constructed from telemetry. That makes the boundary between “what happened” and “what the system believes happened” operationally important. The model may be capable of reasoning about a record, yet still be reasoning from a record whose origin or content has not been established.
Why the record is no longer passive
For an operator, a log entry is usually evidence to interpret alongside other evidence. It may reveal when a failure occurred, which service was involved, what request or account was affected, and what result followed. The value of the record depends on context, consistency, and the ability to compare it with other sources. A human investigator can ask whether a line makes sense, whether it conflicts with a metric, or whether it contains a value that should never have appeared there.
An AIOps agent also needs context, but it consumes that context inside a decision process. It may query several telemetry sources, use what it finds to form a root-cause explanation, and then select a remediation that appears to address the explanation. The record is still descriptive, but it now participates in selecting the next system state. A field that once helped an engineer reconstruct an event can influence what the agent believes the event means.
This is the key shift from observability to control. The telemetry pipeline does not need to give an attacker a direct conversation with the language model in order to matter. If externally influenced data enters an application event, is stored or forwarded as telemetry, and is later retrieved by the agent, the upstream path has become relevant to the agent’s decision. The model is only one component in that path. The application’s logging behavior, the collector, the query interface, the activation mechanism, and the agent’s tools all contribute to the result.
That does not mean every telemetry value is untrustworthy. It means trust must be specific. A timestamp generated by a controlled service, a status value constrained by a schema, and a user-supplied description do not necessarily deserve the same authority. Nor does “logged” mean “verified.” General logging guidance warns that data arriving from another trust zone may be missing, modified, forged, replayed, or malicious. In an AIOps setting, that warning has an additional consequence: an untrusted value may not only corrupt an audit trail; it may steer a diagnosis.
The practical distinction is between evidence and instruction. A telemetry field can be useful evidence without being allowed to prescribe a remedy. A user-controlled value might explain why an error occurred, but it should not silently acquire the authority to tell an agent which software version to install or which security guarantee to weaken. Preserving that distinction is one of the defining design problems for autonomous AIOps.
What autonomy changes
The consequence of a misleading record depends on what the agent is allowed to do. In an advisory system, a false diagnosis may waste an operator’s time or send an investigation in the wrong direction. The human may compare the recommendation with other evidence, reject it, or request a safer test. That does not eliminate the risk: a recommendation that looks technically plausible can still bias a hurried response. But the final change is separated from the model’s interpretation.
In a fully autonomous system, the separation may disappear. The agent can query telemetry, reason about the incident, and use available tools to implement the proposed solution. The paper’s evaluated agents have access to observability tools and shell commands, illustrating why the action boundary matters as much as the input boundary. A model can be wrong in either mode; autonomy determines how far that error travels before another control has a chance to stop it.
This is why a useful security question is not simply, “Can the model be fooled?” It is, “If the model is working from misleading telemetry, what changes can it make, how reversible are they, and who or what must approve them?” A recommendation, a low-impact restart, a configuration change, and a security-sensitive downgrade are not equivalent outcomes. They require different confidence, permission, and recovery expectations.
The same reasoning applies to the telemetry pipeline. An AIOps deployment is not automatically exposed merely because it has a public web interface or uses a language model. The studied attack model requires a combination of conditions: a public interface that an external party can use, at least one telemetry instance that carries information from that interface, and an AIOps process that can later read the resulting record. The agent must also have some route from its decision to a meaningful action for the compromise to become operationally serious.
Those conditions are useful because they turn a vague concern into a systems question. Where can external information enter? Which event types preserve it? How is the event collected and protected? What activates the agent? Which telemetry does it query? What tools are exposed after diagnosis? Which actions are advisory, gated, reversible, or unrestricted? The answers describe the real attack surface more accurately than the label “AI-powered operations.”
The security question this series will follow
The central tension is now visible. AIOps needs rich, timely telemetry to diagnose complex incidents. Logging guidance likewise emphasizes application-level context because the application knows details that infrastructure components may not. But the more context a pipeline preserves, the more carefully it must distinguish trusted structure from values that originated outside the system’s control. Removing all detail would damage diagnosis; treating all detail as authoritative would damage security.
The right response is not to abandon automation. It is to design the telemetry path and the action path together. A trustworthy system should know which parts of an event are controlled, which are user-influenced, how records can be altered or withheld, and what an agent is permitted to infer or change. It should preserve raw evidence for investigation while limiting the decision-maker’s exposure to unverified content. It should also make the consequences of uncertainty visible through approval gates, least-privileged tools, monitoring, and recovery mechanisms.
The rest of this series examines that problem in progressively greater detail. It first makes the normal AIOps control flow concrete, then follows untrusted input through application logging and telemetry, and finally asks how a plausible false diagnosis can influence remediation. From there, the focus turns to protected logging and a targeted sanitization defense, with the evidence and its limits kept in view.
For now, the essential judgment is straightforward: in autonomous AIOps, observability is not outside the security boundary. Once telemetry helps determine what the system will do next, the integrity and provenance of that telemetry become part of the system’s control surface. AIOps can only be as safe as the path from event, to evidence, to decision, to change.