The Difference Between a Defect and a Failure
OS-Sanitizer depends on a distinction that is easy to blur: a defect can exist before a failure. Its runtime reports are meant to identify that latent condition, so the useful question is not simply whether an application has broken. It is what kind of evidence shows that the application could break, under which circumstances, and what would still need to be demonstrated. Keeping those categories separate lets us understand both the promise and the restraint of dynamic defect inference.
The causal chain from mistake to failure
The paper uses five categories to describe software problems: errors, defects, faults, failures, and code smells. They are related, but they do not describe the same moment or the same strength of evidence.
An error is a mistake made by a developer or a user. It might be a flawed assumption about who owns a file, an incorrect expectation about an API, or a decision to omit a check that seemed unnecessary. The error is the origin of the problem, but it is not yet a property of the running program.
A defect is the issue in the program that can lead to unwanted behavior. Once the mistaken assumption is embodied in source code or another executable artifact, it can remain present even when every ordinary test passes. A defect is therefore a latent capability: the program contains a path to bad behavior, but the required conditions may not have occurred.
A fault is the unwanted behavior when that defect is activated. The application may read or modify something it should not, enter an invalid state, or violate a security assumption. The fault is an event in an execution, not merely a possibility in the design.
A failure occurs when the fault becomes visible as a broken guarantee or invalid state. The program may terminate, return an incorrect result, corrupt data, or otherwise fail to do what it promises. In conventional testing, this is often the point at which evidence becomes most concrete: the test has produced a counterexample that can be inspected and repeated.
The chain is not perfectly automatic. A defect does not cause a fault in every environment, and a fault does not always cause an immediate visible failure. Software can be resilient, a defensive layer can absorb the effect, or the relevant condition may disappear before the guarantee is violated. That does not make the defect imaginary. It means that the path from possibility to consequence has conditions and intermediate states.
OS-Sanitizer is interested in the space before the last two steps. It observes behavior from a running Linux system and uses heuristics to infer that a defect may be present, even when the execution has not supplied a failure. The inference is valuable precisely because it is earlier than a conventional failure report. It is also weaker than a reproduced failure, because the observed behavior may be a warning sign rather than a demonstrated violation.
The diagram shows two important qualifications. First, a code smell is a parallel warning path, not a synonym for a defect. Second, context helps determine whether a latent weakness becomes a fault or whether a suspicious pattern remains intended behavior. The Linux verifier can enforce safety properties for an eBPF program that performs the observation, but it cannot decide which of these application-level transitions has occurred.
Why a code smell can be worth finding
A code smell is often described as a design issue or a benign violation of a best practice. That wording can sound dismissive: if the program has not failed, why spend time on it? The answer is that a smell can identify a region where assumptions are fragile, complexity is high, or future use may differ from the current use.
Consider a program that allocates memory with permissions that allow it to be read, written, and executed. That combination is a reasonable security warning because it can make dynamic code generation harder to protect. It is not automatically a defect. Just-in-time compilers may require such memory while generating and running code. A heuristic that reports the behavior may therefore be correct about the observation and wrong if it declares the application vulnerable without considering why the memory exists.
The warning can still be productive. It may point investigators toward a specialized component that has received less testing than the rest of the application. In the study, reports about this kind of behavior helped draw attention to a complex just-in-time implementation, where further testing uncovered issues. The initial signal did not prove those later issues; it helped choose where to look.
The reverse case is possible too. A command assembled from mutable or externally influenced text may look unsafe to a generic heuristic, yet a utility may intentionally treat that text as the command it is designed to execute. The report is then “works as intended” in the observed use. That outcome does not mean the heuristic was useless. It tells maintainers that the behavior needs context, documentation, or a more precise filter so that future investigators do not mistake a design choice for an accidental command-injection path.
This is why the paper resists reducing the method to a false-positive percentage. A report can be false as a narrow claim—“this behavior is definitely a defect”—while true as a prioritization signal—“this behavior deserves an explanation.” The value depends on the question being asked. A security engineer looking for exploitable conditions may need a different threshold from a maintainer looking for brittle interfaces or under-tested code.
A useful interpretation has three levels:
- The event is real. The monitor observed a behavior and, where relevant, its surrounding runtime facts.
- The pattern is suspicious. A heuristic connects those facts to a known weakness, design warning, or misuse pattern.
- The conclusion is validated. Source review, reproduction, fuzzing, additional tests, or maintainer analysis establishes what the behavior means in context.
Only the first level follows directly from the observation. The second is the contribution of OS-Sanitizer. The third remains an investigation task.
What different testing approaches can establish
The distinction becomes clearer when the methods are placed side by side.
Fault-oriented dynamic testing
A conventional dynamic test supplies inputs or actions and checks the resulting behavior. Its great advantage is specificity. If the test produces an invalid result, the team has a concrete failure under known conditions. Reproduction supports debugging, regression tests, and confidence that a fix addresses a real path.
Its limitation is not that it is weak, but that it is selective. The test can only expose conditions it creates or encounters. If no test combines a privileged process with a partly user-controlled path, a context-dependent filesystem defect may remain dormant. Absence of a failure in that test suite is evidence that the tested scenarios did not fail; it is not evidence that no latent defect exists elsewhere.
Static analysis
Static analysis examines the program or its source without waiting for a particular execution. It can search a broad codebase for patterns associated with unsafe calls, missing checks, or suspicious data flow. This makes it valuable for properties that are visible in source and for code that has not yet run.
But source context does not always include deployment context. A library may be called by programs with different privilege levels and ownership models. A source pattern that is harmless for one caller may become dangerous for another. Static analysis can identify the possibility, while runtime observation can supply evidence about what actually happened in a live system. The two perspectives answer different questions.
Runtime predictive analysis
Runtime predictive analysis observes executions and predicts potential defects that have not manifested as faults. This is the closest comparison to OS-Sanitizer. The important difference in the paper’s positioning is scope: OS-Sanitizer uses eBPF to take a system-level view and applies heuristics to several classes of Linux application behavior, rather than limiting inference to one language feature or one narrow bug type.
That system-level view has a cost. Runtime evidence is tied to the environments and events that occur. It can miss source-only problems, higher-level semantics that have no suitable observation point, or defects whose triggering context never appears. It can also produce repeated or ambiguous signals because it sees behavior as it happens rather than reasoning once over a complete abstract program.
These approaches should therefore be treated as complements. A static tool may identify a risky construction before deployment. A dynamic monitor may show that a dependency is used across a privilege boundary in practice. A fault-oriented test may then reproduce the consequence. None of those roles makes the others unnecessary.
What an OS-Sanitizer report actually says
The safest reading of an OS-Sanitizer report is modest but useful: this runtime behavior, in this observed context, matches a heuristic associated with a possible defect or code smell.
That sentence contains several limits. “This behavior” means the report is grounded in an event, not in every possible execution. “This context” means ownership, privilege, timing, or surrounding state may be essential to its meaning. “Matches a heuristic” means the connection is an inference designed by the tool’s authors, not a formal proof of the application’s intent. “Possible defect or code smell” leaves room for intended behavior, a design trade-off, or an issue that only becomes dangerous under a different caller.
A report can still be highly actionable. It can narrow a large search space to a source region, reveal that a dependency is being used in a way its authors did not anticipate, or show that a security guarantee depends on an unstated environmental assumption. It can also tell an investigator what evidence to seek next: the relevant source, the caller’s privilege, the ownership of the resource, a reproduction under controlled conditions, or a test that distinguishes intended from unintended behavior.
The report should not be read as saying that the application has already failed. Nor should a quiet monitor be read as evidence that the application has no latent defects. The method reports what its selected hooks and heuristics can see. A missing report may mean that the behavior did not occur, that the relevant event was outside the configured coverage, or simply that the prototype had no heuristic for it. Those possibilities are not equivalent.
This distinction is especially important when severity filtering is used. Filtering low-confidence or low-impact reports can make a system usable, but it can also discard early warnings. Raising the threshold is an operational choice, not an improvement in the underlying truth of the heuristic. The right setting depends on whether the user is seeking urgent security findings, maintenance leads, or broad research coverage.
The practical rule
Dynamic defect inference adds a valuable middle layer to software testing. It can notice a meaningful relationship before a failure provides a counterexample, and it can observe live context that source-only reasoning may not know. OS-Sanitizer gives that idea a concrete Linux implementation and shows why benign behavior deserves closer attention.
The method’s restraint is part of its value. A defect is not a failure; a code smell is not automatically a defect; and a runtime report is not a verdict. The stronger the consequence, the more important it is to move from observation to inference and then to validation. New evidence—source intent, a reproducible fault, a privilege boundary, or a maintainer’s explanation—can strengthen, weaken, or change the classification.
With these distinctions in place, the next question is mechanical: how can a Linux tool observe enough of an application and the kernel to make a contextual inference without modifying the application itself? That is where eBPF’s attachment points, shared state, and safety checks become central.