The Prompting Gap: How Prevention Slips into Review
For developers working with AI coding assistants, the crucial failure point may occur before any code is returned. This chapter examines what happens when a functional request reaches the assistant without the security conditions that make the requested behavior safe. The goal is to identify how security knowledge becomes a later review obligation, why the USENIX study treats that pattern as a structural prompting gap rather than simple carelessness, and why later review cannot fully substitute for requirements and risk reasoning applied at the start.
A request defines the assistant’s idea of correctness
An AI coding assistant does not receive a developer’s entire mental model. It receives a prompt, the surrounding context made available to it, and the interaction that follows. That boundary matters. A developer may understand that a profile update must be authorized, that input from a client is untrusted, or that a configuration should not expose secrets. If none of those conditions enters the request or its usable context, the assistant has no explicit reason to treat them as part of the definition of a successful implementation.
Consider the difference between two requests for the same endpoint. The first asks for an endpoint that accepts profile fields, updates a record, and returns a result. The second adds the security conditions: the requester’s identity must come from a trusted session, the requested account must be authorized, and input handling must be appropriate for untrusted data. Both requests describe the same visible feature. The second, however, defines a wider correctness boundary. It tells the assistant that “works” includes a relationship between the caller and the record being changed, not only a successful database update.
This is what makes a functional prompt different from a secure specification. Function describes an observable operation. Security often describes a condition on who may trigger that operation, what assumptions can be trusted, what must remain confidential, and how the system behaves when an adversary supplies unexpected input. Those conditions can be less visible than the happy path. They are also easier to omit when an assistant is treated primarily as a fast way to turn a short description into runnable code.
The omission is not necessarily a sign that the developer has forgotten security. It can be produced by the interaction’s default framing. The prompt-then-suggest model presents the developer as someone specifying desired functionality and the assistant as someone filling in implementation details. Security, accessibility, performance, and other non-functional properties fall outside that frame unless the developer actively imports them. The assistant may return a coherent answer because it has satisfied the request it was given, while the developer later realizes that the request was incomplete.
That sequence creates two different definitions of completion. The first is local: the code runs, the endpoint responds, the project starts, or the reported bug no longer appears. The second is security-related: the code preserves authorization boundaries, handles hostile input, protects sensitive information, and behaves safely under the conditions that matter. When the first definition is satisfied before the second has been stated, the developer must reopen a decision that the initial interaction never made explicit.
The USENIX study’s “preventive-to-reactive” language names this change in timing. Preventive security enters while the problem is being framed and the design is being formed. The developer identifies a security property before generation, so that the property can constrain what the assistant produces. Reactive security enters after a functional result exists. The developer inspects it, asks what might be wrong, runs a security check, or responds to a discovered issue.
The two modes can coexist. A developer can specify security requirements and still review the result. In fact, review remains necessary because an explicit request does not guarantee that the assistant interpreted every condition correctly. The concern is what happens when the preventive step is absent and review is expected to reconstruct it. A later reviewer must notice that an important property was never part of the original definition of done, determine which property matters, and inspect whether the implementation satisfies it. That is more work than checking a requirement that was already made visible.
The diagram is not a claim that the two paths always produce different code. It shows where the burden moves. When security is specified early, the assistant’s output can be judged against both functional and security conditions. When security is deferred, the output may still be useful, but the review stage has to discover what condition was missing before it can test it. If no one notices the missing condition, the release decision can be based on functional evidence alone.
The missing constraint becomes review work
The study deliberately made this boundary observable. None of the 14 participants who completed the coding task included security requirements in their initial prompts. That denominator refers to observed coding sessions, not to all 15 interview participants. The interviews showed that participants could discuss security concerns and vulnerability classes, so the result cannot be reduced to “these developers knew nothing about security.” It indicates that relevant knowledge did not reliably cross from conversation or memory into the first AI request.
Only two participants independently caught unstated security issues during the sessions. The remaining 12 either did not review for security or expressed satisfaction with output that contained unresolved vulnerabilities. This pattern is not a controlled estimate of how developers behave everywhere. It is an observation from a small qualitative sample using one assistant setup and deliberately incomplete task prompts. Its value is that it makes the gap concrete: security concern can be present in a person’s knowledge while absent from the interaction that produces the code.
One reason the gap matters is that generated code often presents a strong completion signal. A response can contain the expected route, data fields, error handling, and a plausible persistence call. It may look organized and familiar. A developer who asked for a working feature can therefore receive exactly the kind of answer that feels like progress. The security question—who is allowed to invoke this operation, which values can be trusted, and which states must remain separate—does not necessarily announce itself in the same visual way.
That is how functional plausibility can conceal a security omission. In the study’s profile-update task, the generated implementation used the user_id supplied in the request body rather than a trusted session identity. The endpoint could therefore appear to perform the requested update while allowing a caller to name another account. The defect is not that the endpoint failed to execute its main operation. It is that the implementation treated a caller-controlled value as proof of authority. Functional correctness and authorization correctness crossed different boundaries.
The task also included a contrasting response. After completing an implementation, participant P12 explicitly asked the assistant whether any security concerns remained. That follow-up surfaced denial-of-service issues that the participant had not considered initially. The sequence is important: a deliberate security question changed what the interaction examined after the functional request had already been satisfied. It shows that a later query can be useful. It also shows that the query depended on the developer recognizing that a second security-oriented pass was needed.
The study reports that participants who iterated on their prompts with security in mind produced more secure outcomes than those who issued a single functional prompt and accepted the result. The authors qualify this observation: the sample is too small to characterize the size of the effect. The defensible conclusion is therefore not that a particular wording guarantees safe code. It is that prompt specificity and deliberate security-oriented interaction were associated with better outcomes in the observed sessions, while the tool itself did not consistently elicit those behaviors.
This creates a subtle asymmetry. The developers most able to catch a security problem may be the ones who already possess enough security knowledge to know what to ask next. Developers who rely more heavily on the assistant may be less able to identify which unstated assumptions require inspection. In that situation, the assistant lowers the effort needed to produce functional code while leaving the effective barrier to producing secure code in place—or even making it harder to see where that barrier lies.
The later review burden has several parts. First, the reviewer must identify the security boundary. An ordinary code-reading pass may follow control flow and confirm that the endpoint returns the expected response without asking whether the caller is entitled to affect the target record. Second, the reviewer must select a threat question. “Is this secure?” is broader and less actionable than asking who supplies identity, what input is trusted, or how simultaneous requests affect shared state. Third, the reviewer must inspect the assistant’s assumptions, including assumptions that were never stated because the prompt did not require them. Finally, the reviewer must decide whether the correction is complete or whether the same underlying class of issue may appear elsewhere.
The need for this work does not mean preventive security makes review unnecessary. It changes the review’s starting point. With security requirements visible, review can ask whether the implementation complies with known conditions and whether the design adequately addresses identified risks. Without them, review must first reconstruct the conditions. That distinction is why post-generation review is a compensating control rather than an equivalent replacement for secure-by-design reasoning.
Why the same knowledge is not applied automatically
The observed prompting gap is easiest to misunderstand if it is treated as a character judgment. The study does not show that participants stopped caring about security, nor that every omission arose from laziness. Several forces can produce the same behavior. AI tools are adopted partly for speed, so adding a careful security specification can feel like slowing down the task the tool was meant to accelerate. Developers may assume that familiar patterns are safe because they look conventional. They may expect testing to reveal problems later. Or they may know that security matters in general but not retrieve the relevant question for this particular task.
The interaction itself can reinforce those forces. A developer sees the assistant respond immediately to a functional description. The next turn is naturally used to refine the output, fix an error, or request a missing feature. Security becomes one possible follow-up among many rather than a condition that was present from the start. If the assistant does not mark a request as security-sensitive, the interface offers no visible reason to change course. The absence of a warning is not proof that the code is safe, but it can be experienced as permission to continue.
This helps explain why later review competes with other forms of completion. Developers may read the code, run it locally, and see tests pass. Those checks can establish that some expected behavior works. They do not automatically test whether a malicious caller can impersonate another user, whether a secret is exposed to the assistant, or whether a race condition causes one user’s data to cross into another’s storage location. Security review must ask a different class of question from the functional test, and the assistant’s initial framing may not have prepared either person or tool to ask it.
The timing also affects accountability. When a security requirement is explicit, the team can discuss whether it was implemented, waived, or tested. When it remains implicit, an omission can look like an individual failure discovered only after the code exists. The developer may still be responsible for the result, but the surrounding system has made the responsibility easy to overlook. Treating the problem only as a need for more careful individuals misses the way an interaction model distributes attention.
A useful diagnostic is therefore to follow a security property through the interaction. Was it present in the initial request? If not, did a later prompt introduce it? Did the assistant explain which assumption it used? Did the developer inspect the relevant boundary or only the visible behavior? Was the property tested, and was the result recorded as part of the decision to release? At each transition, security can become explicit, remain implicit, or disappear from consideration. The study’s contribution is to show that these transitions are part of security practice, not merely details of user experience.
The practical conclusion is deliberately limited. Teams should not assume that telling developers to “write better prompts” will solve the problem, because that still leaves security dependent on individual memory. Nor should they assume that adding a generic warning after every answer will create meaningful review; indiscriminate friction can be ignored, and the study did not test a warning program. The evidence supports a narrower direction: security needs to be made structurally visible at the interaction and review points where it is currently optional.
That direction raises the next question. If knowledge does not reliably become a security requirement at the first prompt, how should we interpret the cases where someone does catch an issue—and how should we separate what the study observed from what it merely suggests? A sound answer requires holding three things apart: the developer’s concern, the behavior the session elicited, and the strength of the evidence for the mechanism connecting them.