How an Embedding Becomes a Semantic Leak
This chapter follows NOIR’s central problem at the point where privacy claims are most often overstated: the cloud receives no readable prompt, yet it may receive a representation that preserves enough structure to recover one. The focus is the path from token embeddings or fine-tuning gradients to reconstructed prompts and code. Understanding that path lets the reader distinguish an embedding-inversion attack from a gradient attack and read the later privacy results without treating any single score as a complete definition of safety.
The cloud sees a useful representation
A language model does not receive a prompt as a stream of meanings. It receives tokens, and each token is converted into a numerical embedding. The model then transforms those embeddings through attention layers into richer internal states. The numbers are not readable text, but their usefulness depends on the information they preserve. They must retain enough about the instruction, syntax, relationships among tokens, and likely continuation for the model to produce the requested code.
That is the first tension in private code generation. If an intermediate representation preserves nothing, it cannot support the cloud’s computation. If it preserves everything the model needs, it may preserve enough for a curious cloud to infer the original. Privacy is therefore not achieved merely by replacing a sentence with a vector or by giving the vector an opaque name. The relevant question is whether the mapping from text to representation is difficult to reverse for the adversary who controls the cloud-side computation.
NOIR’s threat model names an honest-but-curious cloud. “Honest” means that the provider follows the intended protocol. “Curious” means that the provider uses the information available during that protocol to learn more than the client intended to reveal. That information includes prompt embeddings during inference and, during fine-tuning, gradients that have passed through the cloud-hosted part of the model. The provider need not break the network or alter the model to pose a risk. It can analyze the objects it legitimately receives.
The first attack path is embedding inversion. NOIR describes PromptRA, or prompt reconstruction attack, as an attempt to recover the client’s prompt from its embedding. The paper uses Vec2Text, an embedding-inversion system with a base stage that proposes text and a refinement stage that repeatedly re-embeds and corrects that proposal to increase similarity to the observed embedding. The result does not have to reproduce every token to be useful. A recognizable instruction, a private function name, or the gist of a test case may reveal the part of the prompt that matters most.
The second path arises during fine-tuning. In split tuning, a client’s example affects the loss at the model output, and the gradient is propagated backward through the cloud-hosted middle blocks. A cloud that observes the relevant gradient can use a gradient-based reconstruction attack, which NOIR calls BiSR, to infer the training prompt and its associated code. This is a different surface from an embedding inversion attack. An inversion attack starts from a forward representation and tries to reverse it. A gradient attack starts from how the model changed—or would change—in response to an example and uses that signal to reconstruct the example.
The distinction matters operationally. A system may protect the forward embedding yet expose a revealing training gradient. Conversely, a design may keep fine-tuning local while sending an inference representation that remains invertible. Privacy cannot be checked only at the moment the prompt crosses the network. It must be checked at every representation that crosses the trust boundary and at every state the cloud can retain or analyze.
The cloud can also use its own model as part of a code attack. NOIR’s CodeRA evaluation includes feeding prompt embeddings into the cloud-hosted middle blocks together with a decoder available to the cloud, feeding reconstructed prompts into a full model, or applying BiSR to gradients to generate an approximation of the client’s output code. These routes exploit different available information, but they share a principle: the cloud does not need the original client-side decoding process if it can construct a plausible substitute around the information it observes.
These routes should not be collapsed into one imagined attack. An embedding inversion attack asks whether the forward representation contains enough signal for a separately trained system to propose the input. A cloud-side code attack may have more context: it can combine the observed representation with the model blocks it already controls and a decoder it chooses. A gradient attack has a different advantage again, because the direction and magnitude of an update carry information about the example that produced the loss. The attack surface is consequently a function of both the message and the cloud’s surrounding capabilities.
Nor is reconstruction a single attempt to “decrypt” a vector. Vec2Text first produces a plausible text hypothesis and then repeatedly re-embeds that hypothesis, comparing it with the target representation and correcting it. This makes the cloud’s auxiliary model and training data part of the practical threat. The quality of the recovered text can vary with prompt length, the presence of code or tests, and how well the attacker’s model matches the target representation. BiSR follows a different route by matching forward and backward signals. Its success depends on the gradients the cloud can collect and the assumptions it makes about the target model, but the existence of that route means fine-tuning cannot be treated as a private afterthought.
Consider a prompt asking for a Python function that finds the first duplicate in an array. If the cloud reconstructs the exact wording, the leak is obvious. But exact wording is not the only meaningful outcome. A recovered implementation strategy, a function name, or a code fragment that passes the same test can disclose the underlying logic. For a commercial system, the same pattern could reveal a proprietary algorithm, a sensitive package, an internal variable convention, or a security-relevant behavior even when the recovered text contains errors.
The important causal chain is therefore:
- The client converts text into a representation that is useful to the model.
- The cloud receives or participates in computation over that representation.
- An attacker uses a learned inversion or gradient-matching method to propose text or code.
- The proposal is judged not only by textual similarity but by whether it reveals a usable gist or functioning behavior.
A privacy mechanism must interrupt that chain without destroying the first step’s utility.
A leak must be judged by what it enables
NOIR separates several outcomes because “reconstruction” is not one uniform event. For prompts, BLEU and ROUGE measure overlap or gist between the recovered text and the ground truth. These scores can show that the cloud has recovered a meaningful portion of an instruction even if the sentence is not identical. For code, the paper also uses CodeBLEU, which is intended to reflect code syntax and semantic similarity. These measures answer an important question—how close is the reconstruction?—but they do not fully answer whether the recovered code works.
Functionality is measured with Fusi, a similarity based on shared unit tests passed by the reconstructed and ground-truth code. It is stricter than ordinary token overlap. A reconstruction can look similar while changing one condition that makes it fail. The reverse can also happen: a short or structurally different reconstruction can still implement the same behavior. If it passes a test that the client’s code passes, the cloud has learned something more consequential than a vague textual resemblance. The source therefore treats any shared functionality as a code-functionality leak under its security game.
The paper adds correctly reconstructed tokens, or CRT, for prompt and code, and separately records whether sensitive information such as imported packages, variable names, and function names appears in the recovered output. This layered measurement is useful because different harms sit at different levels. A low CRT score may coexist with a recognizable prompt gist. A high CodeBLEU score may not imply that a secret identifier was recovered. A functioning code fragment may be the most valuable leak even when its surface form differs from the client’s version.
The defense-free experiment establishes why these distinctions are needed. In the reported MBPP settings, PromptRA achieves strong text-reconstruction scores and high prompt attack success in conditions where test cases are absent. When test cases are included, prompt reconstruction weakens, but CodeRA remains strong: the paper reports high BLEU, ROUGE, and CodeBLEU values, along with substantial functionality leakage. Across the cited training and test conditions, the defense-free mechanism reaches roughly 99 percent code-privacy attack success, about 97 percent code-confidentiality success, and roughly 46 percent code-functionality leakage. The paper also reports CRT values of 0.28 for prompts and 1.00 for code in the highlighted MBPP assessment, with sensitive-information leak near 0.98.
Those figures should be read as measurements under a particular experimental setup, not as a universal law about every embedding. They show that the attack methods can recover meaningful content from the information available to the cloud in the paper’s defense-free setting. They do not establish that every model, prompt length, tokenizer, or training procedure has the same leakage. The threat model also excludes compromised employees inside the client organization and assumes that the cloud cannot query the client’s encoder or see raw inputs. Those exclusions matter because an insider or an oracle could supply information not present in the evaluated attack surface.
The example of a code-generation task also clarifies why privacy and correctness cannot be separated. Suppose a recovered function contains the same algorithm but changes the variable names. Token-level scores may fall, yet the intellectual property may still be exposed. Suppose instead that the recovery preserves much of the wording but fails every test. That may be an information leak without a direct functionality leak. A sensible audit reports both rather than declaring the result safe because one metric is low.
The same caution applies to frequency analysis. If a cloud observes many representations, it may look for recurring token or sequence patterns and match them to public data. Common instruction-template tokens are not equivalent to client-specific secrets, but their recovery can reveal how prompts are structured. NOIR’s later defense is evaluated against such attacks, but the existence of the attack path explains why consistent token handling and repeated observations matter. A one-time inversion result is not the only way a representation can leak.
This chapter’s conclusion is deliberately narrower than “embeddings are unsafe.” An embedding is a mediator between the client’s text and the cloud’s computation. Because it mediates useful model behavior, it may carry recoverable information. The right question is how much information an adversary can infer under a stated mechanism and threat model, and whether the inference reveals a gist, a secret identifier, or functioning code. The reported defense-free attacks show that raw-text confidentiality is not enough; semantic confidentiality requires an additional design layer.
That conclusion changes how NOIR’s architecture should be read. Moving the encoder and decoder to the client can keep raw prompts and final code out of the cloud, but it does not automatically make the intermediate representation unintelligible. The next step is to examine that split itself: which blocks remain local, what the cloud still processes, and which assumptions define the boundary.