When the Prompt Is Gone but the Meaning Remains
This series examines NOIR, a privacy-preserving code-generation design that lets a cloud contribute the middle of an open-source language model without receiving a client’s raw prompt or final code. The question is narrower and harder than whether text is encrypted in transit: can the cloud still infer what the prompt means from the representation it processes? By the end of this series, the reader should be able to follow that boundary, distinguish a privacy guarantee from an assumption, and judge the trade-off among semantic privacy, useful code, computation, and cost.
The privacy problem is semantic, not merely textual
Imagine a developer asking an online coding assistant to complete a function that implements an internal algorithm. The prompt might contain a proprietary data structure, a private test case, a security-sensitive constraint, or a function name that reveals the architecture of a product. The generated code can be just as sensitive: it may expose the algorithm’s logic, the names of internal services, the libraries a company relies on, or a working fragment of a system that has never been published.
The ordinary picture of privacy is easy to draw. The prompt is written on the client, sent to a cloud model, and the answer comes back. If the prompt is not stored, or if the connection is encrypted, it is tempting to say that the sensitive information is protected. But a language model cannot work on an empty interface. It receives some representation of the prompt, performs computation on it, and returns information from which code is produced. The privacy question therefore moves inside the pipeline: what does that representation preserve, and who can interpret it?
An embedding is a numerical representation rather than a readable sentence. That difference in format does not make it meaningless. It is designed to carry the information the model needs for prediction, including relationships among tokens, syntax, instructions, and likely continuations. A cloud that sees the embedding may try to reverse that transformation. The NOIR paper evaluates attacks that use an embedding to reconstruct a prompt, and attacks that use the hosted model or its decoder to reconstruct code. During fine-tuning, gradients crossing the cloud boundary create another route: they can reveal information about the examples that produced them.
The risk is not limited to recovering every character exactly. A reconstruction with a clear gist may reveal what a private function does even if variable names are wrong. A code reconstruction can be dangerous when it preserves a proprietary algorithm, imports a sensitive package, or passes a unit test. In those cases, a low-level mismatch in the recovered text does not make the leak harmless. The cloud may have learned enough to reproduce the useful part of the client’s work.
This is why the title’s phrase “hide prompt semantics” matters. It asks whether the cloud can infer the substance of a request, not whether it can print the original bytes. It also forces a distinction between two directions of leakage. The prompt can be reconstructed from what enters the cloud, while the generated code can be reconstructed from what the cloud sees during or after generation. A design that protects only one direction leaves the other open.
What private code generation must protect
NOIR describes the goal as three related protections. Raw-data confidentiality means that the client’s prompts, private training examples, and generated code are not shared as plain data with the cloud. Encoded-information privacy goes further: the cloud should not be able to turn the representation it receives into an interpretable account of that data. Code confidentiality concerns the syntax, semantics, and functionality of code in the client’s data and outputs.
These are different properties because they fail at different points. Keeping a prompt on the client addresses raw-data transfer, but it does not by itself address an embedding inversion attack. Randomizing a token embedding may make the prompt harder to recover, but a shared tokenizer or output index can still tell the cloud which token was generated. Hiding the text of a training example does not automatically hide the gradients produced while adapting a model. A useful privacy design must follow the data through each transition rather than attach one label to the whole service.
The client and cloud also own different assets. In the NOIR setting, the client owns the sensitive prompt, local training data, and the code that must be generated. The cloud supplies the capacity of the middle part of an open-source LLM. The local and cloud pieces must exchange enough information to preserve the model’s ability to generate code, but not so much that the exchange becomes a second copy of the client’s intellectual property. The client’s tokenizer and local model pieces are therefore security-relevant state, not incidental implementation details.
This leads to a practical vocabulary for examining the system:
- The client owns the input and wants a useful result without surrendering the input’s meaning.
- The cloud performs valuable model computation but is treated as honest-but-curious: it follows the protocol while trying to learn more than it should.
- The representation is the intermediate object that crosses the boundary. It may be numerical, but it is still an attack surface.
- The decoder and tokenizer determine how an enriched representation becomes output code and how output positions map back to tokens.
- The protocol determines which messages, gradients, and states are exposed during training and inference.
This model also exposes what a privacy claim must include. “Private” is incomplete without naming the protected asset and the adversary. Is the goal to hide raw prompts, prevent a recognizable prompt gist, stop recovery of functioning code, keep model parameters secret, or all of these? Is the cloud merely curious, or may it alter its model, collude with an insider, or combine many observations with public data? Does the design preserve utility on one benchmark, or under the workload and latency a real client cares about?
The evidence in the source material makes these questions concrete. In a defense-free setting, the reported attacks recover meaningful prompt and code information, including high code similarity and nontrivial functionality leakage. NOIR’s later evaluation uses separate measures for prompt similarity, code similarity, correctly recovered tokens, sensitive information, and shared unit-test behavior. That separation is important: a system can reduce exact token recovery while still leaking a function’s purpose, or produce code that is semantically useful to an attacker without matching the client’s text word for word.
Two ways to move the boundary
The main subject of this series, NOIR, takes a representation-centered route. It keeps a lightweight encoder and decoder with the client and leaves the middle blocks of an open-source LLM in the cloud. The client encodes the prompt before sending anything across the boundary, and the cloud returns an enriched representation rather than final readable code. The client then performs the last step of generation locally. This placement protects the raw prompt and decoded output, but it does not make the intermediate representation safe by location alone. The representation must also be made difficult to interpret, and the output token mapping must not give the cloud an easy way around that protection.
A second route appears in BLB, which addresses private Transformer inference through hybrid homomorphic encryption and secure multi-party computation. In that setting, the client holds private input and the server holds model weights. Homomorphic encryption handles linear operations such as matrix multiplication, while MPC handles nonlinear operations such as Softmax and GeLU. The computation is protected cryptographically, but the system can become communication-bound: conversions between the two protocols and fixed-point truncations consume much of the budget. BLB responds by fusing compatible operators, using CKKS for fused linear computation, and optimizing the matrix-multiplication packing.
The contrast is useful because the two designs move different parts of the problem. NOIR asks: can the cloud use a powerful model on a representation without recovering the prompt or code semantics? BLB asks: can two parties compute a Transformer result without revealing the client’s input or the server’s weights, while keeping communication and latency manageable? Both are private-inference ideas, but their claims should not be merged. NOIR is built around open-source LLM components, local token handling, and semantic reconstruction attacks. BLB is built around cryptographic computation, numerical representation, and protocol overhead.
Neither route makes privacy a single switch. A representation can be hard to interpret yet still expose patterns through repeated observations. A cryptographic protocol can protect computation yet impose a network cost that changes whether the service is usable. A stronger privacy budget can reduce leakage while lowering code quality, and more candidate generations can recover some utility at additional cost. The right judgment therefore has at least four dimensions: what the cloud sees, what it can infer, how useful the result remains, and which assumptions make the claim valid.
The next step is to examine the first of those dimensions directly. If the cloud receives only an embedding or a gradient, what can it reconstruct, and when does a reconstruction become a meaningful leak rather than an approximation? Until that question is answered, “the prompt never leaves the device” describes a transport boundary—not yet a semantic one.