Does It Work? Utility, Attacks, and the Limits of the Claim
NOIR’s mechanisms now make a precise promise: the cloud should have difficulty recovering client-specific prompt and code meaning from protected embeddings, output mappings, and split-tuning signals. The question here is whether the reported evidence supports that promise without sacrificing functioning code or imposing an impractical client burden. Reading the attack metrics, utility results, cost measurements, and limitations together gives a more useful judgment than any headline Pass@1 or privacy percentage.
What the results support
The first task is to keep the different attack questions separate. PromptRA asks whether the cloud can reconstruct a recognizable prompt from the information associated with it. CodeRA asks whether it can recover the generated code or a meaningful approximation. Frequency analysis asks whether repeated token or sequence patterns can be matched to public data. These attacks do not have the same inputs or the same success condition, so a result for one cannot be silently substituted for a result for another.
NOIR’s measurements reflect that separation. Prompt reconstruction is assessed with BLEU and ROUGE, which indicate overlap or a clear gist. Code reconstruction adds CodeBLEU for code-oriented similarity and Fusi for shared functionality. CRT records correctly reconstructed tokens, while the sensitive-information measure checks whether items such as imported packages, variable names, and function names appear in the recovered code. A system can lower token overlap without hiding a proprietary algorithm, or prevent a functioning reconstruction while still exposing some prompt vocabulary. The metric must match the harm.
The defense-free baseline demonstrates the scale of the problem that NOIR is trying to change. In the paper’s MBPP experiments, the cloud can often recover meaningful prompts and code when no privacy mechanism is applied. PromptRA reaches high prompt attack success in the conditions reported, especially when test cases are absent. Adding test cases makes prompt reconstruction harder for the evaluated inversion methods, but CodeRA remains strong: the paper reports high code similarity and substantial code-functionality leakage. The highlighted defense-free results include CRT of 0.28 for prompts and 1.00 for code, and sensitive-information leakage of 0.98.
The reason these baseline results matter is not that they predict every cloud attack. They establish that a useful model representation can carry enough information for an observer to reconstruct more than a few isolated tokens. A cloud may recover the purpose of a function, identify a package, or produce code that passes a shared test even if it does not reproduce the exact original text. That is already a material privacy failure for a client whose prompt or code contains a trade secret.
Under NOIR, the reported reconstruction results fall sharply. In the experiments discussed by the authors, PromptRA’s privacy attack success on MBPP falls from the defense-free levels to much lower values under the IND protection, with a particularly strong reduction at stringent budgets. For code completion on HumanEval, the paper reports no clear-gist prompt reconstruction in the evaluated PromptRA setting. CodeRA results with the combined INDVOCAB and LTOKENIZER defenses report zero privacy and confidentiality attack success in the highlighted experiments and zero functionality attack success. On MBPP, the paper reports CRT values of 0.028 for prompts and 0.108 for code under NOIR, compared with 0.28 and 1.00 for the defense-free mechanism. Sensitive-information leak falls to 0.038 from 0.98.
These are strong observations within the paper’s test design. They support the claim that the combination of embedding protection and secret output indexing makes the evaluated reconstruction procedures substantially less effective. They do not mean that NOIR’s cloud-visible representation contains no information, that all attacks fail, or that the client can ignore its own threat model. The paper’s thresholds for a clear gist and its chosen attack implementations define what counts as success in these experiments.
The frequency-analysis result adds a different kind of evidence.