Making Tokens Indistinguishable Without Breaking Code

This chapter examines NOIR’s central representation defense: how can a client alter the embeddings sent to the cloud without replacing the prompt with nonsense? The specific focus is INDVOCAB, an indistinguishability-preserving vocabulary, and its adaptive randomized-response mechanism. The reader will learn what the mechanism changes, why it keeps token relationships stable, what the privacy budget ε means, and why a token-level guarantee must not be mistaken for absolute semantic anonymity.

Why ordinary noise breaks the task it is meant to protect

The problem begins with a mismatch between privacy mechanisms designed for simpler tasks and the demands of code generation. In a classification system, changing a word or perturbing a feature may leave the final label unchanged. In autoregressive code generation, a small change can propagate through the entire sequence. Replacing a keyword may alter syntax. Changing a function name may disconnect the prompt from the requested output. Distorting an instruction or a template can make every subsequent prediction less relevant.

There is a second problem when randomization is repeated independently. Suppose the same token appears in an instruction, a prompt, and a code example. If each occurrence is replaced with a different randomized token, the model no longer sees one stable concept. It sees unrelated symbols. The same damage occurs when the instruction template is perturbed separately from the user’s request. The model has learned correlations among the instruction, the task, and the desired output; independent randomization attacks those correlations even when each individual replacement looks small.

NOIR’s design starts from a different goal. The client should keep the token identity it needs for local processing, but the embedding value exposed to the cloud should not reliably identify which ground-truth vocabulary token produced it. This is the role of an indistinguishability-preserving vocabulary, or INDVOCAB. It creates a new set of token embeddings from the original vocabulary. On the client, the prompt still refers to the intended token. When the resulting embedding crosses the boundary, however, a cloud observer should have difficulty telling which original token it represents.

That distinction is easy to miss. INDVOCAB does not ask the client to write a different prompt. It changes the numerical representation associated with the prompt before local encoding and cloud computation. It also does not randomize each occurrence as a fresh, unrelated event. The vocabulary is prepared as a persistent mapping and reused. The client can therefore preserve the relationships among a prompt, its instruction, its template, and its expected code while presenting the cloud with embeddings that are difficult to distinguish by token identity.

The reason consistency matters can be seen in a small coding example. Consider an instruction that says a function should return the first duplicate in an array, followed by a test case that expects a particular result. The words, punctuation, and code symbols form a coordinated request. If “duplicate,” the function name, or the test syntax is replaced with unrelated tokens, the model may generate valid Python that solves a different problem. If their embeddings are changed in a consistent way while the client retains the original token sequence and relationships, the local model has a chance to learn how to interpret the altered representation.