What the Measurements Actually Show
The subject here is not whether PromptEcho can be made to produce plausible text in a demonstration; it is what the study’s measurements establish about semantic exposure from merged-token traffic. This chapter separates data, metrics, interpretation, and limits so that the reported percentages are not mistaken for universal attack rates. The practical judgment is conditional: first-segment results indicate a meaningful privacy channel under the evaluated settings, while longer responses, unusual structures, unseen domains, and changed service behavior progressively weaken the reconstruction.
A percentage has a dataset and a threshold attached
PromptEcho’s evaluation uses two kinds of data because the attack has two different learning problems. The first is a large plaintext alignment set derived from UltraChat and GPT-4 Turbo responses. The paper uses a 570,000-example general-inquiry subset to teach the relationship between a single token’s character length and the language that can produce it. In this setting, the alignment is deterministic: the response is known, the tokenizer can be applied directly, and each observed primitive length has a corresponding token.
The second kind of data is meant to approximate deployment. The authors sample 10,000 prompts for GPT-4o and 10,000 for DeepSeek-V3, send them through the live services, capture the encrypted interactions, and derive merged-token length sequences. They use 9,000 examples from each service for training and 1,000 for testing. They also remove test cases that fully overlap with training or share the first eight words with a training prompt. This reduces a simple memorization explanation, but it does not turn the experiment into a zero-data attack. PromptEcho still receives service-specific merged traffic for adaptation, and the results remain tied to the calibration regime described by the paper.
The metrics operate at different levels. Edit distance measures character-level disagreement. ROUGE-1 and ROUGE-L measure lexical overlap and sequence structure. Sentence-embedding cosine similarity measures semantic alignment between the original response and the reconstruction. A response can have low exact overlap but remain close in meaning, or share a generic template while losing the sensitive part. No single metric captures all of those outcomes.
The paper defines Attack Success Rate, or ASR, as the fraction of samples with cosine similarity above 0.5. That threshold is intended to represent meaningful semantic recovery for topic and intent identification, based on prior human-subject evidence cited by the authors. A cosine score above 0.9 is stricter and indicates high semantic similarity; it is not the same event as passing the ASR threshold. Exact lexical or character matches are stricter still.
This distinction matters when reading the headline figures. “The attack succeeds” in the paper’s main ASR tables means that a reconstruction crossed the semantic threshold of 0.5, not that every word was recovered. The presentation page separately summarizes 42.5 percent topic inference and 25.5 percent of responses reconstructed above a cosine similarity of 0.9 across the evaluated GPT-4o and DeepSeek-V3 setting. The detailed table breaks results down by victim model and by how many leading segments are included. Those figures describe different slices and thresholds.