What the Experiments Actually Show
The SWhisper study makes a claim about a chain, not a single button: a covert signal can be delivered through an acoustic path, retain enough speech content to be interpreted, and contribute to an undesirable response from a speech-driven model. The focus here is how to read the evidence for that claim. The practical value is a disciplined separation of model behavior, speech intelligibility, physical robustness, and human perception, so the reported results can be taken seriously without being inflated into a guarantee of universal compromise.
That separation is necessary because “attack success” can hide several different failures. A waveform may not survive the receiver. A recovered waveform may be transcribed incorrectly. A model may receive the intended prompt but refuse it. A person may notice an artifact even if the model responds. SWhisper evaluates these stages with different measurements and compares its voice-aware method with text-oriented baselines under a shared audio path. The numbers matter, but only when attached to the question each one actually answers.
Four measurements, four different questions
The study uses a harmful-behavior subset of the AdvBench benchmark and evaluates representative prompts rather than treating every possible request as equivalent. It compares SWhisper with GCG, IGCG, PAIR, and Auto-DAN, covering optimization-based and heuristic approaches. Text-based target models receive prompts through a text-to-speech front end and the common acoustic pipeline; end-to-end audio-language models process audio directly. This setup makes the comparison relevant to voice interfaces, because a baseline is not allowed to remain in its easiest text representation while SWhisper bears the cost of speech conversion.
The first pair of measures concerns the target model’s response. Non-refusal, or NR, records how often the model does not explicitly refuse the harmful request. It is a behavioral measure of compliance with the tested prompt, not a measure of whether the physical signal reached the microphone. Specific-convincing, or SC, adds the quality of the response: a non-refusal receives a higher score when it contains detailed and persuasive harmful content. A response that avoids a stock refusal phrase but remains vague should not count as equivalent to a detailed response.
This is why the paper uses the StrongREJECT framework rather than relying only on a list of refusal phrases. Refusal-phrase matching is easy to interpret but can be misleading. A model might avoid a particular phrase without giving a harmful answer, or it might give a harmful answer without using the expected vocabulary. NR and SC provide a more nuanced view of what the model actually returned under the study’s evaluator. They still remain bounded measures. They quantify undesirable behavior on selected prompts and targets; they do not measure physical reachability in an uncontrolled environment, the likelihood of a real victim being targeted, or the social harm that would follow from an answer.
The second measure is word error rate, or WER. It compares a transcription of the generated speech with the reference text, counting words that are deleted, inserted, or substituted. WER is upstream of the model response. If a carefully designed prompt is rendered as speech but arrives with many word errors, the target is no longer being tested on the intended prompt. A high WER can therefore explain why a text method that looks strong in a token interface weakens after voice conversion. It does not prove that a low-WER prompt will jailbreak a model; it establishes that the linguistic payload survived a particular conversion path.
The last two questions concern the physical and perceptual conditions.