Making a Jailbreak Speakable
SWhisper’s voice-aware prompt layer addresses a specific failure in acoustic prompt injection: a prompt that works as text may stop working when it is spoken, synthesized, transmitted, and interpreted by a voice interface. The focus here is not how to write a jailbreak in the abstract, but how speech changes the requirements for a prompt and how SWhisper responds. The reader should come away able to distinguish four jobs that are easy to blur together: preserving meaning, staying within a short voice interaction, remaining intelligible, and transferring to a black-box target.
That distinction matters because language models consume tokens or learned audio representations, while people produce speech. A text attack can exploit a sequence of tokens that has no natural spoken form. A voice interface cannot pass that sequence unchanged: it must be rendered as sound, captured, and either transcribed or interpreted directly as audio. Every one of those conversions can remove, substitute, or rearrange information. The prompt is effective only if the model receives enough of the intended structure to respond as the attack assumes.
Why text success does not survive speech
A conventional text jailbreak can optimize for the target model’s internal token behavior. If a sequence of unusual symbols or awkward fragments lowers the model’s resistance, its lack of conversational meaning may not matter. The text interface accepts the sequence exactly as written. Speech introduces a different contract. The words must be pronounceable, the acoustic signal must survive a channel, and a speech recognizer or audio-native model must recover the intended relation among the words. A suffix that is effective in the text representation can be functionally absent once the system has to hear it.
SWhisper frames this as a three-way constraint. Intelligibility and semantic fidelity require the spoken prompt to remain clear and to preserve the intended meaning. This requirement applies whether a speech-to-text component produces an explicit transcript or an audio-native model works from the acoustic representation itself. If words are substituted or dropped, a target response may no longer be conditioned on the same request. A prompt can therefore fail before the model has an opportunity to refuse or comply.
Brevity is a system constraint, not merely a preference for concise writing. Voice APIs and audio interfaces impose duration limits, and a long exchange creates more opportunities for interruption, detection, or context drift. A multi-turn text strategy may gradually steer a model, but a covert acoustic input is designed around a single query in SWhisper’s threat model. The usable prompt has to carry enough structure in a small time budget. This rules out many verbose templates even when they are linguistically natural.
Transferability matters because SWhisper treats the target as a black box. The attacker may optimize against a surrogate model offline, but does not have the target’s parameters or direct access to the target device. A prompt that exploits one model’s exact token probabilities may lose its effect when the model, audio front end, or policy behavior changes. A voice-friendly design must preserve a meaningful instruction while leaving enough generality for cross-model transfer. That is a different objective from maximizing a white-box loss on one text model.
These constraints interact rather than forming a checklist. Shortening a prompt can remove the context that makes its instruction coherent. Adding linguistic explanation can improve meaning while exceeding a voice interface’s time budget. An unusual suffix may increase adversarial pressure in token space but raise word error after speech synthesis. Optimizing only one dimension can therefore damage the others. SWhisper’s design treats prompt construction as a system with competing objectives, not as a final formatting step after a text jailbreak has already been found.
Word error rate, or WER, makes this interaction measurable. It compares the words recovered from generated speech with the intended reference and counts deletions, insertions, and substitutions. A lower WER indicates that the spoken form preserved more of the reference, although it does not by itself prove that a model will respond unsafely. This metric supplies a bridge between linguistic design and the acoustic pipeline: it shows whether a prompt that appears effective on paper still arrives as the same instruction at the receiver’s software boundary.
That bridge also explains why baseline categories behave differently. Optimization-based methods can generate short sequences, but their token-level artifacts are often unnatural when spoken and can produce high transcription error. Heuristic methods tend to preserve ordinary language better, yet may be verbose or brittle against stronger targets. In a speech-driven system, a method must be judged after the voice conversion, not only at the text interface. SWhisper’s claim is that prompt construction should be optimized for the actual interface in which the model is exposed.
The requirement survives a change in architecture. In a speech-to-text pipeline, the transcript is an explicit mediator: an error in recognition changes the text that the language model receives. In an end-to-end audio-language model, the mediator is hidden inside the model’s audio processing, but the input still has to remain interpretable. An approach that assumes the presence of a perfect transcript can fail on an audio-native target; an approach that ignores speech intelligibility can fail before either target class makes its decision.
This is why SWhisper separates a voice-friendly prompt layer from the acoustic delivery layer. The first makes the intended instruction suitable for speech. The second attempts to preserve the resulting waveform through a variable receiver. The layers depend on each other, but they solve different problems. If the prompt is semantically incoherent, perfect delivery preserves incoherence. If the prompt is well formed but the channel destroys its words, the model receives a different request.
How SWhisper gives the prompt a spoken form
SWhisper uses a structured template rather than presenting an optimized suffix as a complete prompt. The template supplies a predictable linguistic frame around the attack objective. In conceptual terms, it contains a rule-based instruction, an abstract target goal, a desired response direction, and an adversarial suffix that can be refined. The structure gives the model a recognizable instruction pattern while leaving the suffix as the part that optimization adjusts. It also makes the prompt compact enough to be spoken as one coherent interaction.
The structured frame matters because it changes what the suffix has to do. A raw suffix would need to carry much of the attack’s meaning while also exploiting the model’s weaknesses. A template can carry the high-level semantics and conversational form, allowing the optimized portion to act within a more stable context. This is not a claim that structure alone defeats a model. It is a division of responsibility: linguistic organization provides a usable surface, while optimization searches for additional pressure within that surface.
The search is constrained by semantic and phonetic validity. SWhisper restricts the vocabulary available to the suffix so that unpronounceable or semantically irrelevant candidates are excluded. That restriction shrinks the space of possible token sequences, but the reduction is purposeful. A larger search space is not automatically better if most candidates cannot survive speech. The relevant candidate is not simply one that reduces an adversarial loss in text; it is one that remains linguistically plausible enough to be rendered, heard by the machine, and connected to the intended meaning.
Semantic regularization expresses the same idea in another form. SWhisper combines an objective for inducing the targeted model behavior with a language-model term that favors natural continuations. The balance prevents the optimized suffix from drifting too far into token patterns that have little linguistic coherence. The regularizer is not a guarantee of ordinary prose, and it does not turn the prompt into a benign request. Its role is to keep the adversarial search aligned with a distribution that can be spoken and interpreted.
Phonetic validity adds a constraint that a text-only method does not need. The spelling of a token is not enough; the token needs a usable spoken realization. A word can be semantically relevant but awkward to pronounce in context, or it can be pronounceable while contributing little to the intended instruction. SWhisper’s semantic vocabulary and speech-oriented filtering address that interface between token choice and acoustic realization. The point is not to make the prompt elegant. It is to keep the sequence from collapsing when a voice system becomes the owner of the next representation.
The optimization also accounts for errors that will occur after the text leaves the search environment. Because the attacker does not know the exact target microphone or acoustic setting, SWhisper synthesizes the current instruction as speech, applies perturbations meant to represent channel deviations, and evaluates the recovered text. The prompt is thus selected partly for how it behaves after distortion, not only for how it behaves in an ideal text context. This is a cross-layer design choice: the content layer is asked to tolerate some uncertainty in the physical layer.
The flow is not a recipe for producing a harmful command. It is a model of where voice requirements enter the design. The objective is abstract; the structure makes it linguistically legible; the constraints keep candidates compatible with speech; the distortion check tests a later representation; and the output must satisfy brevity, intelligibility, and transferability at the same time. Each stage owns a different failure mode, so a result at one stage cannot be assumed to prove success at the next.
The reported ablation makes that division of labor concrete. In the tested Grok-4 setting, removing the structured prompt template reduces the non-refusal score from 0.780 to 0.178 and the specific-convincing score from 0.733 to 0.169. Removing the optimized suffix produces a smaller decline, to 0.740 and 0.703. Removing semantic regularization produces intermediate values of 0.660 and 0.645. The exact values belong to the study’s selected target and conditions, but the relative pattern is informative: the linguistic frame is not ornamental, and a voice-compatible attack cannot be reduced to appending an arbitrary optimized string.
The ablation should not be read as proof that the template is universally the most important component. It is a controlled removal test, not a complete causal theory of all voice interfaces. The result does support a narrower design judgment. When a prompt must be spoken, the structured instruction supplies a stable semantic context that the other components do not replace. The suffix contributes adversarial strength, while semantic regularization helps preserve naturalness and robustness. Their roles are complementary rather than interchangeable.
This also changes how “optimization” should be understood. It is not simply a search for the most aggressive token sequence. The search has a target behavior, but it operates under a linguistic budget and an acoustic uncertainty. Candidates that are excellent in one representation can be rejected because they cannot be spoken clearly or because their meaning is too fragile under conversion. SWhisper therefore treats the prompt as an interface artifact with multiple owners: the model’s token behavior, the speech generator, the recognition or audio-processing stage, and the downstream policy all constrain what counts as effective.
There are limits to this approach. A universal template may not fit every task or context. A semantic vocabulary can improve pronounceability while excluding candidates that would have transferred better to a particular target. Distortion simulation can make the prompt more robust to modeled deviations without covering every room, device, or future audio front end. And a surrogate model is still not the target: black-box transfer can fail even when the prompt remains intelligible. The paper identifies context-aware templates and transfer-aware or multi-model objectives as possible improvements, which is an acknowledgement that voice compatibility and generalization are not solved once and for all.
The central lesson is narrower and more useful than a claim that jailbreaks have become “audible.” Speech changes the unit of reliability. A text attack is not ready for a voice interface until its meaning can survive pronunciation, duration limits, model transfer, and the transformations introduced by the acoustic path. SWhisper’s prompt layer makes those requirements explicit and couples them to optimization. But a well-formed spoken prompt still has to cross the receiver without being deformed beyond recognition. The next engineering problem is therefore not how to make the instruction speakable, but how to preserve it through air and hardware.