One Model, Many Tasks: Audio, Tokens, and the Shared Interface
This chapter opens OpenAI Whisper’s internal operating model: how a waveform becomes a representation, how that representation reaches a text-generating decoder, and how one token interface supports several speech tasks. The goal is not to tour implementation files or recite Transformer terminology. It is to give the reader a usable boundary map—what each part owns, what crosses between parts, and why the same design that enables flexibility can also produce fluent but unsupported text.
From sound to a fixed representation
Whisper does not feed a raw audio file directly into its Transformer. The released implementation first asks an audio decoder to produce a mono waveform sampled at 16,000 times per second. It then computes a log-Mel spectrogram: a time-by-frequency representation that compresses the waveform into energy patterns across perceptually meaningful frequency bands. In the standard configuration, the spectrogram uses 80 channels, a 25-millisecond analysis window, and a 10-millisecond stride between successive windows.
That transformation gives the encoder a stable numerical surface. A waveform contains very fine-grained timing and amplitude variation; the spectrogram emphasizes how energy is distributed over time and frequency, which is more directly useful for speech patterns. The conversion is not a lossless description of everything in the original signal. It is a choice about what information the model should receive and at what resolution. Sampling, down-mixing, resampling, and spectrogram settings become part of the model’s input contract. A file that fails to decode, contains unusual channel behavior, or is fed at an unexpected rate is not merely an inconvenient input; it crosses the boundary incorrectly.
The standard model context is 30 seconds. At a 16 kHz sample rate, that is 480,000 samples before the spectrogram is formed. The implementation pads shorter audio and trims longer segments to the expected input length. The resulting spectrogram has 3,000 frames in the usual 80-channel configuration. This fixed shape makes the encoder predictable and makes training manageable, but it establishes a hard local horizon. Whisper can represent a 30-second view of a longer recording; it cannot place an hour of audio into one encoder pass. The longer lifecycle has to be built around repeated windows, a question taken up later in the series.
The encoder begins with two one-dimensional convolutional layers using the GELU activation. The second layer has stride two, reducing the time resolution before the audio representation enters the Transformer blocks. Sinusoidal positional information tells the encoder where each remaining feature sits in the sequence. Self-attention blocks then allow each position to relate to other positions within the audio context. The encoder’s output is not a transcript and does not make the final task decision. It is a contextual representation of the sound that another component can query.
That division matters. The front end owns the conversion from an external audio format to the model’s numerical input. The encoder owns acoustic representation within the bounded window. Neither one decides whether the output should be a transcription, an English translation, a timestamped segment, or a silence indication. Those decisions belong to the decoder’s token sequence.
The decoder is an audio-conditioned language model
Whisper’s decoder receives tokens representing the text generated so far and the features produced by the encoder. Its self-attention is causal: at each step, it can use the already supplied or generated token history, but not future output tokens. Its cross-attention looks back to the encoded audio. The decoder therefore combines two kinds of context: a language-side history and an audio-side representation.
This is the source of much of Whisper’s power. If a sound is ambiguous, the decoder can use surrounding words and the broader sequence to prefer a coherent interpretation. Long-range text context can help resolve a homophone or maintain a technical phrase. The model is not forced to identify every word from an isolated acoustic fragment. It predicts a sequence conditioned on the audio and on the instructions encoded in the prefix.
The same arrangement creates a risk that a conventional acoustic classifier would express differently. A language model is good at producing a plausible continuation. Plausibility is not identical to fidelity. If the audio is unclear, silent, badly aligned, or outside the model’s experience, the decoder may lean on learned language patterns and generate text that sounds right but is not present. The model card and paper describe hallucination, repetition, and unrelated output as real failure modes. The architecture explains why those failures are possible; it does not establish how often they occur in every deployment.
The decoder uses learned positional embeddings for its text context and cross-attends to the audio features. Its input and output token representations are tied, so the same learned vocabulary space participates in reading token history and scoring the next token. During inference, the released code can reuse the key and value projections from previous decoder positions through a key-value cache. That cache is an efficiency mechanism: it avoids recomputing the same self-attention projections every time another token is generated. It does not change the model’s authority over the audio or turn its scores into confidence estimates.
This architecture is best understood as a set of interfaces rather than a pile of components. Audio preparation hands a fixed representation to the encoder. The encoder hands contextual features to the decoder. The decoder consumes a structured token prefix, generates one token at a time, and returns scores from which a search strategy selects an output. The tokenizer translates between those token IDs and human-readable text or control symbols. Each boundary has a different responsibility, and a failure at one boundary can look like a failure at another if the lifecycle is not inspected carefully.
Special tokens make the task explicit
The shared interface is built from tokens that have meanings beyond ordinary words. A start-of-transcript token marks the beginning of the decoder’s answer. A language token identifies the spoken language. A task token selects transcription or translation. A no-speech token represents the possibility that the window contains no voice. Timestamp tokens describe positions relative to the current audio segment, with the model’s native timing resolution quantized to 20 milliseconds. Ordinary text tokens then fill in the content, and an end-of-transcript token closes the sequence.
This format turns a one-to-many problem into conditional sequence generation. The audio is the same kind of input, but the requested output can change according to the token prefix. The language token and task token are not an external switch that loads a different model. They are part of the sequence the decoder is trained to interpret. The model can therefore use relationships among tasks: multilingual speech recognition, speech translation, language identification, silence decisions, and timestamped transcription share an interface even when their outputs differ.
The arrangement also explains why the model can be more than a plain word recognizer. Timestamps are generated as tokens interleaved with text, so the decoder can express where a segment begins and ends within the current window. Language identification is performed by examining the distribution over language tokens at the beginning of decoding. The no-speech signal is gathered from the decoder’s probabilities and then combined with other transcription diagnostics. These capabilities are expressed in the same generative vocabulary rather than being separate, fully independent heads.
That unification has a practical advantage: one model can expose a coherent set of tasks without forcing an application to assemble many unrelated interfaces. It also has a hidden coupling. If task selection, language identification, timestamp rules, and text generation share the decoder’s state, a mistake or ambiguity in the early sequence can affect later tokens. The released decoder applies rules that suppress impossible or unwanted choices—for example, it can suppress blank outputs, non-speech symbols, or invalid timestamp orderings. Those rules are important, but they are guardrails around generation, not a proof that the generated sequence is faithful.
The tokenizer is the part that gives these tokens a usable vocabulary. English-only models use an English-oriented byte-level encoding; multilingual models use a refit vocabulary intended to reduce excessive fragmentation in other languages. The tokenizer also defines which language tokens are available, maps language names and aliases to codes, identifies timestamp-token ranges, and splits generated tokens into word-like units for timing. Languages without ordinary spaces require a different splitting strategy: the implementation can use Unicode decoding boundaries because a whitespace rule would be a poor model of their written form.
This is not a cosmetic difference. Tokenization controls how much output context a language consumes, how text is reconstructed, and how word or character timing can be inferred later. A tokenizer that fragments a language heavily may make the decoder spend more positions expressing the same content. A language-specific splitting rule may produce useful units for alignment without making those units linguistically equivalent to English words. The model can support multiple languages through the interface, but the quality and interpretability of the result still depend on training coverage, tokenization, audio conditions, and task-specific evidence.
The interface also has clear non-capabilities. The presence of a no-speech token does not make Whisper a comprehensively evaluated standalone voice-activity detector. A shared decoder and language tokens do not establish speaker diarization, speaker classification, or safe speaker identification. The model card explicitly warns that such capabilities may be possible after fine-tuning but have not been robustly evaluated. A token is an instruction or prediction target; it is not a license to infer a capability the evidence does not support.
What the architecture can—and cannot—tell us
The code and paper directly support a description of how Whisper operates: fixed audio features pass through an encoder; a decoder cross-attends to those features while generating a token sequence; special tokens condition language, task, silence, and timing; and caching and filtering support the autoregressive loop. Those observations are strong because the implementation exposes them and the paper describes the same design.
They do not, on their own, prove which part causes the reported robustness. The authors explicitly leave open how much of Whisper’s advantage comes from the encoder, the decoder, or their combination. The architecture makes several hypotheses plausible. Diverse audio may improve the encoder’s acoustic representations. A broad text-conditioned decoder may support transfer across domains. Multitask tokens may encourage shared structure. But the causal contribution of each mechanism requires ablation or other evidence, not confidence derived from a diagram.
The most useful mental model is therefore conditional. Whisper succeeds when the audio reaches the expected representation, the encoder captures relevant acoustic context, the task prefix matches the desired output, the decoder’s language context helps rather than invents, and the search rules keep generation aligned with the window. It can fail when any of those conditions is weakened: an audio mismatch, a language with thin coverage, an ambiguous task, a long recording that exceeds local context, or a decoder that produces a coherent continuation without enough acoustic support.
That model will become concrete in the next stage of the series. Rather than treating the API as a single action, the next chapter will connect training breadth, the shared interface, and measured behavior into one evidence-calibrated picture. The architecture tells us what could happen; the evidence must tell us what actually happened, under which languages and conditions, and where the explanation remains incomplete.