When the Audio Keeps Going: Decoding Long-Form Speech Reliably
OpenAI Whisper is trained around 30-second audio windows, while the recordings people actually want transcribed often last for an hour. This chapter diagnoses the resulting continuity problem: how timestamp-driven seeking, prior-text context, search fallbacks, silence decisions, and word alignment turn a sequence of local predictions into a long transcript. The reader should come away able to distinguish a bad segment from a bad transition between segments and to design validation around continuity rather than a single short-form score.
A long recording is a state machine
Whisper’s fixed context is not automatically a weakness. A bounded window makes the model’s input shape predictable and keeps the attention problem finite. The difficulty appears when the application must represent a recording that extends beyond that boundary. The system must choose which audio to process next, how much text from the preceding window to provide, when a segment ends, and what to do when the generated sequence looks unreliable.
The released transcription loop treats a long file as a sequence of windows over one padded log-Mel representation. It tracks the amount of original content, the current seek position, the clip boundaries to process, and the tokens accumulated so far. At each step it extracts a window, pads it to the model’s expected length, and asks the decoder for a result. The next seek position is not always the end of the 30-second block. Timestamp tokens can indicate that speech ended earlier, or that a segment continues only partway through the window.
That makes timestamps control state. If the model emits paired timestamp tokens, the transcriber interprets them as segment boundaries and advances to the corresponding point. If a final timestamp indicates that the window contains no further speech, it can move to the next boundary. If the output contains an unfinished segment, the loop can seek to the last useful timestamp rather than discarding all remaining audio. If no usable timestamp structure appears, the implementation falls back to the window’s duration. The exact path depends on what the decoder generated, so a timestamp mistake can change the audio seen by every later iteration.
Previous text adds another state channel. By default, the text produced by earlier windows can be supplied as a prompt for the next decode. This can preserve a name, a technical phrase, or a sentence that crosses a boundary. It can also reinforce an error. The paper reports that disabling previous-text conditioning can make text less consistent across windows but can reduce the tendency to enter repetition loops or lose timestamp synchronization. The option is therefore not a quality slider with one universally correct setting. It changes which failure is more likely: local discontinuity or persistent contamination from earlier output.
The loop also resets context under some conditions. When a high temperature has been used after a decoding failure, the implementation can stop carrying prior output as prompt state, reducing the chance that a recovery attempt inherits a bad sequence. An initial prompt supplied by the user can be carried across windows, but only within the available context. The result is a bounded memory system: prior text is useful state, not an unlimited transcript buffer.