Inference Under Constraints: Quality, Latency, and Failure

In GPT-SoVITS, an inference request is a sequence of state changes, not one call from text to sound. The runtime validates a reference and model configuration, prepares text, predicts semantic codes, renders acoustic structure, and assembles the final audio. The useful question is which control belongs to which stage. Once that is clear, a reader can change one variable at a time, understand why some fast modes cannot be combined, and diagnose a weak result without blaming the wrong component.

A practical request might include a reference clip, a target sentence, a language mode, and a handful of sampling and speed settings. The runtime turns those inputs into cached reference features and text batches, then chooses a semantic-generation path and an acoustic-generation path. Some settings alter what is generated; others only alter how efficiently or how smoothly an already chosen sequence is rendered.

One request crosses several owned boundaries

The request enters the TTS runtime with target text, target language, reference audio, optional reference text and language, and controls such as top-k, top-p, temperature, repetition penalty, seed, speed, batch size, and streaming preferences. The runtime owns the mode decision. It first determines whether the caller has requested ordinary, parallel, segmented, or streaming inference and whether the active model has a vocoder path.

The reference path is prepared before target semantic generation. If the supplied reference differs from the cached one, the runtime loads it, resamples it as needed, computes the reference spectrogram, and extracts prompt semantic codes. Auxiliary references are added when requested. Reference text is normalized and transformed into phones and contextual features when present. If the reference is missing and no valid prompt is cached, or if the prompt clip violates the current three-to-ten-second range, the request fails before the GPT stage has a chance to generate anything.

The target text then passes through the frontend. It is segmented, normalized, converted to phones, and represented with contextual features. In ordinary non-streaming mode, the runtime may group similarly sized text segments into buckets and preserve an index map so it can restore the caller’s original order after batched generation. In streaming or segmented-return modes, it prepares text batches differently because output must be yielded before the whole request is complete.

The semantic stage predicts code sequences. In a prompt-conditioned path, the reference-derived semantic prefix gives the GPT model a speech-side context. The target phones and features tell it what content to continue. The model samples one code at a time until EOS or an early-stop limit. It returns both the sequence and the length needed by the acoustic stage to discard any prompt-only portion and decode only the requested continuation.

The acoustic branch depends on the selected model family. Traditional SoVITS decoding consumes semantic codes, target phones, and reference spectrogram conditions to produce a waveform fragment. v3 and v4 instead produce acoustic features for a dedicated vocoder. Long features can be processed in overlapping chunks, and the runtime uses correlation-based overlap handling to splice those chunks. Finally, the runtime concatenates fragments, adds configured intervals, normalizes amplitude, optionally applies super-resolution where supported, converts the result to int16 audio, and returns it or yields it incrementally.