Training the Bridge: Why Dataset Alignment Decides the Result

For GPT-SoVITS, fine-tuning is not a matter of dropping a few recordings into a folder and waiting for a new voice to appear. The system must turn each recording and its text into linked representations that Stage 1 and Stage 2 can interpret consistently. This training lifecycle explains what the four-field annotation record becomes, why samples are filtered, what the semantic and acoustic stages learn, and when changing model weights is worth the added data and hardware cost.

The central question is whether a training example still describes one utterance after it has been transformed several times. The text must yield the intended phones. The audio must yield features and a waveform with compatible timing. The semantic target must come from the same clip. If those relationships are weak, more training can teach the model the wrong relationship more efficiently.

An annotation record becomes a lineage of representations

The project’s documented annotation format has four fields: a vocal path, a speaker name, a language label, and the transcript. The record is compact, but it is not the final training input. The recommended workflow slices source audio into usable clips, optionally denoises it, runs automatic speech recognition, lets the user proofread the transcriptions, and then prepares the model-specific features. Each step adds or changes evidence that later stages will consume.

The first transformation is text-side preparation. The annotation’s language label guides cleaning and phone conversion. The resulting record keeps the clip name, phone sequence, phone-to-text alignment information, and normalized text. For Chinese records, the preparation script also computes phone-level contextual features from a BERT model and stores them separately. This makes the text more than a transcript: it becomes the exact pronunciation and feature representation that the semantic model can condition on.

The second transformation is speech-side preparation. The project creates a 16 kHz signal for CN-HuBERT feature extraction and retains a 32 kHz waveform for the conventional acoustic path. CN-HuBERT produces a 768-channel SSL feature sequence. The feature extractor checks signal conditions and rejects NaN results; usable features and waveforms are persisted under the experiment’s data area. The semantic-preparation step then loads a compatible pretrained SoVITS model, extracts latent codes from the SSL features, and serializes those codes as semantic targets.

The result is a lineage rather than a list of independent files. A clip name links cleaned phones and normalized text to its BERT features, SSL features, waveform, and semantic sequence. Stage 1 needs the phone and semantic sides, with contextual features where available. Stage 2 needs the text, SSL features, spectrogram or waveform, and any extra speaker representation required by the selected version. The loaders intersect these sets rather than assuming that a file with a similar name is good enough.

Consider a short recording whose ASR transcript has one missing word. Its waveform and SSL feature can still be perfectly readable. Its semantic codes can still be extracted without error. But the text-side record now describes a different utterance from the speech-side record. Stage 1 may learn a false relationship between the phones and the code sequence, while Stage 2 sees a renderer trained on inconsistent text and audio. Proofreading is therefore part of the model’s data contract, not only a way to make labels look tidy.