From Lyrics to Sound: The Four-Stage Lifecycle

YuE2 does not move directly from a score to a waveform. Its public pipeline separates planning, semantic generation, acoustic synthesis, and decoding, and each stage hands a different representation to the next one. This chapter follows that lifecycle closely so a reader can tell what changes at each boundary, what persists, and which artifacts are needed to compare two results without confusing a new generation with a decoder experiment.

The score-centered design only becomes operational when the request's identity survives those transitions. A plan that is changed silently, a latent decoded by a different model, or an output that reached a token limit can all look like ordinary generation if the record is incomplete. The useful question is therefore not “what file did the system produce?” but “which state did it reach, and what evidence connects that state to the request?”

Four stages, four responsibilities

The public Python interface names the lifecycle explicitly: plan() → generate_semantic() → synthesize() → decode(). A one-call generation runs these stages in sequence, but the separation matters because each stage solves a different problem.

Planning creates or accepts the composition. In a new full or melody-mode request, YuE2 uses the style and lyrics to generate an ABC plan. If the user supplies an accepted ABC score, the system uses that score instead of inventing a new one. The result is a symbolic plan: human-readable score text when applicable, encoded ABC IDs, an exact prefix for the next stage, timing information, and a truncation flag. In off mode, there is no generated score; the request proceeds with the non-symbolic prefix appropriate to that mode.

Planning is therefore a state transition with a clear owner. Before it, the system has a request expressed through lyrics, style, and a mode, plus an optional external score. After it, the request has either acquired a symbolic composition or explicitly chosen not to. That distinction matters when a result is revisited later. A generated plan is something to inspect and potentially copy for editing. A supplied plan is an external condition whose provenance must be retained. An off-mode request has no score to recover if the audio result later suggests that the melody or harmony should have been controlled.

The planning result also has an important failure boundary. A score that reaches a token limit may still be saved as text and may look plausible in a quick glance, but its truncation state changes what can be claimed about the composition. A supplied score can avoid model planning, but it does not avoid validation: the text still has to fit the accepted symbolic interface, and the exact encoded representation still has to agree with the recorded plan. Planning is complete only when the system's record says what kind of plan it has and whether that plan is complete enough to pass onward.

This stage owns the question, “What composition is being proposed or supplied?” It does not yet answer what the song will sound like. A plan can be inspected, saved, compared, or revised before any semantic music tokens or acoustic latents are produced. That makes planning the point at which a person or agent can intervene most directly.

Semantic generation turns the plan into a sequence for the song generator. The semantic result retains the symbolic plan and adds semantic music tokens. These tokens are not the ABC score rewritten in another human-readable notation. They are a model-facing representation that carries the request toward musical realization. The exact score-derived prefix is checked before generation, so the semantic stage is tied to the plan that produced it rather than to an untracked copy of the original prompt.

This stage owns a different question: “Given this request and this composition, what musical sequence should condition the acoustic realization?” The answer is still not audio. Treating semantic tokens as sound would collapse a meaningful boundary and make it impossible to explain why YuE2 retains both a readable plan and a separate semantic result.

The distinction is useful when comparing requests. Two runs can share the same score but differ in style, lyrics, sampling settings, or seed; the semantic result records a new model-facing response to those conditions. Conversely, a run can share lyrics and style but use a changed score; its semantic sequence belongs to a different symbolic condition even if the prose prompt is unchanged. The readable plan and the semantic result must therefore be stored together when the question concerns what the generator was asked to realize. Keeping only the audio removes the evidence needed to distinguish those cases.

Acoustic synthesis produces continuous latents. After semantic generation, YuE2 creates an acoustic latent trajectory from noise. The released implementation represents each latent frame with 64 values, partitions a long song into context-bounded chunks, and solves a midpoint ordinary differential equation for each chunk. The implementation's default reference solve uses 32 steps. Each chunk first performs an autoregressive prefix prefill, caches the visible prefix state, and then applies the non-autoregressive flow-matching computation to the latent frames.

This stage owns the question, “What continuous acoustic material could realize the semantic sequence?” Its output is a latent array, not playable audio. The chunking and cache are important for context and memory, but they are not musical edits. A smaller computational unit does not mean that the user has selected a smaller musical phrase for independent revision; it means the runtime is managing a long acoustic computation within its context and memory limits.

The chunk boundary also explains why an implementation detail should not be mistaken for a creative boundary. The synthesis routine draws the complete noise tensor once, then takes the portions needed for historical context cuts. It performs the original chunks serially, pre-filling the autoregressive prefix once for each chunk and solving the latent trajectory with the non-autoregressive path. That arrangement limits the amount of active context and temporary memory, while retaining the allowed prefix information for the acoustic calculation. It does not grant a user a local inpainting command, nor does it establish that every boundary is perceptually invisible. The repository supports the inference mechanism and its default protocol; the audible effect of other chunk or hardware conditions would require separate measurement.

This stage therefore changes the kind of uncertainty in the system. The score and semantic sequence state what the generator is being conditioned on. The latent trajectory is where that condition becomes continuous acoustic material, with noise, context, and numerical solving involved. Saving the latent result makes later decoder comparisons fair, but it does not make the latent itself a human-readable musical plan.

Decoding turns latents into a listening representation. A separate stereo VAE decodes the acoustic latents into 48 kHz audio. The default YuE2-Vae is documented for generation and listening, while YuE2-Vae-legacy is the decoder associated with the reported benchmark protocol. The pipeline can decode the same saved latents with either decoder, and it can use tiled decoding with a core region and halo to bound memory. The decoder therefore owns the final conversion from latent representation to waveform, but it does not recreate the earlier planning or acoustic-synthesis stages.

That distinction gives the lifecycle an evidence boundary. A result can be recorded and playable while truncation flags still show that the requested sequence may not have completed; for a claim of a complete generation, the relevant flags must be clear. A decoder-only comparison starts from existing latents and ends with new audio plus a different decoder identity. It should not be reported as a second musical generation, because no new score or acoustic trajectory was created.

The sequence makes ownership visible. The requester supplies the initial condition and later decides whether a plan should be edited or the same latents should be decoded again. The planner owns the symbolic state; semantic generation owns the discrete model-facing sequence; acoustic synthesis owns the continuous latent trajectory; the VAE owns waveform decoding. The artifact store connects those states so a later result can be identified rather than merely remembered.

What survives each handoff

The representation changes, but YuE2 deliberately preserves enough lineage to make the changes inspectable. A saved plan contains the human-readable score when one exists, the encoded ABC token array, the prefix array, the request, timing data, truncation status, and a manifest of file hashes. Loading the plan checks that the saved files have not changed and that the arrays agree with the recorded plan. It also checks that the score text and encoded form remain consistent.

This is more than defensive bookkeeping. It protects the meaning of an unchanged plan. If a user loads the saved plan without altering it, the runtime can verify that the exact symbolic condition is still present. If the user wants to edit the composition, the documented path is to copy the ABC, change the copy, and submit it as a new input. Mutating the saved plan in place would destroy the baseline needed to say what changed.

A complete result extends that lineage. YuE2 can retain the score and plan artifacts, audio, semantic tokens, acoustic latents, the original request, effective generation configuration, model identities, timings, and a result record containing hashes and truncation flags. In practical terms, this creates several kinds of comparison that should not be conflated:

The exact prefix check is the bridge between the first two rows. In full and melody modes, the encoded ABC is part of the prefix used for semantic generation. Before the pipeline proceeds, it checks that the plan's prefix agrees with the request and exact score IDs. If an artifact carries a score but its prefix has been changed, or if a semantic result no longer belongs to its plan, the runtime rejects the mismatch instead of quietly labeling it as the original composition.

The manifest serves a related purpose at the storage boundary. A readable score.abc file could be replaced while the surrounding directory remains intact; a token array could be changed without updating a JSON description. Hashes and cross-checks do not judge musical quality, but they can expose that the saved representation is no longer the representation that produced the recorded result. That is the minimum integrity needed for a meaningful before-and-after claim.

The latent stage creates another important separation. Acoustic latents are reusable for decoding, but they are not a shortcut for implementing a musical edit. If the user changes the score, lyrics, style, or form, the model must generate new semantic and acoustic material. If the user changes only the decoder, reusing the same latents is the fair comparison because the underlying generated composition and acoustic trajectory stay fixed. The generation guide makes this distinction explicit by recommending separate directories for listening and benchmark decoding.

The final trap is completeness. YuE2 records truncation flags for score and semantic generation, and its documentation warns that a playable audio file can exist even when a token limit was reached. A file that opens proves that some audio was decoded; it does not prove that the requested composition completed. Similarly, a successful process does not prove that the song is musically good or that the audio follows every symbolic instruction. The result record must be read alongside the score, model identities, and actual audio assessment.

This is where lifecycle thinking becomes practical. At the planning boundary, inspect the score and its provenance. At the semantic boundary, confirm that the exact plan condition remains attached. At the acoustic boundary, record the latent result and the computational settings that produced it. At the decoding boundary, keep listening and benchmark decoders distinct and reuse latents for decoder comparisons. At every boundary, ask what is preserved and what is newly inferred.

The score-centered interface does not remove uncertainty; it locates it. A plan can be exact while a performance remains variable. A latent can be reused while decoder output changes. A result can be playable while still truncated. Once those distinctions are visible, YuE2's next architectural question becomes clearer: why does one model use autoregressive computation for some stages and non-autoregressive flow matching for another, and how do those roles cooperate to turn the preserved condition into sound?

References

• Generation guide
• Pipeline implementation
• YuE2 repository README
• Acoustic synthesis implementation
• YuE2 model implementation
• Generation protocol
• VAE implementation
• Listening and evaluation reference
• YuE2 music skill
• YuE2 tokenizer