Inside the Model: Two Kinds of Prediction, One Song

YuE2 combines autoregressive and non-autoregressive computation because its job changes as a request moves from composition to sound. This chapter explains that division of labor: why causal prediction fits the score and semantic sequence, why the acoustic path predicts a flow-matching velocity over latent positions instead of another list of audio tokens, and how both paths use one transformer backbone without becoming the same operation. The result is a practical way to tell architectural control from runtime machinery.

The distinction matters for the same reason the score matters. A component list—transformer, latents, VAE—does not explain who owns a decision or what crosses an interface. YuE2's implementation is more intelligible when read as a sequence of responsibilities: an AR path orders discrete conditions, a routed backbone handles different token types, an NAR path refines a continuous acoustic state, and a VAE turns that state into a listening signal. Each boundary preserves some information and leaves other properties open.

The shared backbone has two operating modes

Autoregressive means predicting the next item while respecting an order: earlier positions can condition later positions, but later positions cannot look ahead. This is a natural fit for YuE2's score and semantic stages. A composition is not just a bag of notes; its tokens occur in an order that carries lyrics, sections, musical events, and termination. Semantic music tokens also form a sequence that the model must extend from the request and the exact score condition. The AR path gives the generator a causal rule for deciding what comes next.

The released YuE2 model exposes that path through a causal language-model interface with key-value caching. A KV cache stores the attention state for the prefix already processed, so the system does not need to recompute the entire history every time it adds another token. The cache is an efficiency mechanism, but it also makes the AR path's state ownership clear: the path owns an ordered discrete history and extends it one position at a time.

The score enters this path as a real protocol boundary. YuE2 serializes the instruction, tags, lyrics, score delimiters, and music-start marker into a model prefix. In full and melody modes, the encoded ABC remains part of the exact prefix used for semantic generation. The pipeline checks that a plan's prefix agrees with the request and the exact score IDs before proceeding. The AR path is therefore not starting from “some music prompt” in the abstract; it is extending a particular, validated condition.

The acoustic problem is different. YuE2 does not need to choose one text-like symbol after another to describe a waveform. It starts with a continuous latent state and needs to move that state toward an acoustic realization compatible with the preceding conditions. Its non-autoregressive, or NAR, path operates on latent positions together rather than emitting one discrete audio token as the next item in a causal chain.

The model's name, AR–NAR Mixture-of-Transformers, describes how those responsibilities meet.