The Generator: Predicting a Market Sequence One Subtoken at a Time

The tokenizer gives Kronos a sequence of coarse and fine symbols, but a vocabulary by itself does not forecast anything. This chapter follows what happens when those symbols enter the decoder-only model: how the two parts are embedded, how the model protects the prediction step from future information, how calendar context enters the sequence, and why the fine prediction depends on the coarse one. The goal is to make a generated forecast legible as a causal sequence of model decisions rather than as an unexplained output from a large neural network.

A history becomes a causal context

Kronos receives two aligned streams of token IDs: one for the first, coarser part of each market token and one for the finer part. The model gives the two streams separate embedding tables. It then combines the resulting vectors into one representation at each time step. This preserves the fact that a market observation has two related components while giving the Transformer a single sequence of hidden states to process.

The distinction between the streams is important at the input as well as the output. If the two IDs were fused into an opaque label before the model saw them, the architecture would have less direct control over their separate roles. Kronos instead keeps the parts identifiable long enough to embed them separately and fuse them into the model dimension. The history remains a sequence of structured tokens even after it enters the shared Transformer backbone.

The backbone uses causal self-attention. At a given position, the model can use the preceding token history and the information available at the current position, but the attention mask prevents it from looking ahead at future positions. This is the condition that turns a general sequence processor into an autoregressive generator. When the model predicts the next token, it must do so from the context that would actually be available at generation time.

That constraint is easy to describe and easy to violate in a time-series experiment. If future observations leak into a representation, normalization window, or attention calculation, a forecast can look better without being usable. Kronos's causal attention addresses the model's internal access to future token positions. It does not, by itself, guarantee that an entire data pipeline is free of leakage; timestamps, train/test construction, and external preprocessing still matter. The architectural guarantee is specific: the Transformer attention calculation is causal.

Kronos also adds explicit temporal information. The implementation extracts minute, hour, weekday, day, and month from the timestamps and maps these fields into temporal embeddings. Those embeddings are summed and added to the token representation before the Transformer blocks process it. A token therefore carries two kinds of context: the learned market-content representation from the tokenizer and a calendar signal describing when the observation occurred.

Calendar features are not external market knowledge. They do not tell the model that a news announcement happened, that a central bank changed policy, or that an exchange experienced an outage. They provide a compact way to represent recurring time coordinates and intraday, weekly, or monthly structure.