Overview

This chapter traces SIMA’s path from image and instruction to eight-step keyboard-and-mouse actions. Pretrained SPARC and Phenaki representations are adapted alongside language conditioning, cross-attention, Transformer-XL memory, behavioral cloning, and a goal-completion objective. At inference, classifier-free guidance strengthens language influence, while buffering and time-offset prediction address delay. These components support the loop but do not individually guarantee generality.

Learning outcomes

SIMA combines adapted image–text and video representations with language conditioning and temporal memory before choosing actions.
Eight-action chunks and a goal-completion objective support short-horizon control, but neither is a full-game plan.
Classifier-free guidance strengthens learned language influence at inference; it does not add world knowledge or solve ambiguity.

Keywords

SIMApolicySPARCPhenakimemoryguidance

Inside the Agent: From Pixels and Words to Actions

SIMA’s training distribution supplies varied demonstrations, but demonstrations do not act by themselves. This chapter follows the functional path from an image and an instruction to a time-sensitive keyboard-and-mouse action. The goal is to understand what each part of the agent contributes—representation, language conditioning, temporal context, policy, guidance, and timing—and to separate mechanisms learned during training from controls applied only when the agent is running.

A useful way to read the architecture is as a sequence of responsibilities rather than a catalogue of model names. One part must turn images into a representation rich enough for a policy to use. Another must represent the instruction. A temporal mechanism must preserve relevant context while the world changes. A policy must convert the combined state into actions, and the execution system must deliver those actions before the situation has moved on. SIMA combines pretrained components with models trained or adapted for this particular control problem.

sequenceDiagram participant U as User participant E as Environment participant V as Pretrained visual encoders participant L as Language representation participant M as Temporal memory participant P as Policy participant C as Classifier-free guidance participant B as Latency buffer U->>L: Provide instruction E->>V: Provide image observation V->>M: Send visual representation L->>M: Send language representation M->>P: Build temporal state P->>C: Compare conditioned and unconditioned policy C->>B: Produce guided action chunk B->>E: Deliver buffered keyboard-and-mouse actions E-->>V: Return environment response

A layered path from pixels and words to a policy state

At the front of the system are pretrained representations. SIMA incorporates SPARC, a model trained for fine-grained image–text alignment, and Phenaki, a video prediction model. The report says these representations are further adapted using the project’s data: SPARC through behavioral cloning and Phenaki through video prediction. Their role is not to solve a task by themselves. They give the downstream policy a starting representation shaped by broad visual or temporal experience, rather than requiring every useful visual distinction to be learned from the collected game trajectories alone.

That distinction matters because the agent must recognize more than objects in isolation. It may need to relate a phrase to a visible target, understand a change across frames, or preserve a scene feature while the camera moves. An image–text representation can make visual-language correspondences easier to use; a video representation can provide information about how scenes evolve. The report presents the two sources as complementary, but it does not claim that either transfers uniformly to every world or that the high-level description proves each one is necessary in every task.

The language instruction is encoded alongside those visual inputs. The from-scratch transformer components cross-attend to the different pretrained vision representations and the encoded instruction. “Cross-attend” is a compact way of saying that the policy’s internal state can relate information from different streams rather than treating an image and a sentence as independent labels. The instruction can influence which part of the visual scene matters, while the visual context can constrain what a phrase should mean in the current world.

The result is not yet an action. SIMA also uses a Transformer-XL mechanism that attends to past memory states.

Published by DeepMox

FAQ

What do SPARC and Phenaki contribute to SIMA?

SPARC provides fine-grained image–text alignment and Phenaki provides video-prediction representations that SIMA adapts for its control problem. Neither model solves the tasks by itself.

What is classifier-free guidance doing in SIMA?

Classifier-free guidance compares language-conditioned and unconditioned policy outputs and shifts action logits toward their difference. In tested research environments, it strengthens language influence rather than creating it from nothing.

Why does SIMA use temporal memory and eight-action sequences?

SIMA uses temporal memory to preserve context across frames and eight-action sequences to provide short control momentum with frequent visual updates. The design is a small control loop, not a full-game plan.

Previous10 minSIMA’s Multiworld Training Data

Next9 minSIMA’s Evaluation Signals

SIMA’s Multimodal Control Architecture

Inside the Agent: From Pixels and Words to Actions

A layered path from pixels and words to a policy state

FAQ