When Language Has to Work in a World
SIMA, the Scalable, Instructable, Multiworld Agent project, is an attempt to connect free-form language with perception and action across many simulated 3D worlds. The central question is not whether an AI system can describe a scene or produce a plausible answer, but whether an instruction such as “go to the spaceship” can change what the system sees, how it moves, and what it does next. This series uses SIMA to build a concrete judgment: when does a shared language-and-control interface create reusable behavior, and when does it merely make a collection of demonstrations look general?
That distinction matters because language is useful precisely when it compresses experience into abstractions. A person who understands “go downhill,” “pick up the object,” or “use the workbench” does not need a separate vocabulary for every visual scene. But an agent cannot act on an abstraction until it connects words to objects, spatial relations, movement, timing, and consequences. In a changing 3D environment, the meaning of an instruction is partly established by what is visible and partly by what actions make sense there. SIMA treats this grounding problem as the subject of its research rather than attaching language to an already-solved game controller.
The relationship runs in both directions. Language can help an agent organize experience into goals, explore more intelligently, and compose familiar actions into a new instruction. At the same time, experience in rich environments can force language representations to become more systematic: “open,” “move,” or “use” acquire meaning through the objects and situations in which those actions succeed. The claim is not that a virtual world automatically teaches language. It is that language and grounded behavior can provide useful structure for one another, which gives a reason to study them in one shared system.
The missing link between words and action
Large language models have shown that machines can manipulate symbols with remarkable fluency. That fluency does not by itself provide a body, a point of view, or a way to change the situation being discussed. A grounded agent must decide which visual feature matters, infer how the environment responds to a control, and keep acting while the world changes. It must also handle instructions whose completion is not a single obvious button press. “Go to the spaceship” requires finding a target that may not be in view; “use the analysis visor” requires recognizing both an object and an interaction pattern.
A useful way to see the gap is to compare an answer with an action. An answer can be judged by whether its words are coherent or factually appropriate. An action is judged by what it changes. If the agent turns toward the wrong object, opens a menu unnecessarily, or reaches the target after an irrelevant detour, the result may be visibly different even if the original instruction was short and familiar. The system has to bind the instruction to a current scene and maintain that binding long enough for the environment to respond. This is why SIMA's problem is closer to operating a continuously changing interface than to filling in a textual response.
SIMA’s answer is intentionally concrete. The agent receives the same kind of screen observation available to a human player and produces keyboard-and-mouse actions. It is not given the environment’s internal state, a reward signal, or a high-level command API. The project also uses open-ended natural language rather than a small command grammar. These restrictions make the problem harder, but they define a common contract that can be reused across worlds. The agent does not need a separately designed action space for every game, and human behavior can provide direct demonstrations of how language, vision, and control fit together.
The interface is therefore more than a technical convenience. It is SIMA’s first generalization hypothesis: if different environments expose enough common structure through a human-compatible mode of interaction, then skills such as navigation, looking, gathering, or manipulating objects may be learned once and reused. The hypothesis is deliberately limited. A shared keyboard and screen do not erase differences in physics, menus, object affordances, or game-specific knowledge. They create a place where those differences can be encountered by one system instead of being hidden behind custom adapters.
The same choice also makes control a systems problem. Commercial games can continue running while the agent computes, and observations and actions may cross a network boundary before they reach the screen or the keyboard. A visually correct decision can therefore arrive too late. SIMA addresses this in the training and evaluation loop through time-offset action prediction, buffering, accelerator scheduling, and cached state. Those details matter conceptually even before they matter operationally: a general interface is useful only if the policy can act within the timing conditions of the worlds it is meant to share.
SIMA is a multiworld agent, not a game-specific trick
To test that hypothesis, SIMA combines commercial video games with research environments. Commercial games contribute visual richness and open-ended mechanics: a player may explore, mine, build, craft, drive, fly, or manage a menu in worlds that were not designed as tidy laboratory benchmarks. Research environments provide the opposite advantage. They can be controlled more precisely, offer repeatable scenes and tasks, and expose capabilities such as object manipulation or physical reasoning in ways that are easier to measure. The two settings are complementary. One supplies variation that can reveal whether a behavior travels; the other supplies control that can reveal what the behavior actually is.
The contrast is visible in the kinds of problems each setting exposes. A commercial sandbox may require the agent to discover how navigation, tools, menus, and resources interact inside a visually busy world. A research setting such as Construction Lab can instead make the physical relationship itself the point: building a bridge, ramp, or connected structure tests whether the agent can manipulate objects and anticipate what the arrangement permits. The first kind of world broadens the distribution of experience; the second helps isolate a capability that might otherwise be hidden inside a long game sequence.
This portfolio also changes what “general” can mean. It does not mean that SIMA has mastered every possible world. It means that training and testing deliberately cross boundaries that a single environment would conceal. Navigation may recur across games, while crafting rules or vehicle controls may not. A system that succeeds only when the world offers a familiar arrangement has learned a scene prior; a system that can follow different instructions from the same initial situation has stronger evidence of language use. Generality is consequently a relationship between an agent and a distribution of worlds, not a label attached to the agent in isolation.
That distribution has to be populated by more than screenshots. SIMA relies primarily on behavioral cloning from human gameplay. The collected material includes visual experience, recorded actions, natural-language instructions or dialogue, descriptions, and marks of success or failure. Some trajectories are annotated after free play; others come from setter–solver sessions in which one person instructs another while sharing a single-player view. The team filters and reweights these experiences because a large quantity of behavior can still teach the wrong association if the wording, scene, and action are not aligned. The eventual claim about language therefore depends partly on how the experience was collected.
The project’s long-term ambition is correspondingly broad: an instructable agent able to accomplish anything a human can do in any simulated 3D environment. The reported work is an initial step toward that goal, not its completion. The system concentrates mainly on instructions that can be completed in roughly ten seconds. Longer activities can be expressed as sequences of shorter instructions, which makes basic skills reusable, but this decomposition leaves open the harder problems of sustained planning, error recovery, and deciding what to do next without a new command.
From a promising idea to evidence
The useful way to read SIMA is as a connected system. Varied worlds create experiences; human players supply language-linked demonstrations; a multimodal agent turns observations and instructions into actions; and a portfolio of evaluation methods tests whether those actions reflect the command rather than a convenient scene cue. If any link is weak, the generality claim becomes difficult to interpret. A broad dataset with poor language alignment can teach plausible behavior without instruction following. A capable controller tested only in one predictable setting can look strong without demonstrating transfer. A benchmark that accepts any successful-looking action may reward environmental affordances rather than comprehension.
The evaluation problem is not a minor afterthought. Suppose a scene contains a knife, a cutting board, and a carrot. An agent may cut the carrot because the arrangement makes that action obvious, even if it has ignored the instruction. To test language conditionality, evaluators need situations in which several actions are possible, or multiple instructions can be issued from the same initial state. Research environments can sometimes expose ground-truth success directly. Commercial games may instead offer completion text that can be read with optical character recognition, or require expert judges to inspect a recorded trajectory. Each method makes different compromises among speed, cost, objectivity, and coverage.
The rest of this series follows those dependencies. It first examines the interface as a boundary that both enables transfer and imposes low-level control costs. It then turns to the worlds and human data that make the boundary informative, before tracing how SIMA combines visual and language representations with temporal control. Only after that foundation does the question of measurement become meaningful: how ground-truth checks, on-screen signals, and human judgments support different kinds of evidence. The final assessment will separate the transfer SIMA demonstrates from the arbitrary, long-horizon embodied competence it hopes eventually to achieve.
The central lesson at the outset is modest but important. Language does not become an action policy merely because a model has read many words. In SIMA, language has to survive contact with pixels, controls, timing, and consequences. A shared interface may make that contact scalable, but it also exposes every place where generality can fail. Understanding that trade-off begins with the interface itself.