The Hard Part Is the Interface
For SIMA, the human-compatible interface is the first serious test of whether skills can travel between worlds. This chapter follows what the agent is allowed to observe, what it must infer, and how it turns an instruction into keyboard-and-mouse control. The practical judgment is simple but demanding: a shared interface can expose reusable structure across games, yet it also leaves the agent responsible for timing, interpretation, and recovery that a privileged game API would normally hide.
What the agent is allowed to see and do
SIMA receives two kinds of input: a language instruction and an image observation. It produces keyboard-and-mouse actions. That description sounds ordinary because it is meant to resemble human play. The important detail is what the system does not receive. It is not handed the internal state of the game, a reward signal, or a high-level action such as “walk to the ship” or “select the correct resource.” It must work from the rendered screen and the same low-level controls a person would use.
This creates a boundary between the environment and the policy. The environment owns the underlying world: object positions, hidden variables, physics, menus, and the consequences of an input. SIMA owns only its current interpretation and its next action. A new image crosses the boundary toward the agent; an input event crosses back. The agent cannot directly ask the world whether a target exists, whether a tool is equipped, or whether an action has completed unless the answer becomes visible in the screen or in the subsequent change of the scene.
That arrangement forces several inferences into one control loop. If the instruction is “go to the spaceship,” the agent must connect the noun to a visual target or to a route it has learned to recognize. It must decide whether to turn, move, or look around. If the target is not visible, it must use the surrounding scene and the history of its movement rather than a symbolic location supplied by the environment. The instruction remains linguistic, but its consequences are spatial and temporal.
The same is true for apparently simple commands. “Pick up the object” does not specify which object unless the scene or the preceding context supplies that distinction. “Use the workbench” may require approaching the right structure, orienting the view, and applying the interaction control at the appropriate moment. The model therefore has to combine semantic interpretation with visual selection and motor precision. A high-level API could collapse those steps into a single action; SIMA keeps them visible because the project wants one interface that can be reused across environments.
The choice also rules out an easy form of specialization. A game-specific controller can take advantage of exact object coordinates, legal action lists, reward shaping, or a purpose-built navigation primitive. Such a controller may be efficient within its world, but its assumptions do not automatically survive a change of game. SIMA gives up some of that convenience in exchange for a stable contract: images and language come in, human-like input goes out. The contract is not enough to make worlds identical. It is a common surface on which similarities and differences can be learned.
That common surface supports direct imitation of human behavior. People already solve the perception-to-control problem through the same display and controls, so their play can provide examples of how an instruction becomes movement or interaction. The demonstration is not just a record of a final state. It links a view of the world, an instruction, and a sequence of inputs under the timing conditions of play. Behavioral cloning can use that link without requiring a separate action vocabulary for every environment.
The interface also makes failure legible in a useful way. If a policy has access to an internal goal state, it can appear competent while bypassing the visual problem. If it receives a reward designed for one game, it can optimize that reward without being able to follow an arbitrary command. By restricting the agent to what a human player sees and does, SIMA makes the desired capability harder to fake. The cost is that errors can arise at every boundary: language may be misread, the relevant object may be missed, a control may be mistimed, or the world may change before the next decision arrives.
The boundary is especially important when the screen is ambiguous. A rendered image can show several objects that fit the instruction, or show none of them because the target is outside the current view. The environment does not hand SIMA a symbolic explanation of that ambiguity. The policy must use the available visual evidence and the changes produced by earlier inputs to decide what to do next. If it chooses the wrong target, the resulting image may still look perfectly ordinary; the failure is a mismatch between language and control, not necessarily a visibly broken game state.
This also means that the action channel is part of the meaning of a command. “Look at the planet” and “go to the planet” may begin with similar camera movement, but they require different stopping conditions and different relationships between the view and the world. “Pick up” is not complete when the cursor moves toward an object; it is complete only when the interaction has produced the intended change. Because SIMA acts through the same controls as a person, the agent has to learn these distinctions from demonstrations and consequences rather than receiving them as a formal list of outcomes.
A human-compatible interface thus exposes a hidden coupling between interpretation and execution. Better language understanding is useful only if it changes the selected visual target or action sequence. Better visual recognition is useful only if the policy knows which recognized object the instruction refers to. Better motor control is useful only if it is applied at the right time. The interface does not separate these responsibilities into independent modules with guaranteed contracts; it forces them to meet in the loop that changes the world.
Shared controls, moving worlds, and short horizons
A generic interface can support transfer because not every useful behavior depends on a world’s unique rules. Three-dimensional environments often share basic patterns: the agent must orient a view, move through space, approach an object, and use a control to interact with it. The visual details and consequences can differ, but the relationship between an instruction and a family of actions may recur. Learning to navigate toward a visible or remembered target is not the same as learning the exact map of one game. Learning that an object can be selected or manipulated may carry farther than learning the recipe for one specific tool.
This is the strongest version of the interface argument. SIMA does not need every environment to share all mechanics. It needs enough recurring structure for a policy trained on several worlds to build abstractions that are useful in another. The shared keyboard-and-mouse convention helps because the motor vocabulary does not have to be translated at the boundary. A left-click-like interaction, camera movement, or directional input can express related intentions in more than one setting, even though the object affected and the resulting state may differ.
But transfer is conditional. A control that moves a character in one world may steer a vehicle in another. A menu may respond differently from a physical object. The same visual pattern may have different affordances, and a skill that seems generic may depend on a hidden convention of the training environments. A shared interface makes these questions measurable; it does not answer them in advance. The agent still has to learn which aspects of an instruction are stable and which must be inferred anew from the current world.
The environment is also asynchronous. Many commercial games continue to run while an agent computes its next action. The observation is already a little old by the time the policy has processed it, and an action can arrive after the world has moved on. Network transmission between a rendered game and an agent client adds another source of delay. In a turn-based setting, the system could stop the world and ask for a decision. In SIMA’s setting, the world owns the clock.
That changes the meaning of a correct prediction. Suppose a policy sees an object in the right position and predicts the control that would interact with it. If the agent waits too long while computing, the view may have shifted, the character may have moved, or the interaction window may have closed. The action is semantically appropriate but operationally wrong. Timing is therefore part of the action’s meaning, not a layer that can be bolted on after the policy has learned what to do.
SIMA addresses this by aligning training and execution with the delay. During behavioral cloning, the agent predicts actions offset in time from the visual input. During evaluation, observations and actions are buffered so that the system reproduces the same timing assumption. The report also describes scheduling inference on accelerators, caching neural-network state across timesteps, and choosing batch sizes with latency in mind. These measures do not make the world synchronous; they help the policy learn the world it will actually encounter.
The same reasoning explains the project’s short-horizon focus. The visual input and action spaces are high-dimensional, and a natural-language instruction can describe a task that unfolds over much longer than one control cycle. SIMA therefore concentrates mainly on tasks that can be completed in approximately ten seconds. A user can, in principle, express a longer activity as a sequence of smaller instructions: move to the station, open the menu, select the material, and begin the next operation. Short skills then become reusable building blocks across settings.
That decomposition is useful, but it has a precise boundary. It does not demonstrate that SIMA can decide the whole sequence on its own, maintain a long-term objective, or recover when an intermediate step fails. A human or another planning system may provide the next instruction. The policy’s demonstrated responsibility is narrower: given a current view and a command, produce the next useful behavior over a short interval. Treating that as a limitation does not diminish the interface experiment. It prevents a local control result from being mistaken for open-ended autonomy.
The decomposition also changes how errors accumulate. If a short instruction ends in the wrong place, the next instruction begins from an altered state. A reliable long-horizon system would need to notice that the expected transition did not occur, interpret the new scene, and decide whether to retry, revise the plan, or ask for help. The report does not claim that the short-horizon policy supplies this entire supervisory loop. Its approach makes the unit of learning manageable, while leaving composition and recovery as explicit questions for future work.
The human-compatible interface can also be a liability. Low-level input is expressive enough to cover many tasks, but it is less efficient and less explicit than a purpose-built API. The agent has to spend capacity on looking, positioning, menu navigation, and timing. A new world may introduce unfamiliar physics or controls even when the screen-and-keyboard contract remains unchanged. The interface can support transfer only when the training distribution teaches the policy which patterns are shared and when the evaluation distinguishes a truly instruction-driven action from a plausible default behavior.
That is why “generic” should not be read as “frictionless.” SIMA’s interface is general in the architectural sense that the same input and output form can be carried to multiple worlds. It is demanding in the behavioral sense that the policy must ground language through the visible scene and act under the world’s clock. The project chooses this tension deliberately. A specialized interface could make one environment easier; a common interface creates a harder but more informative test of whether a skill has become portable.
The next question is where the variation needed for that test comes from. A shared contract alone cannot teach a policy which visual patterns, controls, and consequences recur across worlds. SIMA therefore needs a training distribution broad enough to expose common structure, but curated enough that language remains linked to the behavior demonstrated. The interface defines the boundary; the worlds and human demonstrations fill it with experience.