ReAct in Motion: From Subgoals to Long-Horizon Decisions

ReAct becomes a different kind of system when the task is no longer a short chain of searches. In ALFWorld and WebShop, the model must preserve a goal across many actions, interpret changing observations, and decide whether a partial success is actually enough. This chapter follows that transfer from a household task to a noisy product-search environment, showing why sparse thoughts can guide long-horizon behavior and why merely repeating environmental feedback is not the same as reasoning about it.

The practical question is whether the loop can keep a task organized as the state changes. A single answer can sometimes be corrected by one more search. A long task can fail because the model loses a subgoal, acts before reaching the right location, or selects an item that matches one attribute while violating another. ReAct's value in these settings depends on whether its language traces preserve the structure of the goal without turning every routine action into unnecessary narration.

A household goal is a chain of dependent subgoals

ALFWorld is a synthetic, text-based environment aligned with household tasks. An agent may be asked to examine an object under a lamp, place a clean item on a surface, or complete another high-level instruction by navigating a room and executing text actions. A task instance can contain more than 50 locations and can take an expert policy more than 50 steps. The difficulty is not just the number of commands. The agent must search systematically, use commonsense expectations about where objects are likely to be, and remember which parts of the goal have already been completed.

A command-by-command policy can lose that structure. It may visit a plausible location, find an unrelated object, and continue to another location without recording what the observation ruled out. It may pick up the right item but attempt to clean it from a different location because it has not represented where the item now is. Or it may complete a subgoal and then repeat the same action because nothing in the trajectory marks the transition to the next one. The action is locally plausible, but the sequence no longer serves the high-level goal.

ReAct demonstrations address this by placing thoughts at points where the task state needs interpretation. At the beginning, a thought can decompose the goal into a sequence such as find the object, take it, clean it, and place it. A second thought can use commonsense to prioritize likely locations instead of searching every location at random. After an observation reveals the object, another thought can mark the search subgoal complete and select the next action. Later, the thought can record that cleaning has happened and redirect the agent to the destination. These statements do not change the room; they preserve the state of the plan while actions change the room.

That distinction is visible in the paper's matched examples. In an action-only trajectory, the model can find a knife but then try to clean it while it is still at a countertop rather than at the sink. The environment returns “nothing happens,” yet the model continues moving among counters and repeating the invalid clean action.