Teaching the Model Which Calls Earn Their Place
Toolformer does not learn from every API request it can generate. This chapter focuses on the selection mechanism that turns a possible call into training evidence: compare the model's prediction of the future with and without the returned result, then retain the call only when the result makes a meaningful difference. The goal is to reconstruct the full path from proposal to inference and to see why predictive improvement is useful evidence without being a guarantee of truth, safety, or value to a user.
A result earns its place by changing the prediction
The central problem is easy to state. A model can produce a well-formed request that runs successfully and still gain nothing from the answer. It might ask for a fact already stated in the passage. It might send an irrelevant search term to a retrieval system. It might calculate a number that the next sentence never uses. If Toolformer trained on all such calls, it would learn that invoking an API is valuable simply because an API invocation appears in the data. The method needs a test tied to the model's actual job: predicting what comes next.
That test uses cross-entropy loss. In plain language, the loss is a penalty that becomes larger when the model assigns low probability to the actual continuation tokens. A lower loss means the model predicted the observed continuation more successfully. For a candidate call at position i, the paper measures the loss over the tokens that follow the call. One version, written Lᵢ⁺, gives the model both the API request and the result. The comparison version, Lᵢ⁻, is the lower of two losses: the loss when there is no call at all, and the loss when the model sees the call but receives no result.
The choice of the lower baseline matters. Toolformer does not ask whether a result beats only the no-call situation. It asks whether the result beats the better of two alternatives already available to the model. Perhaps the model can predict the continuation without any external help. Perhaps the presence of the request itself provides a useful structural cue, even before a response arrives. In either case, the returned information must do more than make the sequence look like a tool interaction. It must improve prediction beyond the strongest relevant baseline.
The acceptance rule is expressed as Lᵢ⁻ − Lᵢ⁺ ≥ τf, where τf is a filtering threshold. The difference is a gain in predictive performance: if the loss with the result is sufficiently smaller than the chosen baseline loss, the call survives. If it is not, the candidate is removed from the augmented training data. A high-quality request can therefore fail to earn a place if its answer does not matter to this continuation, while a narrow answer can pass if it supplies exactly the missing information the next words require.
Imagine, without assigning any particular numerical values, that the model predicts a sentence almost equally well with no call and with a call that has no result. If inserting the returned fact makes the next words much easier to predict, the difference between the baseline and the with-result loss is large enough to matter. If the result leaves the prediction unchanged, the difference is small and the call is discarded. If the result makes the continuation harder, the candidate is not rescued merely because the request was grammatical or the API answered successfully. The test asks what changed in the model's prediction, not whether the exchange looked complete.
The threshold controls the strength-versus-coverage trade-off. A stricter threshold keeps fewer calls, each required to produce a larger measured improvement. A looser threshold admits more marginal examples, increasing the chance of useful coverage while also admitting more noise. Toolformer's threshold is therefore part of the training-data design, not a universal definition of usefulness. It determines how much evidence a candidate must provide before it influences the model.
The paper weights tokens by their distance from the call, with weights that decrease as the continuation moves farther away. This makes the signal more sensitive to information that helps soon after it is provided. The choice matches the intended relationship between a call and its result: a retrieved fact or calculated number should usually affect nearby text. Without distance weighting, a call might receive credit for a broad change in loss far downstream that is difficult to attribute to that particular response. The weighting does not prove causation; it is a way to make the measured relationship more local and therefore more useful for filtering.
The filter is easier to understand through the paper's examples. A question-answering call asking for the approximate length of the Nile is useful when the surrounding sentence is about that length, because the returned value helps predict the sentence's next wording. A calculator call that computes a ratio can be useful when the continuation states the same ratio. By contrast, a search result about a “fast train” can be topically related to a page but fail to provide information needed by the next sentence. The score generally tracks this difference, although the paper also reports exceptions.
Those exceptions reveal what the test can and cannot see. A result may reduce language-modeling loss because it resembles the continuation without being independently correct. A response can be factually valid but irrelevant to the user's real goal. A call can survive with noise that does not look intuitively useful, and the authors suggest that some noise may even discourage the fine-tuned model from blindly trusting every returned answer. The filter is a predictive criterion, not a truth checker or a policy for deciding whether a real service call is authorized.
The same limitation applies to operational cost. Candidate execution consumes resources, and different APIs can have different latency or monetary expense. Toolformer's reported selection rule does not include those costs. It retains a call when the response helps predict the continuation, even if another design might prefer a cheaper or faster way to obtain the same information. The paper also notes that useful calculator examples are sparse enough that heuristics are needed when processing a very large corpus. Predictive usefulness is the teaching signal, but data-generation efficiency remains an engineering constraint.
From accepted call to live continuation
Once the filter has selected calls, Toolformer merges calls for different APIs and interleaves them with the original text. The resulting corpus is called C*, while the plain corpus is C. Apart from the inserted API exchanges, the text in C* remains the same. The model is then fine-tuned with the ordinary language-modeling objective. It is not trained with a separate label saying “call now”; it learns from sequences in which a request and its result occur at a position where the result helped predict the original continuation.
This construction teaches several behaviors at once. The model must learn the circumstances in which an API-opening marker can occur. It must produce an API name that matches a usable interface and an input that the service can execute. It must also learn the local syntax around the returned result and the transition back to ordinary prose. These behaviors are coupled in the training sequence, but they are not the same behavior. A model can learn to emit the marker while still producing a poor argument, or learn to copy a result while failing to recognize when a call is unnecessary.
The order of the data-building stages matters. The response is obtained before fine-tuning, so the model can be trained on a completed interaction rather than being asked to invent an answer for a tool it has not yet learned to invoke. The usefulness test is also applied before the calls are mixed into the corpus. Otherwise, a large volume of irrelevant requests could teach the model a strong but undesirable prior to call APIs whenever a prompt makes the syntax available. Filtering limits which interactions become part of the model's learned distribution.
Keeping the original text is important for the paper's claim about generality. The augmented examples do not replace ordinary language with a separate tool-use dataset of unrelated tasks. The target remains the same continuation that appeared in the language-modeling corpus. Tool calls add a route to information; they do not remove the model's responsibility to generate the surrounding words. This is why fine-tuning on C* can teach new behavior while preserving the content distribution that the base model was meant to model, at least under the paper's later controls.
That preservation has a precise meaning. It says that the model is exposed to the same underlying passages plus inserted interactions; it does not say that the augmented corpus has the same statistical distribution in every respect. Calls add tokens, alter local context, and concentrate examples around positions that passed the filter. The authors test the practical consequence later by disabling calls at inference and measuring ordinary language-modeling perplexity. The construction makes preservation plausible, while the control experiment determines how well it holds in the reported setup.
The lifecycle changes when the model is used rather than trained. During inference, Toolformer decodes ordinary text until it emits the marker that indicates a result should follow. Generation is then interrupted. The relevant API receives the request, produces a response, and the response plus the closing marker are inserted into the sequence. Decoding resumes with the returned text in the model's context. The model is not asked to hallucinate the calculator's number or the calendar's date as ordinary continuation tokens; the external service owns that operation.
The boundary makes ownership and triggers explicit. The model owns when to begin a call and what input to pass. The API owns the transformation from that input to a response. The serving loop owns the pause, the external request, and the insertion of the result before generation resumes. If the model never emits the call marker, the normal continuation proceeds. If it emits the marker but the request is invalid or the service fails, the expected response is unavailable; the paper does not turn such failure into a reliable fallback. If a valid response is returned, the model still has to use it correctly in the remaining text.
This means that the call marker is a control signal with a concrete side effect. Before it appears, the system is doing ordinary decoding. At the marker, the next step is no longer another model-generated word: the external service must be contacted. After the result is inserted, the model again controls the text. The transition is useful because it makes the specialist's operation explicit, but it also introduces a dependency on service availability and response format. A language model that has learned the syntax can still be interrupted by an unavailable or unusable tool.
The paper limits inference to at most one API call per input. That restriction prevents a loop in which the model continually emits new requests instead of producing output, but it also ends the lifecycle before dependent tool use can begin. A date could be obtained from a calendar, for example, but the same example cannot then automatically pass that date into a question-answering request. The training data does not contain the necessary chain because calls for the different tools are generated independently. The limit is thus both a safety valve against runaway calling and a hard ceiling on the workflows the model can learn in this experiment.
The one-call boundary also changes how to interpret a successful result. Toolformer is not demonstrating a general planner that decomposes a task, stores intermediate state, and chooses the next tool based on the previous response. It is demonstrating a learned transition from a text prefix to one external exchange and back to text. That smaller capability can still be valuable, especially when one narrow operation is the missing ingredient, but it should not be described as multi-step reasoning merely because an API was involved.
The state diagram shows two related but distinct paths. Offline, a candidate is executed and either rejected or carried into the augmented corpus before fine-tuning. Online, the fine-tuned model enters an inference state, pauses when it requests a result, and resumes after the API response is inserted. The one-call ending is not a statement that all useful tasks require only one request; it is a boundary imposed by this implementation and its training examples.
This distinction also explains why “self-supervised” should be used carefully. The model supplies candidate requests and the continuation supplies a measurable signal, but people still choose the tools, define their interfaces, provide demonstrations, and set thresholds. The filter reduces reliance on manual labels; it does not remove the design of the environment or verify the world outside the text. Toolformer learns a calling pattern from predictive feedback, not an all-purpose standard for deciding whether any external action ought to happen.
The result is a compact causal story. A possible request becomes a candidate because the model can express it at a likely position. It becomes training evidence only if the returned text lowers future-token loss beyond the best baseline by enough. Fine-tuning exposes the model to those retained sequences. At inference, an emitted request triggers a real API call, and the response becomes context for the continuation. The next question is what this shared process looks like across different specialists: whether factual lookup, arithmetic, retrieval, translation, and temporal context produce recognizably different kinds of help.
References