A Tool Call Is Still Text
Toolformer solves a very specific interface problem: how can one language model request help from many different APIs without giving up the text-generation machinery it already has? This chapter focuses on the representation and proposal stage. By the end, the reader should be able to read a Toolformer call as part of a passage, explain why text makes unlike tools look uniform, and distinguish a possible call generated by the model from one that has been shown to help.
The model learns a common language for tools
A tool call has three conceptual parts: the name of the API, the input passed to it, and the result returned. Toolformer writes the call as a sequence of tokens inside the text it is generating. In the paper's notation, a call without a result can be represented as <API> ac (ic) </API>, while the completed form is <API> ac (ic) → r </API>. Here, ac is the API name, ic is its input, and r is the response. The markers tell the model where the request begins, where the result appears, and where the inserted exchange ends.
The notation is less important than the design decision behind it. The model does not need one special neural architecture for a calculator, another for search, and a third for translation. It sees each request as a predictable extension of the current token sequence. A sentence can lead to a question for a fact-retrieval system; a numerical expression can lead to a calculator request; a phrase in another language can lead to a translation request. After the result is inserted, the model returns to the same job it had before: predicting the next token.
The paper's practical implementation uses token sequences such as square brackets and -> rather than requiring new vocabulary entries for the conceptual markers. That choice keeps the existing language model usable without modifying its vocabulary. It does not turn the APIs into interchangeable services. The calculator still understands expressions, the search system still expects a search term, and the translation system still expects text to translate. The common layer is the outer syntax, not the meaning of every input.
For the language model, the result is part of the prefix it can condition on. Before a call, the model predicts whether the next span should remain ordinary prose or begin a marked request. Inside the request, it predicts the API name and its input. Once a serving process inserts the external result, the model can predict the next ordinary words with that result available as context. The call is therefore a controlled detour inside a single sequence, not a separate conversation that requires a new model to interpret it.
That detour also preserves the position of the original text. The response is interleaved with the passage rather than replacing the words that came after it. If a retrieved fact is useful for a sentence, the relevant continuation remains the target the model must predict. This matters for learning: the system is not rewarded merely for producing a plausible API exchange; it is evaluated in relation to the text that the exchange is supposed to help complete.
This distinction can be made concrete with a factual sentence. Suppose a passage says that Pittsburgh is also known as a particular nickname, but the model needs help retrieving the nickname. The inserted text can ask a question such as “What other name is Pittsburgh known by?” and place the returned answer inside the marked call before the sentence continues. To the external service, this is a question-answering request. To Toolformer, it is a span of text that provides context for the tokens that follow.
The same pattern works for operations that are not factual lookup. A sentence may contain two numbers and need their ratio. The model can generate a calculator expression, receive a textual number, and continue the sentence with that value. A non-English phrase can be surrounded by an English continuation that becomes easier after a translation result is supplied. A temporal statement can receive a date from a calendar. In every case, the service crosses a narrow interface: it accepts text, returns text, and leaves the language model to connect the response with the surrounding passage.
That interface also establishes ownership. The language model owns the decision to express a request and the continuation after the response. The external API owns the operation that turns the request into a result. During data creation, the result is attached to the proposed call; during generation, the model does not invent the returned value token by token. A serving process pauses generation, obtains the response, inserts it with the closing marker, and lets the model proceed. The boundary is useful precisely because the two sides have different responsibilities.
It also creates failure conditions. If the API's input is malformed, it may return no usable result. If the returned text is irrelevant, the model has gained context without gaining useful information. If the model places a call in the middle of a phrase where no outside fact is needed, the serialization still looks valid but the behavior is poor. Text makes the interaction learnable; it does not make the interaction semantically correct.
The sequence separates proposal from acceptance. The base model proposes the call, and the data-building process executes it so that a response can be observed. The original continuation remains part of the example; the returned text is not a replacement for the passage. At this stage, the interaction is only a candidate. Toolformer still needs a way to determine whether the result helps with the continuation rather than merely fitting the syntax of a call.
From a passage to a candidate request
Toolformer does not wait for a human to mark every useful position in a huge language-modeling corpus. For each API, the authors provide a prompt that tells the language model what kind of calls to add and what a helpful request looks like. The prompt is a small amount of guidance about the interface. It does not tell the model which position in every future sentence should receive a call.
The base model then examines an ordinary text sequence from left to right. At each position, it estimates how likely it is to begin an API call, given the API-specific prompt and the text prefix that has already appeared. In the paper, this probability is written as pᵢ, the probability assigned to the API-opening marker at position i. A sampling threshold keeps only positions where the model has enough inclination to begin a call. If many positions qualify, only the highest-ranked ones are kept, up to a fixed cap.
This is an important difference from a hand-written router. A rule-based router might say that every expression containing numbers should go to a calculator, or that every named entity should trigger a search. Toolformer's proposal step instead asks the model to recognize a possible need in context. The surrounding words matter. A number in a historical date may be part of a fact, while a number in an equation may be an operation. A foreign-language phrase may need translation in one passage but be quoted intentionally in another.
At each retained position, the model samples one or more possible API calls. It generates both the API name and the input, using the text before the position as context and the API's closing marker as the end condition. A question-answering prompt may produce differently worded questions for the same sentence. A calculator prompt may produce different expressions, some well-formed and some not. Sampling several possibilities gives the later selection step alternatives to compare instead of treating the first generated request as authoritative.
The position score is a gate for expensive exploration, not a final confidence estimate. A high probability of beginning a call says that the prompted model finds the call format plausible at that point; it does not say that the API response will be correct or that the response will improve the continuation. Likewise, the cap on retained positions prevents every possible location in a long passage from generating requests. The method is balancing coverage against the cost of executing many external calls, even before the paper's later usefulness filter is applied.
Because the candidates are conditioned on the prefix, a request can be locally sensible but globally misplaced. A sentence may mention a number that is already sufficient for the next phrase, or a named entity whose identity is clear from the surrounding context. The later comparison must see the entire continuation to determine whether a specialist supplied something the model actually needed. Candidate sampling creates the evidence pool; it does not decide the case.
The process also has a basic syntactic gate. If the model fails to generate the closing marker, the example is discarded. An unclosed call has no reliable boundary between the request and the surrounding passage, so it cannot be inserted cleanly. This is a small rule, but it reveals the layered nature of the system. A call must first be structurally recognizable, then executable by the API, and only later judged useful for prediction. Passing one layer does not imply passing the next.
The prompts make the request format concrete without making the model a specialist in the tool. For the question-answering API, the model is asked to insert questions whose answers supply information needed to complete the text. For Wikipedia search, it is asked to add a search term where retrieved material could help. For the calculator, the requested input is an expression rather than a natural-language question. For translation, the input is the phrase to render in English. For the calendar, the call has no input. These differences are necessary at the API boundary, but the resulting interactions share the same surrounding text structure.
That shared structure is what lets Toolformer use one base model for heterogeneous capabilities. The model does not need to predict a special internal object called “arithmetic assistance.” It needs to produce a token sequence that names the calculator and supplies an expression. It does not need a separate output head for retrieval. It needs to produce a marked search request that can be executed and placed back into context. The conceptual choice—what sort of help is needed—appears through ordinary token prediction.
But the proposal distribution should not be confused with a learned policy that is already reliable. The model has been asked to annotate text, so it can propose calls because the prompt makes that behavior likely. It may ask a question whose answer is already visible in the sentence. It may generate a search term that is too vague. It may calculate an expression unrelated to the continuation. It may produce a valid translation in a place where the original phrase was not a missing input at all. The paper's sampling stage deliberately exposes these possibilities so a later criterion can separate them.
The distinction between a candidate and an accepted example is therefore central. A candidate says, “This request could be inserted here, and the external system returned this text.” An accepted call must meet a stronger condition: its result must make the model better at predicting the future tokens in that particular passage. A candidate can be syntactically complete and semantically plausible yet fail that test. Conversely, the value of a request is judged in context, not by how informative its result sounds in isolation.
This arrangement is why Toolformer's self-supervision is neither ordinary human annotation nor unrestricted self-belief. People specify the available tools and demonstrate their intended format. The base model generates a large set of possible interactions. The external APIs provide responses. The continuation of the original text supplies the signal used to decide whether those responses helped. Human effort defines the interface, while the model's own prediction problem provides the large-scale source of candidate examples.
The proposal stage also explains why the result remains tied to the model's original abilities. Toolformer can only ask for what it has learned to articulate. Its language affects which API it selects, where it places the request, and how precise the arguments are. An API may have an excellent answer available, but that answer cannot help if the model never requests it or requests it with the wrong input. The text interface gives the model access to specialists; it does not remove the need for the model to formulate a useful question.
At the end of this stage, nothing has yet established that a call deserves to influence the model's future behavior. The system has transformed plain passages into a pool of executable alternatives, each carrying an input, a result, and the continuation that followed in the source text. The next question is more demanding than “Can the call run?” It is whether giving the result actually changes the model's prediction in a useful direction. That comparison is what turns generated tool use into training evidence.
References