When a Model Meets a Task: Context, Contracts, and Retrieval
The next problem in ai-engineering-from-scratch is not how a language model generates tokens. It is how an application gives those tokens the right information, turns them into a usable result, and connects them to an external capability without treating fluent output as proof. This chapter follows the repository’s LLM-engineering path through context, structured outputs, retrieval, and tools so the reader can see what each layer guarantees—and where the uncertainty remains.
A useful application is not a larger prompt wrapped around an API call. It is a chain of boundaries. Context decides what the model can see. A schema constrains the shape of what it returns. Retrieval supplies external material. A tool loop crosses from language into an operation. Validation and graceful degradation decide what happens when one of those boundaries fails.
Context is a scarce allocation
The repository treats context engineering as a resource-allocation problem. A model’s window contains more than the user’s latest sentence: system instructions, tool definitions, conversation history, examples, retrieved documents, and room for the answer all compete for space. Adding one component displaces another. The practical question is not “how much text can fit?” but “which information deserves to be present for this decision?”
That distinction changes how an engineer reads a long context window. A large capacity is not the same as a large effective signal. Stale conversation turns can crowd out the current task. Irrelevant tool descriptions can make the available capabilities harder to select. Retrieved passages can add noise, contradictions, or instructions that have nothing to do with the user’s question. The repository therefore emphasizes budgeting, relevance filtering, compression, ordering, and dynamic assembly.
Imagine a support request that asks whether a customer qualifies for a refund. A static system prompt may define the role. The conversation may establish the product and date. A retrieval step may supply the current policy. A tool may look up the order. The answer still needs generation space, and each piece must be placed so the model can use it. If the policy is buried among unrelated documents, or if an old conversation turn overrides the current order state, the failure is not solved by giving the model more words.
This is also why context engineering sits between model capability and application behavior. The model’s weights contain general patterns, but the application chooses which task-specific facts and instructions enter the current computation. The same model can behave differently because the context differs. A useful diagnostic is to ask whether a surprising answer follows from missing information, badly ordered information, an irrelevant retrieval, or an incorrect interpretation. Those causes require different repairs.
The repository’s context lesson also connects short-term, long-term, and episodic memory to this budget. Current conversation belongs directly in the window. Durable facts can be stored and retrieved later. Past interactions may be recalled by similarity. None of these stores is automatically true merely because it is persistent. Persistence changes availability; it does not grant authority. A remembered preference can be stale. A retrieved document can be wrong. The system still needs validation appropriate to the task.
The refund example exposes why these choices cannot be optimized separately. If the system retrieves the policy but omits the order date, it has evidence without the fact needed to apply it. If it includes the order but keeps an obsolete policy in the conversation history, it has facts with competing authority. If it loads every policy version, it spends context on material that the current request does not need. A context assembler should therefore preserve provenance and relevance, not simply concatenate sources. The learner’s practical question is: what must the model see to make this decision, what can be left out, and which component can prove that the selected material is current?
That question also clarifies the difference between memory and retrieval. Memory can preserve a durable preference such as a preferred language; retrieval can fetch the policy that governs today’s refund. Treating both as undifferentiated “context” hides their different update paths and trust levels. A system that remembers a user’s preference for concise answers should not use that memory as authority to approve a refund. The repository’s layered view is useful precisely because it forces the engineer to identify what each piece of context is for.
A contract can make output usable without making it true
Free-form text is difficult to pass safely to another component. A program can ask a model to “return JSON,” but that request alone does not guarantee valid syntax, stable field names, correct types, or complete values. The repository describes a progression from prompt-based formatting, through JSON mode, to schema-based output and constrained decoding. The progression matters because it separates kinds of guarantees.
Syntactic validity answers one question: can the consumer parse the result? Schema validity answers another: does the parsed object have the required fields and types? Neither answer proves that the values match the source material. A model can return a perfectly valid object with an invented price, an incorrect date, or an unjustified classification. The contract protects the interface, not the truth of every claim inside it.
That distinction is central to the repository’s evidence theme. A schema is valuable because it moves part of the system’s responsibility out of prose and into a checkable boundary. Required fields make omissions visible. Types prevent some categories of malformed output. Enumerated values can reduce ambiguity. Retries can respond to a validation error. But factual validation still belongs to the application or to a later evaluation step.
The same logic applies to tool use. A tool definition gives the model a name, description, and typed argument shape. The model can select that tool and emit structured arguments. The application owns execution: it validates the call, invokes the capability, captures the result, and returns an observation to the model. The returned observation then becomes input to the next turn. The model’s request is not the operation itself.
This ownership boundary prevents a dangerous shortcut. A model cannot create a capability by mentioning it, and a schema does not authorize every side effect the schema can describe. A weather lookup, database query, file write, or message send needs an application-side decision about identity, permission, error handling, and scope. The tool loop makes delegation possible, but it does not remove the need for authority.
Retrieval supplies evidence, not a verdict
The repository presents retrieval-augmented generation as a three-part pattern: retrieve relevant material, augment the model’s context, and generate an answer grounded in that material. This is useful when the needed information is fresh, private, or too specific to rely on the model’s learned weights. It also makes source visibility possible: the application can record which passages were selected and ask the response to cite them.
But retrieval is a pipeline, not a truth switch. Documents must be loaded, divided into useful chunks, represented, indexed, searched, and selected. A chunk that is too small may lose the reference needed to interpret a sentence. A chunk that is too large may dilute the relevant passage with unrelated material. A query may use the wrong vocabulary for the document. A vector match may be close in representation space without answering the user’s actual question.
Generation introduces another boundary. Even when the retrieved passage contains the answer, the model can misread it, combine it with an unrelated passage, or state a conclusion more strongly than the evidence allows. That is why the repository pairs retrieval with attribution and evaluation. Retrieval quality can be measured separately from answer quality. A citation can show the source passage, but a citation alone does not establish that the answer faithfully represents it.
The codebase-RAG capstone makes this architecture more explicit. It proposes combining different retrieval signals, reranking the candidates, requiring file-and-line citations, observing retrieval and synthesis separately, and rejecting answers that lack citations. Those choices do not make the system infallible. They make its evidence path more visible and its failures easier to localize. If the right code was not retrieved, the problem is upstream of synthesis. If the right code was retrieved but the citation is absent, the problem is in the response contract or post-filter. If the citation is present but the conclusion is wrong, generation or interpretation still needs investigation.
The practical test for a RAG design is therefore not “does it have a vector database?” It is “can the system show what it retrieved, why it was selected, how it entered context, and how the final claim was checked?” That is the repository’s distinction between grounding as a mechanism and groundedness as an evaluated behavior.
One request crosses several ownership boundaries
The repository’s production-application path combines these parts into a single request flow. An incoming request may pass through authentication and rate limiting, input checks, prompt or route selection, cache lookup, retrieval, model generation, output validation, evaluation logging, cost tracking, and streaming. The order matters because each stage receives a different kind of input and produces a different kind of evidence.
The model does not own the whole path. The gateway owns entry controls. The context assembler owns selection and budget. The schema or tool registry owns an interface contract. The application owns external execution. Validation owns a check against defined criteria. The service layer owns the decision to retry, bypass a secondary component, switch to a fallback, or stop.
That ownership map becomes especially important when a secondary component fails. The repository separates provider failures, model/output failures, and application failures. A provider may time out or reject a request. A model may return malformed data or an invalid tool name. A cache or vector store may be unavailable even though the primary model path is healthy. Treating all three as the same error produces poor recovery. The first may need bounded retry. The second may need a corrected request or a validation response. The third may justify bypassing the component with higher latency or lower context quality.
Graceful degradation is not the same as hiding failure. If retrieval is unavailable and the service answers without it, the response should carry the reduced-evidence condition in the system’s records and, where relevant, in the user-facing behavior. If a cache is bypassed, cost and latency may change. If a tool fails, the model should receive an observation that describes the failure rather than an invented success. Recovery is reliable only when the next decision sees what actually happened.
The repository therefore treats the production-shaped application as an assembly of explicit contracts. The route from prompt design through evaluation, cost, guardrails, serving, and observability is not a checklist of fashionable components. It is a way to make each transition inspectable. The application earns trust when it can say what information entered the model, what the model requested, what the software executed, what evidence returned, and why the final response was accepted.
The boundary before a shared ecosystem
A local function-calling loop is enough for a single application to call a small set of known capabilities. The repository’s next question is what happens when tools and procedures must be shared across applications and hosts. That move requires more than copying function definitions. It requires explicit protocol behavior, discovery, authority, and verification.
The next layer will use MCP and Agent Skills to examine that boundary. The distinction matters: a protocol can carry a capability surface, while a skill can carry a procedure for using capabilities. Neither should be confused with permission, and neither turns local tests into proof of a public deployment. The application chapter has now established why those distinctions are necessary: context supplies information, contracts shape exchanges, retrieval exposes evidence, and tools cross into action. Shared ecosystems must make each of those crossings visible rather than relying on ambient trust.