The Architecture of the Split: Device, Server, and Boundary
This chapter looks inside Apple’s two foundation-model paths and asks a narrower question than the opening one: what responsibilities belong to the model, the device runtime, the app framework, and Private Cloud Compute? The answer matters because an Apple Intelligence feature is not just a prompt sent to a neural network. It is a chain of components, and the quality of the result depends on how those components divide work.
The useful distinction is between a model’s internal capability and the system that makes that capability available. Apple’s report describes an approximately 3-billion-parameter on-device model and a scalable server model for Private Cloud Compute, but it also describes the surrounding hardware, runtime, and developer interfaces. Those pieces are not interchangeable. A compact model can be more valuable when the framework turns its output into a reliable app action; a large model can be less useful if the surrounding system cannot meet the task’s privacy or responsiveness requirements.
The on-device path: capability close to the application
The on-device path begins with physical proximity. The model runs on Apple silicon rather than waiting for a remote service to produce every token. That placement does not make the model independent of software around it. It changes the constraints that the surrounding software must manage.
Apple describes the on-device model as optimized for low-latency inference and minimal resource use. Its intended scope is also deliberately focused. The report lists summarization, entity extraction, text understanding, refinement, short dialogue, and creative generation among the tasks it supports, while explicitly saying that the model is not designed as a general chatbot for world knowledge. This is a product and architecture decision at the same time. If an app needs a short summary or a structured extraction, the model does not need to present itself as an all-purpose conversational authority. The app can define a narrower job and use the model where that job fits.
That specialization changes what “the model” means in practice. The model produces tokens, but the application may need a field, a classification, a rewritten passage, or a sequence of tool calls. The Foundation Models framework sits between those needs and unconstrained generation. Apple describes guided generation in which a developer’s Swift data structure is turned into a response-format specification. The system then uses constrained decoding so the output conforms to the expected structure and can be materialized as a Swift value. The developer is not required to prompt the model into inventing a format and then hope a string parser can recover it.
The framework therefore owns an important part of the interface contract. The model still determines the substance of the generated content, but the framework constrains the shape of that content. This distinction is easy to miss. A structurally valid object can still contain an incorrect date or an incomplete extraction. Constrained decoding reduces one class of failure—malformed structure, hallucinated tool names, or invalid arguments—but it does not turn uncertain model judgment into verified fact.
Tool calling makes the division more visible. A developer supplies a tool implementation that exposes a particular information source or service. The framework uses the same guided-generation foundation to keep tool names and arguments structurally correct and to manage serial or parallel call graphs. The model can decide that a tool is needed and generate the call, but the tool executes outside the model’s token prediction. Its result then becomes part of the conversation state or the input to a later response. The boundary is not between “AI” and “software” in the abstract; it is between a probabilistic generator and a controlled operation with its own data and failure conditions.
State also belongs somewhere. Apple’s LanguageModelSession is described as append-only and coupled to the model’s key-value cache. A session preserves context while the model generates more content, and the framework tries to prevent developers from accidentally invalidating the cache. That is a runtime responsibility, not a new language capability. Filling the cache can introduce latency, so preserving it can make repeated interaction more efficient. Streaming snapshots similarly let the framework handle partially generated output while the developer sees a Swift object grow over time.
This design produces a layered on-device path:
- Apple silicon supplies the local compute and memory environment.
- The approximately 3-billion-parameter model generates language and supports the report’s described text and multimodal features.
- The runtime manages inference behavior and the state associated with generation.
- The Foundation Models framework converts model output into constrained structures, tools, sessions, and streams.
- The application decides what to ask for, what tools to expose, and what to do with the result.
Those layers have different failure modes. A device can be resource-constrained. A model can misunderstand a request. A schema can be valid while its values are wrong. A tool can fail or return stale information. A session can contain context that should not be carried into a later task. Keeping the ownership boundaries visible is more useful than calling the whole stack “the model.”
The same principle explains why Apple’s framework is not presented as a general chatbot interface. A focused app can supply context that the model does not have as general world knowledge, constrain the output to the application’s data shape, and expose only the tools needed for the job. The model’s limitation becomes part of a controlled design rather than a defect to hide. But that control depends on app design and does not remove the need to test content accuracy and safety.
The server path: scale organized around communication
The server path has a different physical problem. It can use a distributed environment, but distributing a transformer creates communication and synchronization costs. Apple’s server model addresses those costs with a Parallel Track Transformer, Parallel Track Mixture of Experts, and an interleaved pattern of local and global attention.
A standard decoder can be imagined as one long stack of layers through which each token passes. In Apple’s Parallel Track design, the model is partitioned into smaller transformers called tracks. Each track contains blocks made of multiple layers. The tracks process tokens independently within a block, and synchronization happens at the boundaries of those blocks rather than after every layer. Apple gives an illustrative example in which a block depth of four reduces synchronization overhead by 87.5 percent relative to the tensor-parallel pattern described in the report.
The important idea is not the example’s percentage by itself. It is the location of synchronization. If every layer requires all the distributed pieces to coordinate, communication can dominate the benefit of adding more accelerators. If independent work can continue for several layers before a boundary exchange, the system has more room to overlap computation with communication. Track parallelism is therefore a response to a distributed execution constraint, not simply a new naming scheme for a larger model.
The mixture-of-experts layer adds a second form of selective computation. Instead of sending every token through the same dense feed-forward network, the model routes tokens to a selected subset of experts. In Apple’s design, experts are local to their respective tracks, which allows communication to overlap more effectively with computation. The report says grouped matrix multiplication is used for top-k routing and that tokens are not dropped during routing. This can improve the relationship between total model capacity and the amount of computation used for a particular token, but it also creates routing and load-management concerns. The report does not disclose every expert count, routing load, or service-capacity condition, so the architecture should not be mistaken for a complete operational description of PCC.
Long context creates another problem. A model that attends globally across a long sequence must maintain and read a large amount of information. Apple’s server architecture alternates three local-attention layers, using a sliding window of 4096 tokens, with a global-attention layer. Local attention limits most interactions to a neighborhood; the global layer restores access to information across the wider sequence. The report also describes omitting positional embeddings in that global layer to improve length generalization and reducing the KV-cache size for long-context inference.
Local and global attention are not competing philosophies in this design. They divide labor. Local layers handle repeated nearby interactions efficiently; occasional global layers reconnect the broader context. The arrangement is another example of an architecture that is shaped by a deployment cost. The goal is not simply to maximize the amount of information the model can theoretically see. It is to make long-context access affordable enough for a server system that must serve many requests.
The server path can therefore be described as a set of responsibilities too:
- Track parallelism organizes when distributed components must synchronize.
- Mixture-of-experts routing spends computation selectively while keeping experts close to their tracks.
- Local and global attention balances nearby context with long-range access.
- Private Cloud Compute supplies the intended server deployment environment described by Apple.
- The feature or application still determines what task is requested and how a response is used.
The architecture supports a scalable service, but “scalable” needs a condition attached to it. It is plausible only if the communication pattern, routing behavior, hardware, request mix, and service controls work together. Apple’s report explains the model-side mechanisms and presents the server model as designed for PCC, but it does not publish the full production capacity plan, routing thresholds, per-request cost, or failure behavior. Those omissions do not invalidate the architecture. They define the edge of what the report allows a reader to conclude.
The two paths and their interfaces can be pictured as follows:
The decision diamond is a conceptual model, not a disclosed Apple routing algorithm. The report does not tell us that every feature follows exactly this sequence or that the developer framework controls both paths identically. What it does establish is that the paths have different intended roles and that the surrounding platform is part of the system. The diagram keeps those responsibilities separate while showing how both can serve one feature-level result.
This separation also clarifies what crosses the boundary and what does not. A model’s weights and attention mechanism define how it transforms input into output. The operating system and hardware determine how that transformation is scheduled and stored. The framework determines how output is constrained, streamed, and connected to tools. A PCC deployment supplies a different execution environment for the server architecture. The supplied evidence does not disclose every data-flow or attestation detail, so one should not invent a precise handoff protocol. But any real handoff must be judged by the information it moves, the latency it adds, the state it preserves, and the failure path it creates.
The architecture of Apple’s system is thus a division of responsibilities rather than a catalogue of model features. The on-device path brings computation close to the user and narrows the task so a compact model can be useful. The server path organizes parallel capacity around communication and context costs. The framework turns model output into an app-facing contract, while PCC provides the environment for the server design described in the report. Once those responsibilities are visible, the next question becomes concrete: which caching, quantization, sparsity, and attention mechanisms make the two paths affordable to run?