Where the Speed Comes From
Apple’s two foundation-model paths are defined not only by where they run but by what each path spends its limited resources on. This chapter follows the main efficiency mechanisms in the on-device and server designs—cache sharing, low-bit weights, sparse experts, and mixed attention—and asks what each mechanism saves, what it costs, and how the reported quality changes should be read. The practical result is a way to connect an internal optimization to something a user can notice: memory pressure, first-token delay, throughput, or answer quality.
On-device efficiency: reduce what must be stored and computed
Autoregressive generation has two distinct phases. During prefill, the system reads the prompt and builds an internal representation of its context. During decoding, it generates one token at a time, using the prior context to choose the next token. The first phase affects time to the first token; the second affects the time and memory required to continue the response.
A transformer does not need to recompute every earlier token from scratch at each decoding step. It keeps a key-value, or KV, cache. The cache stores intermediate attention information from earlier tokens so later steps can reuse it. Reuse saves computation, but the saved information occupies memory. On a device, that memory is not an abstract accounting line. It competes with the operating system, the application, the image input, and other active work. A design that reduces cache size can make longer or more interactive sessions practical even if the model’s nominal parameter count does not change.
Apple’s on-device model changes the way the cache is produced. The report describes dividing the transformer into two blocks. The second block keeps its transformer layers but removes its own key and value projections; it directly shares the cache generated by the first block. Apple reports that this reduces KV-cache memory use by 37.5 percent. Because the second block does not need to produce keys or values during prefill, the prefill stage can bypass that computation, and Apple reports an approximately 37.5 percent reduction in time to the first token.
The two percentages express related but different savings. The first is about persistent state: less cache must be stored. The second is about work before generation begins: some computation can be skipped. Neither means that every response is 37.5 percent faster from start to finish. The reported design concerns the described cache and prefill paths, while end-to-end behavior also depends on prompt length, generated length, device hardware, scheduling, and other runtime work. The mechanism is valuable precisely because it targets two common sources of friction without requiring the model to become a different kind of product.
The other major on-device intervention is weight compression. Apple reports using quantization-aware training to compress the model to 2 bits per weight, quantizing the embedding table to 4 bits and the KV cache to 8 bits. Quantization replaces higher-precision numerical values with a smaller set of representable values. That reduces storage and the amount of data that must move through memory, but it also introduces approximation error.
Why does quantization require training? A simple post-training conversion takes a model trained with one numerical representation and substitutes another after the fact. The model may be sensitive to small changes in many layers, especially when only a few low-bit values are available. Quantization-aware training instead simulates the quantization effects while the model learns. The forward computation sees the approximate values, so training can adjust the weights around the distortion.
The report describes a straight-through estimator for the rounding operation, which is not differentiable in the ordinary way. During backpropagation, the estimator supplies an approximate gradient so the model can still learn. Apple also uses a learnable scaling factor for each weight tensor, rather than relying only on a fixed scale derived directly from the weights. Under aggressive 2-bit quantization, the choice of scale determines which values are represented well and which are clipped or crowded together. The report says an iterative initialization procedure is used to make the early training dynamics more stable.
That detail reveals an important principle: low-bit inference is a training-and-runtime co-design problem. The runtime wants a small, regular representation. The training process must learn weights that remain useful under that representation. Apple also reports choices such as AdamW instead of Adafactor, lower learning rates, gradient scaling, exponential moving averages, and a balanced four-value set for the 2-bit weights. These are not separate performance decorations. They are ways to keep optimization stable when the model has little numerical room to express change.
Even with QAT, compression can lower quality. Apple reports a representative comparison in which the on-device model moves from 16 bits per weight to 2 bits per weight, with MMLU falling from 67.8 to 64.4 and IFEval falling from 85.1 to 82.3. The figures show a cost, not a failure. A model that cannot fit the device is not an available baseline for the product, so the relevant comparison is between the quality of a deployable compressed model and the value of the capability it enables. The numbers also do not establish equal quality across every feature, language, prompt length, or device.
Apple uses low-rank adapters as one way to recover quality lost during compression. An adapter adds a small learned correction while the main weights remain frozen. The correction can compensate for recurring compression artifacts without retraining the entire base model. In Apple’s description, adapters are also used with the server compression path. This is a recurring systems pattern: reduce the expensive common artifact first, then store a smaller correction targeted at the quality that matters.
The on-device stack can therefore be read as a sequence of linked decisions. Cache sharing reduces the state that must remain in memory and the work needed during prefill. Quantized weights reduce storage and bandwidth. Quantization-aware training teaches the model to tolerate the smaller numerical representation. Low-rank adapters recover selected quality. Each step exists because the previous step creates a new constraint or a new opportunity.
The process looks like this:
The diagram separates the two moments that users experience as one response. Prefill builds the state that makes decoding possible; decoding repeatedly reads weights and updates the cache. Compression affects the movement of weights and state throughout the process. Cache sharing primarily changes how state is produced and stored. A first token can arrive sooner while later tokens still depend on the cost of repeated decoding.
Server efficiency: move synchronization out of the hot path
The server model faces a different bottleneck because its capacity is distributed across accelerators. More hardware can increase available computation, but only if the system does not spend the added capacity waiting for every part to synchronize. Apple’s Parallel Track Transformer changes the shape of that coordination.
In a conventional sequential picture, every token travels through one long stack of layers. A distributed implementation may split the work across devices, but the devices need to exchange intermediate information. Apple’s design partitions the model into several tracks, each containing multiple track blocks. The tracks process tokens independently within a block, and synchronization is applied at block boundaries. The report gives a block-depth example in which this reduces synchronization overhead by 87.5 percent compared with the tensor-parallel pattern it describes.
The underlying saving comes from allowing independent work to continue for longer before a communication point. A synchronization boundary is not free: devices may wait, exchange data, and then resume. Moving the boundary outward can improve overlap between computation and communication. But it also makes each block a larger unit of work and can affect how errors, load, and intermediate state are managed. The report explains the intended latency benefit; it does not provide a complete service-level model for all workloads.
Parallel Track Mixture of Experts adds selective computation. A dense feed-forward layer applies the same transformation to every token. A mixture-of-experts layer contains several expert transformations and routes each token to a selected top-k subset. The total collection of experts can be large, while any one token uses only part of it. In Apple’s design, experts are local to their tracks. That locality matters because remote expert selection would add more communication to the very operation meant to save computation.
Apple reports using grouped matrix multiplication for routing and says that tokens are not dropped. Keeping every token in the calculation avoids one failure mode in sparse systems, where capacity limits can cause an overflow token to be discarded or diverted. It does not eliminate all routing concerns. A set of tokens that concentrates on a few experts can still create an uneven load, and the report does not publish every routing or capacity condition. The safe conclusion is that PT-MoE is designed to make sparse computation more compatible with distributed execution, not that it removes the operating costs of sparsity.
Long contexts create a related but distinct problem. Full global attention lets every position interact with every other position, but maintaining that access becomes expensive as the sequence grows. Apple’s server model interleaves three local-attention layers with one global-attention layer. The local layers use a sliding window of 4096 tokens, restricting most interactions to a neighborhood. The global layer reconnects the broader sequence. Apple also describes omitting positional embeddings in the global layer to improve length generalization and reducing KV-cache size for long-context inference.
These layers divide labor rather than choose one kind of attention as universally correct. Local attention handles repeated nearby relationships at lower cost. Global attention appears often enough to reintroduce broader context. The design can preserve useful long-range access while reducing the cache and computation burden that would result from global attention everywhere. The trade-off is that the system’s behavior depends on where the global layers occur and how the local window interacts with the task. A long sequence is not automatically understood merely because it fits into a nominal context limit.
The server path therefore uses a different efficiency stack from the on-device path:
- Track parallelism reduces how often distributed components must synchronize.
- Track-local experts keep sparse routing closer to the computation that uses it.
- Top-k routing limits the active expert work per token while retaining tokens in the described implementation.
- Local attention limits most interactions to a sliding window.
- Global attention restores broader connections at selected points.
- Server-side compression reduces weight storage and bandwidth, with hardware-aware ASTC decompression and low-rank quality recovery.
The compression results make the trade-off visible. In Apple’s representative table, the server model moves from 16 bits per weight to about 3.6 bits per weight. MMLU changes from 80.0 to 79.2, while IFEval changes from 89.1 to 90.2. The mixed direction of those changes is a reminder that compression effects are metric-specific; a small loss on one benchmark does not imply a uniform loss everywhere, and an improvement on another does not prove that compression improves the model in general. The table is evidence about the reported evaluation setup, not a complete energy, cost, or real-world quality model.
Apple’s server compression uses ASTC, a block-based format originally developed for graphics textures. The report says the model weights are arranged in 6-by-6 blocks and encoded into 128-bit values, producing a storage footprint of approximately 3.56 bits per weight. Apple leverages fixed-function decompression hardware in Apple GPUs so the compressed values can be decoded while the model runs. Because the described HDR channel does not represent negative values directly, Apple subtracts each block’s minimum before compression and stores that minimum separately; the minimum is added back during computation.
The important system-level consequence is not the format’s history. It is that the hardware already knows how to decode a compact representation, so neural-network weights can use a path that avoids occupying general-purpose compute resources for decompression. Apple reports no added latency during prompt-time inference for this transformation. That is a claim about the described hardware and integration, not a promise that every source of server latency disappears. Communication, routing, batching, cache behavior, and service load still matter.
Across both paths, the central pattern is now clear. Efficiency is not one switch labeled “optimization.” It is a series of decisions about what to store, what to recompute, where to synchronize, which experts to activate, how far attention should reach, and which numerical errors can be recovered. The price is that every saving has a boundary. Lower precision can affect quality; sparse routing can complicate load balance; local attention can limit direct context; block-level synchronization can change granularity; hardware-aware compression depends on compatible hardware.
That is why the right conclusion is a deployment frontier rather than a universal winner. The on-device path spends engineering effort to make a compact model responsive within a device’s memory and bandwidth limits. The server path spends engineering effort to make distributed capacity communicate efficiently while preserving long-context and model quality. Apple’s reported benchmark table shows that the compressed versions remain useful under selected tests, but it also preserves enough movement in the metrics to show that deployability is purchased, not granted for free. The next judgment is therefore conditional: which combination of sensitivity, latency, context, quality, and scale can a feature afford?