Making the Frozen Graph Fast Enough

Samsung's one-for-all system solves the task-switching problem with a shared graph and runtime adapters; this chapter focuses on the next problem: making that graph fit mobile hardware and respond quickly enough. The useful judgment is to map each optimization to the bottleneck it addresses—storage, operator execution, shared prompt processing, serial decoding, or token verification—and to recognize the specialization each improvement requires.

A language model is not just a collection of weights waiting for a faster processor. It is a computation graph: a sequence of operations that transforms token data into the next-token probabilities used for generation. A phone's accelerator is good at some shapes and precisions, less good at others, and constrained by memory movement as well as arithmetic. Samsung's approach therefore changes both the representation of the graph and the schedule by which it produces tokens.

Shape the graph for the hardware

The first group of optimizations changes how the same mathematical work is presented to the device. In a transformer, multi-head attention runs several attention heads whose results are combined. Samsung describes decomposing that multi-head operation into parallel single-head paths so they can be mapped to independent accelerator resources. The computation's meaning is intended to remain the same, but its shape becomes easier for the target hardware to schedule in parallel.

This is a crucial distinction between algorithmic equivalence and deployment equivalence. Two graphs can represent the same function while producing different kernel choices, memory access patterns, and launch overhead on a phone. A transformation that looks like bookkeeping at the model level can therefore change latency if it aligns the work with the accelerator's preferred operations.

The same reasoning motivates representing dense linear layers as 1x1 convolutions. A linear layer and a point-wise convolution can be mathematically equivalent, but mobile chips have often received highly optimized convolution kernels because of their graphics and vision workloads. Recasting the layer allows the language model to reuse those execution paths rather than demanding a special kernel for every dense operation.

Graph simplification adds another layer. Constant folding precomputes operations involving values that do not change at runtime. Operator fusion combines adjacent operations so the device launches fewer kernels and moves intermediate data fewer times. These changes do not make the model more intelligent. They reduce the overhead around intelligence by making the compiled representation more economical for the hardware that will run it.

The ownership boundary is visible here. The foundation model defines the computation, but the compiler and runtime decide how that computation is represented for the device. A product that supports only a generic graph may leave performance on the table. A product that specializes the graph for one accelerator can gain speed but may give up portability. Samsung's reported results belong to its deployment path on Qualcomm-based Galaxy hardware; they should not be read as proof that the same transformation has the same benefit on every phone.