The Edge Is Not One Number

The two on-device deployments in this series report impressive numbers, but the subject here is how to interpret them without turning a conditional result into a universal promise. The focus is the connection among device, model, workload, metric, baseline, and product requirement. That connection gives the reader a practical decision rule: a benchmark supports a product only when the conditions that produced it match the conditions under which users will depend on the feature.

This matters because “fast on-device AI” can describe several different things. It might mean a low time to the first token, a high steady-state token rate, a short total response, a small model file, or a quality score close to a higher-precision baseline. Those measures answer different questions. A feature can be fast after loading but slow on first use, efficient for one short output but expensive for eight, or close to its baseline in one language while falling behind in another.

Read the unit behind the headline

Samsung's one-billion-parameter experiment ran on a Galaxy S24 using the device's neural processing unit. Its one-for-all table reports 967 MB of peak memory, 624.1 ms of load time, 155.9 ms to the first token, 24.19 ms per token, a generation rate of 41.32 tokens per second, and 2.85 seconds end to end. Each number describes a different part of the user path. Peak memory affects which devices can run the model alongside the rest of the application. Load time affects first use and model transitions. First-token latency affects how quickly the interface begins to respond. Per-token latency affects the duration of longer output. End-to-end time is the closest of these figures to a complete interaction, but it still describes the measured workload rather than every use case.

The three-billion-parameter experiment on the Galaxy S25 Ultra reports a different envelope. Across the displayed use cases, peak memory is around 2,463–2,486 MB and initialization is around 1,487–1,577 ms. With dynamic self-speculative decoding, the reported token rates range roughly from 25 to 45 tokens per second, depending on the task. It would be misleading to place those values beside the one-billion-parameter results and announce a simple winner. The devices, model sizes, decoding methods, and use cases differ. The figures show that a larger local model can be deployed under a particular accelerator and optimization stack; they do not show that every larger model is faster or that every phone can reproduce the result.

The accuracy table needs the same care. Samsung compares its quantization-aware on-device model with an FP32 version using G-Eval scores, with ChatGPT 4.5 as the evaluator. Its relative-accuracy measure is the on-device score divided by the FP32 score, multiplied by 100. A value near 100 therefore means “close to that comparison baseline,” not “correct for nearly every user.” Values above 100 are not literal human accuracy, and values below 100 can matter even when an aggregate impression looks favorable.