Designing an On-Device AI Feature That Can Survive Reality
The question running through these two on-device deployments is not whether a phone can run a language model. It is where responsibility should live when the model must share a small, fixed, and sometimes unpredictable environment with the rest of a product. The practical answer is a design framework: keep hard guarantees with product logic and deterministic data, give the model a bounded task that creates user value, and use the runtime and hardware to make that task fit the device. This framework lets a team decide when to narrow the model's role, when to invest in controlled adaptation and acceleration, and what evidence is needed before shipping.
That answer is more demanding than either “put the model on the device” or “use the biggest model that fits.” It treats on-device AI as a complete product architecture. The model is one component in a path that includes a user contract, data, parsing, validation, context, adapter or task selection, graph execution, measurement, and failure handling. A feature survives reality when each part has an owner and the boundaries between them are explicit.
A product architecture of explicit ownership
The first design decision is to write down what must be true for the user. In a word game, the hidden word must obey the game's rules and the game must remain playable. In a writing feature, the output may need to appear quickly enough to feel interactive and may need to stay within a defined task or language. These requirements are not automatically model responsibilities just because the model is the most visible AI component.
Palabrita makes the distinction unusually clear. Its early design asked the local model to select a word, describe it through several fields, and produce five hints in structured output. That placed hard lexical requirements, numerical reasoning, multilingual behavior, and creative writing in one probabilistic response. The final design moved word selection to curated language-specific assets and kept the model's job to three short hints. The product could check the word before play began; it could accept, retry, or replace the hints after generation; and it could show generic deterministic hints if the model failed.
The change did not remove intelligence from the product. It changed what intelligence was responsible for. The model still contributed language that could be contextual and varied. It simply no longer decided whether the puzzle was valid. That is what “the model owns less” means in a complete architecture: not that the model is decorative, but that its failure cannot invalidate the product's foundation.
A useful ownership map has five layers. The product contract defines the user-visible promise and decides whether a result may be committed. Deterministic data and logic own facts and rules that must be exact. The bounded model owns a transformation or creative contribution for which some variation is acceptable. The runtime owns loading, context, task or adapter selection, parsing, validation, retry, and fallback. The target hardware owns the execution envelope in which memory, initialization, and decoding are measured.
These layers are coupled, but they should not be confused. The model can return a candidate; the runtime can interpret it; only the product should commit it as an accepted result. The runtime can select an adapter; that does not mean the model can accept any adapter shape. The hardware can execute a quantized graph quickly; that does not establish that the output meets the product's quality bar. Explicit ownership prevents a success in one layer from being mistaken for a guarantee in another.