The Constraint Stack: Why the Phone Changes the Architecture
The subject here is the path from a user's request to a trustworthy on-device language feature. The focus is the constraint stack—readiness, memory, decoding time, context state, parsing, validation, and framework choice—and the judgment it enables: before selecting a model, a team should know which budget is binding and which layer can absorb the limitation.
A local model does not answer a request in one indivisible act. The application must make an engine available, prepare the prompt, wait for generated tokens, interpret the result, and decide whether that result can enter product state. Each step can be acceptable on its own while the complete path is unacceptable to a user. The architecture changes when the team measures the whole path rather than the model's isolated output speed.
A response has a lifecycle, not a single latency
Consider a foreground request: a player asks for a hint, or an application needs text before it can finish a screen. The first question is whether the engine is ready. Loading a model is a one-time cost in some sessions, but it is still part of the user's experience on first use, after a process restart, or after a model change. Once the engine is ready, the runtime performs a prompt-processing step often called prefill, then generates output token by token during decoding. The final text still needs to be parsed and validated before the application can use it.
Those phases have different behavior. In Palabrita's Pixel 7 Pro measurements, Gemma 4 E2B initialized in about ten seconds and decoded at roughly 30 tokens per second. Qwen3 0.6B initialized in about five seconds and decoded at roughly 35 tokens per second. The smaller model therefore had an advantage in startup and a modest advantage in decode rate on that setup, while the larger model produced more coherent hints and followed instructions better. Neither number, by itself, says whether a feature feels fast. The answer depends on how much text the feature asks for and how many attempts it needs.
A short hint of about 50 tokens took roughly one to two seconds on that device, which was acceptable for a single request. The original Palabrita design was much more demanding: a complete puzzle could require about 150–200 tokens, and one puzzle took roughly 5–7 seconds before accounting for retries. Generating seven puzzles with an average of 2.5 attempts per successful puzzle pushed the total beyond two minutes. A user waiting at onboarding experiences the total, not the model's per-token rate.
That distinction explains why a progress indicator cannot repair a bad interaction budget. If the operation spends most of its time generating content the user did not ask to watch being generated, or retries output that cannot be accepted, showing a percentage does not make the wait shorter. The relevant unit is the complete path: readiness plus generation plus recovery plus storage, under the device conditions that real users encounter.
It also means the application has to define what “ready” means. An engine may have been constructed but still be loading weights, selecting a backend, or waiting for enough memory. A screen that treats construction as readiness can invite a request into a path that is guaranteed to stall or fail. Conversely, an application that knows the engine is unavailable can choose a safe degraded response, keep the rest of the interface responsive, and make the boundary visible to the user. Readiness is therefore a state owned by the product, even though the runtime supplies much of the information used to determine it.
The same ownership question applies at the end of generation. Tokens arriving from the engine are provisional until the parser and validator accept them. A user can cancel while the model is decoding; a process can be interrupted after output is produced but before it is persisted; a retry can produce a second answer after the screen has already changed. The application needs a commit rule: only an accepted result enters durable product state, and an interrupted or invalid attempt does not masquerade as success. This is ordinary systems discipline, but local generation makes the discipline more important because the output is variable and the operation can be slow enough for state changes to race with it.
For that reason, a useful performance test measures more than an average. It asks how the feature behaves on a cold start and a warm start, after repeated requests, after an invalid response, and on the slowest supported device. A short average can hide a long tail created by initialization or retries. The user does not experience the benchmark row that looks best; they experience the path taken by their device, their request, and the current state of the engine. Choosing a workload whose slow path remains acceptable is part of deciding what the model is allowed to do.
The diagram also shows why the model cannot be the only owner of correctness. It emits tokens, but the application decides whether those tokens are structurally usable and whether they are safe to commit. If generation fails, the system must have a defined branch rather than an implied hope that another attempt will work.
Context is a runtime resource
A language model's conversation session is more than a record of what has already been said. The runtime carries prior turns through a context representation, including a key-value cache used by the attention mechanism. As more prompts and responses accumulate, the available context becomes crowded. Older material may be compressed or displaced, and the model's attention can become less effective. The result is not only more memory use. It can be a change in output quality.
Palabrita observed this degradation during manual development sessions. Qwen3 0.6B began to produce repetitive or less relevant hints after about three generations in a session. Gemma 4 E2B held up for roughly five to seven generations before a similar decline became apparent. The authors describe these as development observations, not controlled measurements, but the operational consequence is clear: keeping one session alive indefinitely was not neutral. A session had a freshness budget.
This creates a coupling between two seemingly helpful features. A persistent conversation gives a retry access to the reason its previous answer was rejected. That context can improve compliance when the model needs a reminder about a word length or an excluded value. The same accumulated context can later make outputs repetitive or cause the model to echo earlier answers. A fresh session removes some useful history while restoring a cleaner context. Session policy is therefore a product and runtime decision, not just a chat-interface detail.
The distinction matters for design reviews. If a team measures only the first request, it may miss the failure that arrives on the fourth. If it measures only a long batch, it may miss the startup cost that affects the first screen. A useful test asks how the feature behaves at each transition: cold start, ready state, repeated generation, invalid output, cancellation, and process restart. The model is one participant in each transition; the application owns the state machine that makes those transitions meaningful.
Parsing and validation close the probabilistic boundary
Generated text is not automatically product data. Even a response that looks correct to a person can fail a machine contract. In Palabrita, early local models often wrapped JSON in markdown code fences, translated keys such as word and hints into the requested language, or produced replacement characters that broke parsing. Other responses were valid JSON but contained the wrong word length, repeated a prior word, used the wrong language, or placed the target word inside a hint.
This is why parsing, validation, and fallback belong in the architecture even when a model performs well on a benchmark. A benchmark may measure the quality of an answer under a defined prompt. The product must decide whether this particular answer can be trusted at this moment, in this language, within this schema, and without violating the application's rules. Those are different questions.
The boundary should be explicit. The model can propose text. A parser can recover a supported representation from ordinary formatting variation. A validator can check conditions that the product cannot compromise. Only then should the application persist the result or expose it to the user. When the result is invalid, a retry may be useful if the failure reason can be made clear. But a retry is still another probabilistic attempt; it is not enforcement.
This layered view also changes how a team interprets “robustness.” A defensive parser may handle code fences or translated field names, but it cannot guarantee that a word has the right number of letters. A prompt can emphasize a constraint, but it cannot turn a small model into a deterministic counter. A fallback can keep the feature available, but it may provide less specific output. Each layer has a different job, and calling the whole system reliable requires saying which jobs are guaranteed and which are merely improved.
Framework choice changes the speed of learning
The inference framework is often treated as plumbing selected after the model. In a short development cycle, it determines how quickly a team can test a hypothesis about the product boundary. Palabrita's comparison illustrates the difference.
LiteRT-LM provided a Kotlin integration, coroutine support, selectable CPU, GPU, or NPU backends, emulator support, and control over prompts and sampling settings. It also accepted several model families in a common format, so changing from Gemma 3 1B to Qwen3 0.6B did not require moving to a different inference stack. That flexibility mattered because the team was changing prompts and responsibilities every one or two days. A slow model-evaluation loop would have made the observed design boundary harder to find.
MLC LLM offered broad model support, but the reported Android path required a more involved Rust, NDK, and TVM compilation toolchain, with model-specific preparation. Its documentation also noted limits around emulator testing and a hardware-specific issue affecting some Adreno layouts. AICore and the associated ML Kit APIs reduced integration work by managing the model service, but the cited Prompt API restricted model choice to Gemini Nano, exposed less sampler control, and targeted selected flagship devices rather than offering broad CPU fallback. These are not simply feature checkboxes. They change which experiments are cheap enough to run and which devices can be represented during development.
Model choice carries a similar trade-off. In Palabrita's final configuration, Gemma 4 E2B was a roughly 2.6 GB premium option for devices with at least 8 GB of RAM, while Qwen3 0.6B was a roughly 614 MB compact option aimed at devices with at least 2 GB. The smaller file lowered the distribution and memory barrier, but it also showed weaker instruction following and more format violations. Supporting both meant maintaining different reliability profiles, not merely offering two download sizes.
That is why model selection is not solved by asking which model is “best.” A team needs to ask best for which responsibility, on which device, with what recovery path, and at what iteration cost. A larger model may improve instruction following, but it also changes startup time, storage, memory pressure, and distribution. A framework with more control may require more engineering. A managed system may reduce setup while narrowing the experiments a product can perform.
When a constraint should move the boundary
The constraint stack is useful because it tells a team when to change the task instead of adding another prompt instruction or immediately buying a larger model. If the problem is formatting, defensive parsing may contain it. If the problem is an invalid hard value, validation and a deterministic source may be stronger. If the problem is repeated context, session rotation may help. If the problem is a two-minute foreground wait, the most effective fix is likely to reduce the amount of generation or remove generation from the critical path.
A larger model remains a possible engineering choice, but it is not a universal escape hatch. It does not automatically eliminate the need for parsing, validation, or fallback. It may not fit the target memory tier, and it cannot reduce a workload that asks for seven long outputs to one short output. The evidence from Palabrita points to a more fundamental response: when the model repeatedly violates a hard product rule, move that rule to a component that can enforce it.
The next case makes this decision visible. Palabrita did not reach a workable design by finding a magic prompt. It reached one by reducing what the model had to own. The failures in the constraint stack are therefore not just obstacles to overcome on the way to a predetermined architecture. They are evidence about where the architecture's boundaries should be drawn.