When a Model Fits the GPU but Not the Machine
FreeToken is an edge-native Mixture-of-Experts serving engine for running frontier-scale open-weight models on personal and consumer hardware. The question it makes concrete is not whether a model fits inside a GPU in isolation, but whether a whole machine can move and retain the right weights and state at an acceptable latency. That distinction is the foundation for understanding FreeToken: it gives us a way to judge how local inference works under real agent workloads, rather than treating GPU memory as the only meaningful limit.
The real subject is a serving system, not a smaller GPU
A model checkpoint and a usable service are different things. The checkpoint contains parameters; the service must decide where those parameters live, when they cross an interconnect, which computations run on which processor, and how much context can remain available for the next request. Those decisions become unusually visible with frontier MoE models, because their architecture separates the amount of computation performed for one token from the amount of model state that must be stored somewhere.
FreeToken's defining move is to treat a personal machine as an elastic inference platform. The GPU is still the center of execution: non-expert weights and most of the fast path remain there. But the CPU, host memory, and PCIe link are not treated as inferior leftovers. Host memory can hold the complete routed-expert pool. The GPU can hold a changing subset of those experts. The CPU can compute some expert work directly when moving it would be slower. The interconnect becomes a resource whose measured behavior influences the schedule.
This matters because local frontier inference is often discussed as a capacity question: if the model is larger than the card, the answer appears to be “it cannot run.” FreeToken asks a more precise question: which parts of the model must be resident for correctness, which can be moved or recomputed, and how can the machine keep the most useful working set close to the GPU? The answer is not a single placement chosen at startup. It is a serving design that adapts to the request phase, the routing pattern, the available memory, and the balance between CPU and PCIe bandwidth.
The scale of the movement makes this distinction visible. In the paper's DeepSeek-V4-Flash example, the deployed FP4 configuration has roughly 140 GB of expert weights to move during a broad prefill. The paper reports that this adds about two seconds on an RTX 5090 with a PCIe 5.0 x16 link, about five seconds on RTX 4090- and 3090-class PCIe 4.0 desktops, and ten seconds or more on laptop links commonly running at x8. Those figures are not universal guarantees; they show why a model can be computationally plausible and still feel unusable if the serving path exposes every transfer as GPU idle time. A practical system must therefore optimize the route from host memory to computation, not just the arithmetic inside the expert kernel.
The paper presents this as an accessibility problem. Open weights make a model obtainable, but they do not by themselves make sustained use affordable or practical for an individual or a small team. A local engine therefore has to solve more than weight loading. It has to support a service interface, streaming responses, tool calls, reasoning output, cancellation, and the repeated context changes of coding and other agents. FreeToken's repository reflects that scope: it exposes OpenAI- and Anthropic-compatible endpoints and launch paths for coding agents, while the runtime underneath coordinates model execution and cache state.
Why sparse computation still becomes a memory problem
The basic MoE idea is easy to summarize. A layer stores many experts, but a router sends each token through only a small top-k subset. If a layer has 256 routed experts and a token activates six, the arithmetic for that token is much smaller than evaluating all 256. The paper uses DeepSeek-V4-Flash to illustrate the gap: it describes a 284-billion-parameter model with about 13 billion active parameters and six of 256 routed experts participating in each layer.
But “active” is not the same as “resident.” The inactive experts still exist in the checkpoint, and a future token may route to them. At deployed precision, the active computation may fit in a 32 GB card while the complete expert pool does not. The machine must consequently maintain a second level of storage for weights that are valid but not currently in GPU memory. A miss then has a cost: the expert must either cross PCIe into a GPU cache or be evaluated where it already lives in host memory.
The difference between prefill and decode makes the problem sharper. During decode, one new token usually activates only a small collection of experts in each layer. Consecutive tokens may also reuse many of the same experts, so a cache can follow a relatively small working set. Yet cold starts, route changes, and limited cache capacity leave misses that must be served on every step. A policy that fixes placement once, before the generation pattern is known, can leave the GPU or the link idle while too much work falls back to the CPU.
During prefill, the input may contain thousands of tokens. Each token is still sparse, but the union of their routes across a layer can cover nearly the entire expert set. In that phase, fetching one expert only when a route first appears exposes a long sequence of transfers. The practical problem looks less like a small sparse lookup and more like streaming a large fraction of the model through the host-to-device link. That movement directly affects time to first token, which is especially costly when an agent starts a new turn after a tool call or a context edit.
Agentic use adds a second form of repetition. A coding or tool-using agent does not simply prompt a model once and stop. It may generate a tool call, receive an output, rewrite or compact part of the conversation, and submit a new prompt that preserves some earlier context but not all of it. Hybrid-attention and recurrent architectures make this consequential: some layers retain history as KV entries or as a recurrent state, and a later edit can invalidate a checkpoint taken after the edited block. Recomputing an entire long prefix wastes work even when most of the conversation is unchanged.
The location of an edit matters as much as its size. If an agent removes a thinking block or replaces an old tool result, the tokens before that block may still be exactly the same, while state after it no longer describes the new conversation. A serving engine that stores only an undifferentiated “last context” has to choose between discarding too much and risking reuse of invalid state. FreeToken's later semantic-anchor design starts from the narrower observation that agent edits tend to occur at marked boundaries. The opening need not assume that every edit is predictable; it is enough to see why a reusable boundary can be more valuable than a uniformly spaced checkpoint.
The resulting challenge has three coupled dimensions. First, the system must move experts efficiently when a prompt activates broadly. Second, it must make sensible decisions about the smaller residual misses during decode. Third, it must retain the kind of context state that is most likely to survive the next agent edit. A design that improves only one dimension can still feel slow: fast decode does not repair repeated prefill, and a large context cache does not help if expert movement leaves the GPU waiting.
One machine, two regimes, one adaptive question
This is why FreeToken should not be reduced to “an MoE offloader.” Offloading describes where the complete expert pool can wait; it does not describe how a serving engine uses that pool. FreeToken's central theme is adaptive orchestration: the same host-resident weights support different movement and execution paths as the request changes.
For prefill, the important question is how to keep expert transfer from becoming a serial pause between layers. FreeToken's answer is full-layer double buffering: while one layer is computed, the next layer's experts can be transferred in the background. That is a response to broad prefill activation, not a universal rule for all inference. For decode, the important question is different: which misses should fill a GPU cache for future reuse, and which should be computed on the CPU now? That answer depends on the machine's actual host-memory and PCIe behavior, not merely on the model's parameter count.
For agentic contexts, the question shifts again: which state should be preserved at a semantic boundary so that a later edit can resume rather than restart? For a shared desktop, another question appears: what happens when a browser, a game, or a growing KV cache changes the memory available to the engine? FreeToken addresses these pressures with semantic-aware state caching and runtime-adjustable pools, but those mechanisms do not abolish constraints. They make the constraints explicit and give the runtime more than one way to respond.
The useful judgment for a local deployment is therefore multidimensional. Parameter count tells us how large the complete model is, but not how much arithmetic one token performs. VRAM tells us how large a fast resident cache can be, but not how quickly host memory can feed it. PCIe bandwidth tells us how fast weights can move, but not whether direct CPU execution would use otherwise idle host bandwidth. Context length tells us the theoretical ceiling, but not how much of a conversation can be reused after an agent edits it.
The paper's evaluation reinforces why specifications alone are insufficient: its bandwidth table uses measurements taken on the deployed tensor shapes, and the same GPU silicon behaves differently with different host memory systems. FreeToken's deployment choice is therefore empirical by design. A machine with a narrow laptop link and strong host memory may favor a different split of work than a desktop with a faster link but a weaker CPU memory subsystem. The relevant comparison is the behavior of the complete serving path under the intended workload.
The rest of this series follows those relationships through FreeToken's operating model. It first separates the prefill and decode paths, then examines the shared expert cache that connects them. From there, the q-star policy turns bandwidth into a concrete decode decision; semantic anchors extend the same resource logic to recurrent and windowed context; and storage, fallbacks, runtime reconfiguration, and service interfaces show what is required to make the design deployable. The central question stays the same throughout: how can FreeToken compose the resources already present in a personal machine so that a frontier MoE model is not merely loadable, but usable?