What FreeToken Changes—and What It Cannot

FreeToken changes the question of local frontier-model serving from “Does the whole model fit in GPU memory?” to “Can this machine coordinate the weights and state that the workload actually needs?” This final synthesis connects the paper's reported performance and model-scale results to the mechanisms behind them—phase-aware movement, routing-following residency, q-star execution, semantic state reuse, and elastic memory. The resulting judgment is conditional: FreeToken can move the feasible operating point for a personal machine, but only when its measured bandwidth, checkpoint format, attention path, and client workload satisfy the design's assumptions.

flowchart TD W[Agentic workload changes context and routing] --> P[Prefill streams full layers] W --> D[Decode follows a sparse route] P --> H[Shared host expert source and GPU cache] D --> L[LRU retains the current expert working set] L --> Q[q-star divides residual misses by measured bandwidth] H --> Q Q --> X[Graph-compatible CPU and GPU execution] X --> Y[Exact model output] Y --> A[Semantic anchors preserve reusable context state] A --> W M[VRAM, host RAM, driver, and link conditions] --> E[Elastic sizing and conservative fallbacks] E --> P E --> D E --> O[Changed feasible operating point; resource limits remain]

What the reported results demonstrate

The paper's evaluation is strongest when read as a mechanism-to-outcome argument rather than as one universal speed claim. It evaluates four agentic scenarios on six GPU systems and compares FreeToken with actively maintained edge engines. The scenarios include mathematical reasoning, coding through OpenCode, coding through Claude Code's Anthropic-compatible path, and a multi-turn email/calendar agent through OpenClaw. These workloads matter because they mix two costs that a short one-shot benchmark can separate: generated-token decode and repeated prompt processing after tools or context edits.

On an RTX 5090, the paper reports 77–83 tokens per second for Qwen3.6-35B-A3B and 22–25 tokens per second for DeepSeek-V4-Flash. It reports FreeToken at 1.8–2.3 times the strongest baseline for Qwen3.6 and 1.5–1.9 times for DeepSeek-V4-Flash across the tested workloads. Those results are consistent with the architecture's division of labor. The LRU cache reduces the number of expert rows that must move, q-star gives the remaining misses two possible execution destinations, and the graph-compatible path keeps the decision overhead from returning to Python at every layer.

The agentic stability result is more revealing than the peak rate. FreeToken's decode rate stays within 12% of its single-turn mathematical workload across the three agent workloads in the paper. The paper contrasts that with a context-sensitive baseline whose DeepSeek-V4-Flash rate has already fallen 31% on the first coding workload. This does not prove that every agent trajectory will have the same stability. It shows that preserving context state and adapting expert movement can prevent the repeated turns of the tested workloads from turning into an entirely different performance regime.

Time to first token separates the mechanisms further. The paper reports that FreeToken's worst turn stays below 44 seconds in every evaluated cell, while each baseline crosses 150 seconds somewhere; the cited worst cases include 232 seconds for llama.cpp, 179 seconds for Ollama, and 946 seconds for KTransformers. A tail this large is not merely a less attractive average. It can cross the idle watchdog or request-timeout behavior of real clients. The result supports the claim that prefill transfer and context recomputation are availability concerns for agents, not just background costs in a benchmark.

The paper also reports a change in capability tier. On an 8 GB RTX 4060 laptop, a Qwen3.6-35B configuration using the official NVFP4 release reaches 39.3 tokens per second, which the paper compares with the 33-token-per-second median decode rate it cites for Codex production traces. On a 32 GB RTX 5090 desktop, it serves the 284B DeepSeek-V4-Flash configuration interactively. On a single 96 GB RTX PRO 6000, it reports 14.9 tokens per second for GLM-5.2 against 7.3 for llama.cpp. These examples are important because they describe a capability frontier—models that a particular machine can serve at all—not simply an improvement to a model already comfortably resident.

The attribution experiments explain why the results are plausible. With full-layer prefill overlap, the paper reports an 8,192-token chunk completing in 1.19–1.22 seconds on the RTX 5090, close to the time to stream the measured 64.4 GB expert pool at 52.7 GB/s.