Overview
FreeToken treats local frontier MoE inference as a whole-machine serving problem: host memory stores authoritative expert banks, GPU slots follow routing, prefill streams full layers while decode handles sparse misses, and q-star splits residual work by measured PCIe and CPU bandwidth. Semantic state reuse and elastic pools extend the design to agent turns and changing hardware. Reported gains apply to tested models, formats, machines, and workloads; bandwidth, compatibility, and state-validity limits remain.
Learning outcomes
- GPU memory is only one constraint: FreeToken also coordinates host memory, PCIe, CPU compute, and retained state.
- Prefill and decode need different movement units: full-layer streaming versus routing-driven cache misses.
- q-star makes residual-miss placement a measured CPU-versus-PCIe decision, not a universal optimum.
- Semantic anchors and elastic pools help only within valid boundaries, compatible formats, and available memory.
Keywords
Preview every chapter · Unlock full text, audio, and video from chapter 4
PlansFAQ
What problem does FreeToken solve for local MoE inference?
FreeToken makes some frontier-scale MoE models usable on personal hardware by coordinating host-resident expert weights, GPU cache space, CPU work, PCIe transfers, and context state instead of treating VRAM as the only limit.
Why does FreeToken treat prefill and decode differently?
FreeToken treats prefill as broad layer movement and decode as a small, changing miss set. It streams complete layers during prefill, then uses routing-driven residency and miss handling during decode.
How does FreeToken support agentic context edits?
FreeToken supports some agentic edits by caching recurrent snapshots at aligned semantic boundaries and reusing matching full-attention prefixes. If an anchor is invalid or evicted, it retreats to an earlier boundary or recomputes.
When is FreeToken's hybrid CPU/GPU execution useful?
Hybrid execution is useful when a supported CPU path and machine-format profile show that host-memory bandwidth can compete with PCIe gathering. Without that evidence or capability, FreeToken falls back to a more conservative mode.