From Checkpoint to Running Service

FreeToken's adaptive execution only becomes useful after a checkpoint has been turned into the right physical and operational state. This chapter follows that bridge from stored tensors to a running service: how expert banks reach their final host layout, how platform limits choose between DMA and CPU paths, how cache capacity can change at runtime, and how protocol adapters expose the result to agents. The practical judgment is not which command has the most options, but whether the checkpoint, hardware, lifecycle, and client interface can support the intended serving mode.

Startup is a data-layout and residency decision

A model checkpoint is organized for distribution and loading, not necessarily for repeated expert movement. Its expert tensors may be scattered across shards, arranged in a format that does not match the serving kernel, or stored without the alignment needed for direct I/O. If every server start has to discover those tensors, repack them, allocate a second copy, and then make the copy transferable, startup becomes part of the user-visible cost of local inference.

FreeToken's FreeToken Weight format addresses that physical path. FTW stores dense weights and repacked expert-bank tensors in one logical byte region. Tensor starts, padding, and shard boundaries are aligned to 4,096-byte blocks, so a loader can read aligned ranges directly into the final host banks. The format is divided into bounded shard files for filesystem practicality, but the reader treats them as one logical region. The important change is that the serving layout is established before the engine launches; startup can read the representation the kernels will consume instead of rebuilding it from the original checkpoint.

The distinction between dense weights and expert banks matters. Non-expert weights are loaded onto the GPU and remain part of the model's resident computation. Expert banks are the complete source pool for offload-family execution and stay in host memory. FTW can hold both kinds of tensor, but it does not turn the host bank into a disposable staging area. The bank remains the authority from which GPU slots are filled or CPU experts are computed. That is what lets later cache resizing discard GPU residency without reloading the model's expert source.

When FTW is not available, FreeToken has a slower path that reads the original checkpoint and normalizes its expert pieces into the same bank contract. The loader can choose a parallel direct-I/O reader for checkpoints whose experts are scattered across many small tensors, while a serial path is safer for packed layouts or low-memory hosts. The choice is not a claim that parallel is always faster: a parallel reader can hold a large transient shard buffer on top of the resident banks, so the loader falls back to a more reclaimable serial read when available RAM cannot cover both.

Host residency is the next decision. A GPU transfer needs a host allocation that the CUDA driver can address for asynchronous movement. FreeToken's default host bank starts as a lazy, page-aligned mapping. The loader fills it with real expert data first and calls host registration afterward. Registering empty lazy memory would fault in and zero every page only to overwrite it during the read; pin-after-fill avoids that redundant pass. The populated bank is then ready to feed the PCIe movement path.

That optimization is small in code but large in scale. Expert pools can be tens or hundreds of gigabytes, so a needless memory-fault and zero-fill pass can add a noticeable startup interval before the disk read has even done useful work. The paper describes the same principle as part of fast bootstrap: read directly into the final layout, pin only after the data exists, and avoid a separate warmup whose purpose is merely to make the cache nonempty.