From Bandwidth to Tokens per Second

This chapter explains how llmfit turns a model configuration into a speed estimate and then places that estimate inside a broader score. The key judgment is not whether a displayed tok/s figure looks precise, but what it assumes: memory bandwidth for single-request decode, fp16 compute for prompt processing, run-mode penalties, and task-specific weights for ranking. Keeping those models separate makes the result easier to understand and easier to challenge.

A model can fit in memory and still respond too slowly. It can generate quickly while offering the wrong quality or context for the task. It can also receive a speed number from a formula when no matching measurement exists. llmfit addresses these cases by treating throughput, fit, quality, and context as related outputs with different evidence bases rather than as interchangeable meanings of “good.”

Decode speed starts with memory traffic

For ordinary single-request generation, the first-order bottleneck is often memory bandwidth. Each new token requires the inference engine to access a large amount of model weight data. If the hardware can move more bytes per second, it can potentially produce tokens faster; if the model’s representation occupies fewer bytes, the same bandwidth can cover more token steps.

llmfit expresses that intuition as a roofline estimate. In simplified form, raw throughput is the available GPU memory bandwidth divided by the model’s weight size in memory. The project then multiplies that ceiling by an efficiency factor and a run-mode factor. The default efficiency is 0.55. It absorbs costs that the simple division leaves out, including kernel launch and scheduling overhead, KV-cache reads, and inefficiency in the memory controller at high utilization.

The formula is useful because it names a physical constraint instead of assigning one universal speed constant to every GPU. Two accelerators can offer similar VRAM capacity but different memory bandwidth. They may therefore receive similar fit verdicts and different decode estimates. Conversely, a larger model on a faster-memory device can sometimes approach the throughput of a smaller model on slower hardware. Capacity answers whether the weights can reside; bandwidth helps answer how quickly the resident weights can be used.

The bandwidth input has an explicit resolution order. A user or hardware profile can supply a GPU-bandwidth override. If there is no override, llmfit looks up the detected GPU name in its bandwidth table. If the name is not recognized, the estimator falls back to a per-backend constant. The project documents fallback values for CUDA, Metal, ROCm, SYCL, CPU ARM, CPU x86, and Ascend. These constants keep an unfamiliar device analyzable, but they are less specific than a measured or recognized bandwidth.

This distinction is visible in the estimate basis attached to a fit. A row can report that it used a GPU-bandwidth roofline, a backend constant, or a CPU constant, along with the assumed efficiency and context. A reader can then tell whether the value is grounded in a named hardware bandwidth or in a broader heuristic. The tool does not have to hide uncertainty in order to produce a sortable table.

The run mode changes the simple roofline as well.