When the Estimate Meets the Machine

This chapter follows how llmfit replaces a formula-only expectation with evidence from a live runtime. The focus is the benchmark lifecycle and the provenance rules around it: what is measured, how timing differs by provider, which runs are comparable, and how a result becomes local evidence, community evidence, calibration, or remains only an estimate. The practical outcome is a way to decide which number to trust and how far that trust should extend.

A throughput estimate and a benchmark result may use the same unit—tokens per second—but they are not the same kind of statement. An estimate starts from assumptions about hardware, model bytes, bandwidth, and run mode. A benchmark observes one model, one runtime, one configuration, and one workload on a particular machine. The observation can correct the estimate for that setting, but it does not automatically become a law for every machine or provider.

A benchmark is a protocol, not a magic number

The project’s documented workflow begins with a model that has been selected and downloaded. The user serves it through a live provider, refreshes the installed-model state, and starts the benchmark from the interface. This sequence matters because llmfit is not benchmarking an abstract model record. It is testing an actual serving path: a provider, a loaded model, a weight format, a context, and a machine.

Before collecting results, the benchmark performs a warmup request. Loading a model, initializing kernels, filling caches, or waking a provider can make the first request unlike the steady-state requests that follow. The warmup is not counted as one of the measured runs. In the TUI flow, llmfit then performs three real inference passes. The repeated passes do not eliminate every source of variation, but they provide more information than a single first response and allow the result to record an average as well as minimum and maximum throughput.

The measured fields depend on what the provider exposes. For Ollama, llmfit sends a generation request and can use the service’s native evaluation durations. That gives it a provider-specific time to first token and generation timing. For OpenAI-compatible providers such as vLLM, MLX, Ferrum, or a llama.cpp server, the current benchmark path sends a chat-completion request and measures wall-clock request time. It records output tokens per second from that elapsed time, but it does not claim a measured TTFT because the non-streaming path does not expose the point at which the first token arrived.

This difference is not a cosmetic schema detail. TTFT describes the latency before the first output token, while total wall-clock time includes prompt processing, generation, and any other request overhead. A provider that reports native evaluation durations and a provider that is timed around a complete HTTP request are not measuring identical intervals. The result can still be useful, but comparing the fields as though they came from one instrument would create false precision.

The benchmark runner uses a small set of prompts that vary in task and length, including explanation, code generation, comparison, and code review. That makes the result more representative than a single fixed phrase, but it still does not turn the benchmark into a complete workload simulation. A production application may use longer prompts, different output lengths, concurrency, batching, tool calls, or a different sampling configuration.