What Does It Mean for an LLM to Fit?
Suppose a model is advertised as “30B,” a computer has 16 GB of graphics memory, and a download page offers several quantized versions. Is that enough information to decide whether the model is a good local choice? Not quite. The model may load only after its weights are compressed. Its advertised context window may be much larger than the memory can sustain. A GPU that has enough capacity may still be too slow for the intended task. And a model that looks attractive in a general list may be a poor choice for coding, reasoning, or multimodal work.
This series studies AlexsJones/llmfit, an open-source tool built to answer that practical question: which open models can a particular machine run comfortably, through which runtime, and with what evidence behind the answer? The important word is “particular.” llmfit does not treat a model as having one universal hardware requirement or one universal speed. It turns the model and the machine into a conditional comparison.
That makes llmfit more than a catalog and more than a calculator. Its central idea is hardware-aware right-sizing: choose a model configuration that matches the memory topology, acceleration backend, workload, and evidence available on the machine in front of you. The result is not a permanent verdict about the model. It is a reasoned answer under stated conditions.
The problem hidden by model lists
A model list usually makes the model the only visible object. It shows a name, a parameter count, perhaps a context length and a recommended amount of memory. That presentation is convenient, but it hides the decisions that determine whether local inference will be useful.
Parameter count is the first example. For a dense model, more parameters usually mean more weight data to keep in memory. But the number does not tell us whether that data is stored at full precision or in a compressed format. It does not tell us how much memory the key-value cache will consume as the conversation grows. Nor does it tell us whether the model will stay on the GPU, spill into system RAM, or run entirely on the CPU.
The hardware side is equally important. Two machines can both be described as having “24 GB available,” while one has a discrete GPU with its own VRAM and the other has a unified memory pool shared by the CPU and GPU. Several GPUs may be able to divide a model between them, but that does not mean every runtime will combine them in the same way. A detected GPU may have a known memory bandwidth, or it may be an unfamiliar device for which llmfit must fall back to a rougher estimate.
Even “fits” has more than one meaning. A model may fit in memory but produce too few tokens per second for an interactive application. It may generate quickly but lack the context needed for a long coding session. It may be excellent at reasoning but not the best choice for a chat workload. It may be compatible with one local runtime and unavailable in another.
llmfit keeps these questions connected without collapsing them into one hidden assumption. A fit result describes memory headroom and an execution path. A speed estimate describes expected throughput. A composite score combines quality, speed, fit, and context according to a use case. A confidence field says whether the throughput is measured, calibrated, estimated, or unavailable. Reading the result means keeping those dimensions separate long enough to understand how they interact.
This is why “fit” belongs to a configuration, not to a model in isolation. A model may be a sensible choice at one context length and quantization but not at another. It may be a good local coding assistant on a discrete GPU and a poor choice when the same weights spill into system RAM. A smaller model may be preferable when responsiveness matters, while a larger one may be worth the extra memory when its quality advantage matters more. The right answer depends on what the user is trying to do and what the machine can sustain at the same time.
The distinction also prevents a common category error: treating a performance estimate as a quality judgment. Tokens per second says how quickly output is generated under the modeled conditions. It does not establish whether the output is accurate, follows instructions, handles code well, or supports the required modality. Conversely, a model can be capable enough for a task but still be impractical if every response arrives too slowly. llmfit’s value is in making these trade-offs visible so that a user can choose deliberately rather than letting one convenient number stand in for the whole decision.
Right-sizing is a chain of evidence
The project’s design can be understood as a chain rather than a single formula. It begins with observations about the machine: RAM, CPU cores, GPUs, VRAM, unified-memory status, and acceleration backend. It then reads model information such as parameter count, weight format, native context, architecture, capabilities, and—when relevant—the number of experts that are active for each token.
Those inputs support a memory analysis. The tool estimates the weight footprint, adds context-dependent state and runtime overhead, and searches for the highest-quality quantization that fits the selected memory pool. The same model can therefore receive a different best quantization on a different machine. A long native context can also become a much smaller usable context when the weights leave too little room for the cache.
Next comes the execution question. llmfit distinguishes a model fully resident on the GPU from one using expert offload, partial GPU offload, CPU-only execution, or tensor parallelism. These are not merely labels for display. They describe where the model’s work and data live, and they change the performance that a memory fit can realistically deliver.
Only then does the tool produce a ranked result. The ranking is multi-dimensional and use-case dependent. A coding request can give more importance to a family’s coding alignment; a reasoning request can give more weight to quality; a chat request can care more about speed. This prevents “largest model wins” from becoming the default decision rule.
The final link is evidence from reality. llmfit can compare its estimate with benchmark results from the user’s own machine, matching community hardware, or a supplemental benchmark source. It records how much confidence to place in the attached throughput number and exposes the inputs behind a formula estimate. A user can download and serve a model, run live measurements, and use those results to improve the estimates for that hardware. The decision is therefore allowed to become more empirical over time.
This chain also explains why a single headline number is inadequate. The same 42 tokens per second can mean a measured result on the user’s machine, a community result on an identical configuration, a locally calibrated formula, or an uncalibrated guess. The number is not fully interpretable until its provenance is known.
One core analysis, several ways to use it
llmfit presents this reasoning through several interfaces, but the project’s architecture gives them a common center. The interactive terminal interface lets a reader browse models, filter by fit or use case, inspect details, simulate hardware, plan a configuration, download a model, and open benchmark views. The classic command-line interface exposes tables and JSON for scripts. The web dashboard and REST API expose node-local hardware and model-fit data to dashboards, schedulers, and other tools. An agent can use the MCP interface to request the same kind of analysis.
The interface changes the way a result is consumed, not the basic question being answered. A person may look at a row and ask why a model is marked “Good.” A script may request the top runnable coding models. A scheduler may ask several nodes for candidates. In each case, the useful result includes more than the model name: it includes the memory path, context assumptions, runtime, estimated or measured speed, and the conditions that limit the conclusion.
That shared center is also a boundary. llmfit is not a universal performance oracle. Hardware detection depends on drivers and platform interfaces, and some environments expose no useful GPU information. Catalog metadata can be incomplete or need correction. Formula estimates simplify the behavior of real inference kernels. A benchmark measures a particular runtime, quantization, context, prompt pattern, and hardware configuration; it does not automatically generalize to every other setup. Task quality requires its own tests rather than being inferred from throughput.
The project makes these limits useful instead of treating them as an embarrassment. It can show a missing prefill estimate rather than inventing one. It can distinguish a native context window from the context that actually fits. It can keep an unrankable catalog entry searchable while excluding it from recommendations when its metadata would make ranking misleading. It can preserve a measured number’s source and match it only to comparable hardware.
So the practical meaning of “fit” is not simply that a file can be loaded. It is that a model configuration has a plausible memory path, a usable context, an appropriate runtime, an acceptable performance expectation, and enough evidence for the decision being made. Sometimes the result points to a model. Sometimes it points to a lower quantization, a shorter context, a different runtime, a live benchmark, or more memory. The answer is allowed to change when one of those conditions changes.
The next question follows directly from this framing: what exactly must llmfit know about the machine and the model before it can make that comparison? Understanding those two inputs—especially the difference between discrete VRAM, system RAM, and unified memory—is the foundation for reading every later estimate correctly.