The Two Things llmfit Must Understand First: Your Machine and the Model
This chapter examines the two inputs that make llmfit a right-sizing tool rather than a generic model directory: the machine that will perform inference and the model configuration that will be placed on it. The practical goal is to distinguish memory pools, model metadata, and runtime constraints before asking whether a particular combination fits.
That distinction matters because “I have 24 GB” is not a complete hardware description, just as “this is a 30B model” is not a complete model description. The first statement leaves open whether the memory is dedicated VRAM or shared by the CPU and GPU. The second leaves open the weight format, context, architecture, and execution runtime. llmfit can only make a defensible comparison when those hidden parts become explicit.
A machine is a topology, not a single memory number
llmfit begins by building a system profile. It reads total and currently available RAM, counts CPU cores, identifies the CPU, and probes for GPUs. The GPU probes differ by platform and vendor: NVIDIA can be discovered through nvidia-smi or a Linux sysfs fallback; AMD through ROCm or DRM/sysfs; Intel Arc through lspci and dedicated-memory information; Apple Silicon through system_profiler; Ascend through npu-smi; and Vulkan can serve as a fallback when vendor-specific tools are absent. These mechanisms are implementation details only insofar as they reveal the important design choice: the tool does not assume that one universal interface describes every machine.
The result is a hardware topology. It includes not only a primary GPU name but also the detected GPUs, their backends, their counts, and the memory information that can be trusted. That information later determines which memory pool llmfit grades and which speed model it can use. A missing probe does not simply remove a line from the display. It can change the confidence of the entire recommendation.
It is useful to separate capacity from present availability. Total RAM describes the installed pool; available RAM describes what the operating system can offer at the moment of analysis. A model may fit when other processes are quiet and become too tight when that pool is occupied. GPU capacity is handled differently: for a discrete card, the fit calculation can use the reported VRAM, while for unified memory the shared pool represents the capacity available to both sides. The values are therefore not interchangeable labels for “free space.” They describe different boundaries and different moments in the machine’s state.
The backend classification is another input, not a performance measurement. CUDA, Metal, ROCm, Vulkan, SYCL, CPU, and Ascend describe the acceleration route llmfit expects to model. The project can use a known GPU name to obtain a more specific bandwidth assumption later, but detecting a CUDA device does not mean that the tool has measured its throughput. Detection establishes what kind of calculation may be appropriate; benchmarking is a separate stage that tests what the live runtime actually delivers.
Consider two machines that each appear to offer 24 GB for a model. On a discrete-GPU system, the 24 GB may be VRAM that the GPU can access directly while the CPU has a separate RAM pool. On a unified-memory system, the CPU and GPU share the same physical pool. The capacity number looks similar, but the execution choices are different. A model that spills from a discrete GPU into system RAM crosses a boundary between pools. On a unified-memory machine there is no separate system-RAM pool to spill into, so llmfit treats the shared memory as one GPU-capable pool rather than inventing a CPU-offload path.
This distinction is especially important for Apple Silicon, but it is not limited to Apple. The implementation also recognizes certain AMD unified-memory APUs and NVIDIA Grace/Tegra paths. In those cases, a small dedicated-memory reading or a BIOS carve-out may not represent the memory that is actually available to the combined workload. llmfit therefore records unified-memory status separately from the raw GPU value. The reader should interpret “VRAM” on such a machine as part of a shared topology, not as an isolated graphics card specification.
The same logic applies to multiple GPUs. When the system contains several cards of the same model, llmfit keeps a per-card description and a count, then uses the aggregate VRAM for fit analysis where supported multi-GPU inference can split a model across cards. This is a useful capacity model, but it is not a universal pooling guarantee. A summed number says that the analysis is willing to consider the cards together; it does not prove that every provider, kernel, interconnect, or model format can use them efficiently. The distinction between “the capacity is available to the fit calculation” and “the runtime will deliver good scaling” is part of the answer, not a footnote.
The primary GPU has a different role from the aggregate pool. llmfit uses the best detected device for the headline name and backend, while the broader list preserves the rest of the topology. A mixed system can therefore contribute more than the primary card’s memory to a fit calculation even though the primary name remains the one a user sees first. This separation prevents a convenient display value from silently replacing the resource pool used for the decision.
Hardware detection also has a failure surface. A driver may be missing, a command may report no VRAM, or a platform may expose a GPU through a path llmfit does not currently support. Android and Termux are an explicit example: CPU and RAM detection usually work, but mobile GPUs are not currently visible through the desktop/server probing interfaces used by the project. A manual memory override can produce GPU-style recommendations in that setting, but it is a scoring workaround, not genuine mobile-GPU detection.
The safe interpretation is therefore conditional: llmfit knows what the probes report, how it classifies the memory topology, and which values it can use for fitting. It does not claim that a failed probe has discovered a hidden accelerator or that aggregate capacity automatically becomes aggregate performance.
A model record is a bundle of constraints
The second input is the model record. llmfit does not reduce a model to its display name and parameter count. A record can contain the declared and raw parameter counts, RAM and VRAM requirements, a default quantization, native context length, use-case category, architecture, attention and key-value-head information, layer count, head dimension, attention layout, MoE topology, capabilities, languages, license, and known download sources. Each field matters only in relation to a later decision.
Parameter count is a useful starting point because it gives the tool a way to estimate the weight footprint. But it is not the same as a complete physical description. The record’s quantization says how the weights are represented; the context length affects the state needed while the model is running; and architecture fields can make the key-value cache estimate more precise. A model with the same headline size as another can therefore require a different amount of memory once its format and architecture are taken into account.
The record also separates facts that are easy to merge incorrectly. A native context length describes the window the model advertises. It does not say that the current machine can hold that window alongside the weights. A model’s recommended_ram value is catalog metadata, not a live measurement of the host’s available pool. An MoE entry can carry both total and active parameter information, which later allows llmfit to treat storage pressure and per-token work differently. Those distinctions keep a name, a model capability, and a current hardware result from being mistaken for one another.
A model’s format is another constraint. GGUF-style entries can participate in the dynamic quantization choices used by llama.cpp-oriented paths. MLX-specific entries are intended for Apple unified-memory systems. AWQ, GPTQ, and AutoRound entries are pre-quantized formats that cannot simply be re-quantized through the same hierarchy. A TTS entry may have a small enough footprint to fit comfortably and still be marked unsupported because it needs a specialized runtime that llmfit does not yet model. In each case, the model can be rejected or routed differently before a memory score is meaningful.
The runtime is therefore a bridge between the machine and the model, not an accessory selected after the score. llmfit’s automatic choices prefer MLX on Apple unified memory, use vLLM for pre-quantized formats or cluster mode, and otherwise use llama.cpp unless the user forces a different runtime. Those choices are based on compatibility signals in both the system and the record. A model that fits in a numerical sense but has no compatible execution path is not an ordinary runnable recommendation.
A model record is not always equally complete. Some architectural values come from upstream configuration metadata; others may be inferred from names or conservative heuristics. The project keeps a distinction between a parameter value it can trust and a fallback it uses to keep analysis working. When a field is missing, the result may still be useful, but the estimate should be read as coarser. The absence of a precise layer count or head layout is not evidence that the model has no such structure; it is evidence that llmfit cannot use that structure directly in this analysis.
Take a catalog record for a model such as Qwen3-30B-A3B. Its name provides a useful identity, but the analyzer needs more than the identity to make a right-sizing decision. The record can carry the model’s total and active parameter information, mark its MoE architecture, describe the available weight format, record its context window, and preserve architecture values that later help estimate memory or per-token work. Those facts should not be compressed into a single “30B” label. Total parameters can matter for storage, active parameters can matter for generation, and the format can determine whether the selected runtime can use the entry at all.
This is also why catalog hygiene affects the quality of the result. A record whose name, declared size, and footprint disagree may remain useful to search or diagnosis, but it should not silently outrank trustworthy records. Likewise, a model discovered in an upstream catalog is not automatically a model that a local provider can download or serve. llmfit carries identity, metadata, and runtime compatibility forward as separate questions so that a failure at one boundary is not misreported as a memory failure at another.
The relationship between the two inputs can be summarized as follows:
This is not a component inventory. It is a dependency model. SystemSpecs tells llmfit what resources and memory boundaries exist. LlmModel tells it what must be placed into those resources. InferenceRuntime determines whether the proposed placement is meaningful for the selected format and backend. ModelFit is the derived object that brings those facts together.
That dependency model explains several otherwise surprising outcomes. If a GPU is detected but its VRAM is unknown, llmfit may fall back toward CPU analysis rather than claim a precise GPU fit. If a model is marked as MLX-specific and the machine is not an Apple unified-memory system, backend compatibility can remove it from ranked results even when its nominal size looks small. If several GPUs are present, the fit calculation may use their combined capacity while the speed estimate still depends on the backend and the model’s ability to split work. If the model record carries only a rough context or architecture description, the result can remain rankable while its memory estimate is less exact.
The two inputs also have different kinds of authority. Hardware detection describes the current host, but “available RAM” can change with other processes. Model metadata describes an upstream artifact, but scraped catalogs can contain incomplete or misleading entries. llmfit’s job is not to erase those differences. It preserves them so later stages can attach notes, choose a fallback, or ask for a manual override instead of quietly turning uncertainty into false precision.
The practical rule is simple: before reading a fit level, identify the machine’s memory topology, the model’s format and architecture, and the runtime that connects them. Only then does a capacity number become interpretable. The next step is to see how those inputs become a memory requirement, where quantization changes the footprint, and why the model’s advertised context is often not the context the machine can actually sustain.