From a Candidate Row to a Running Model

This chapter follows the boundary between a model that llmfit believes can fit and a model that a real runtime can serve. The focus is compatibility: provider identity, weight format, model-name mapping, installed state, downloading, and the interface through which the result is consumed. The practical goal is to diagnose whether a problem is memory, runtime support, availability, or an operational mismatch.

A fit row is not yet a running process. It is an analysis of a model configuration against hardware assumptions. To generate a token, the model still needs a compatible provider, a reachable endpoint or local binary, a model artifact that the provider understands, and a name that can be matched to the catalog entry. llmfit treats those conditions as part of right-sizing rather than assuming that a favorable memory result settles them.

Compatibility is a handshake

The provider layer connects model records to inference software. It supports several kinds of local runtime: Ollama, llama.cpp, MLX, Docker Model Runner, LM Studio, vLLM, Ferrum, and RamaLama. They do not all expose the same interface or the same model naming scheme. Some manage downloads through a daemon, some serve an OpenAI-compatible API, and llama.cpp can work from locally cached GGUF files without requiring a daemon for every operation.

The first operational question is not “which process is listening on this port?” It is “what evidence identifies the process?” Port numbers are convenient defaults, not identities. vLLM and Ferrum can share a default port; llama.cpp and MLX can share a default port as well. A generic OpenAI-compatible response does not prove which runtime produced it.

llmfit therefore uses positive endpoint evidence where possible. OpenAI-compatible servers can identify themselves through an owned_by marker in the model list or, for llama.cpp, a distinctive server header. A llama.cpp server also answers a dedicated properties route that separates it from an MLX server on the same default port. LM Studio is recognized through its native model route and native fields, rather than through a framework-wide header that another server might also emit. Docker Model Runner has its own model-list evidence and a recognizable root response when the model list is empty.

This identity check protects more than a display label. If llmfit imports the wrong model list as though it came from MLX, vLLM, or another provider, installed state and download actions can be attached to the wrong runtime. A false positive can make a user believe a model is ready when the endpoint cannot load that format. Positive identity evidence is therefore a precondition for importing provider state.

Format compatibility is the second part of the handshake. An MLX-specific model is intended for Apple unified-memory systems and an MLX provider. GGUF models belong to the llama.cpp-oriented path and its compatible download sources. AWQ and GPTQ are pre-quantized formats associated with vLLM-compatible GPU execution; they cannot be treated as though llmfit can dynamically turn them into arbitrary GGUF levels. A model may be numerically small enough to fit and still be excluded because the selected backend cannot execute its format.

The contrast between MLX and AWQ/GPTQ makes the point clear. On an Apple Silicon machine, an MLX-native repository may be the natural route because the runtime and memory topology match.