Why Local Inference Is a Systems Problem
llama.cpp is a C/C++ inference system for running large language and vision-language models across local, cloud, CPU, GPU, and mixed hardware. This opening asks a practical question: why can obtaining a model still leave so much work before a useful application exists? The answer gives this series its organizing perspective: llama.cpp makes the trade-offs among the model artifact, hardware path, resource budget, and application interface visible enough to tune and verify.
A model is not yet an application
A language model is often described as if it were a self-contained capability. Download the weights, provide a prompt, and receive an answer. That description is useful at a high level, but it hides the conditions that determine whether the answer arrives at all, arrives quickly enough, or arrives in a form an application can use.
The weights must be stored in a representation the runtime can load. They must fit somewhere: perhaps in system memory, perhaps in a GPU, or partly in both. The computations must be covered by the selected execution backend. A conversation must be converted into the token sequence the particular model expects, and the generated result may need to satisfy a protocol, a JSON schema, or a tool-call format. Each condition belongs to a different boundary, but a failure at any one of them can look to a user like “the model does not work.”
llama.cpp addresses this whole path. Its stated goal is LLM and VLM inference with minimal setup and high performance across a wide range of hardware, locally and in the cloud. The project is built on ggml, which supplies tensor operations and hardware backends, but llama.cpp is not just a collection of kernels. It also provides model loading, context management, tokenization, sampling, conversion and quantization tools, command-line programs, and an HTTP server. The useful subject is therefore the system that makes inference possible on a machine, not only the neural network being executed.
This distinction matters whenever resources are limited or hardware is varied. A model that fits in a workstation’s memory may not fit in a laptop’s GPU. A build that includes a CUDA backend may still have incomplete support for a particular operation or model architecture. A quantized file may be small enough to load but may alter quality in a task where exact formatting or reliable tool use matters. A server that can answer one request may need different scheduling and memory choices when several people use it at once.
The same model name can consequently describe different experiences. The difference is not necessarily a change in intelligence. It may be a change in representation, placement, context length, batching, thread scheduling, or interface behavior. llama.cpp’s importance is easiest to see in this gap between a model as an artifact and inference as an operating system for that artifact.
The layers that make inference usable
The project’s boundaries are easier to understand if one request is followed from the outside in. At the top is an application: a terminal session using llama-cli, a web UI, or a client sending an HTTP request to llama-server. That application supplies text or other input and eventually consumes generated text, embeddings, structured output, or a tool call.
The tools call the libllama interface. Its public C API separates a loaded model from a runtime context. The model contains the loaded architecture, vocabulary, metadata, and weights; the context holds the working state needed to process batches and continue sequences. A batch can contain tokens, positions, sequence identities, and flags saying which results should be returned. The API can submit that batch for encoding or decoding, then expose logits or embeddings and manage saved state. These distinctions are not mere software organization. They identify who owns persistent model data, who owns changing conversation state, and where a request enters computation.
Below that interface, llama.cpp constructs the operations needed for the model. ggml represents tensor computations and connects them to backend implementations. The backends may target ordinary CPU instructions, Apple Metal, NVIDIA CUDA, AMD HIP, Vulkan, SYCL, and several other accelerator or platform families. Build settings decide which families are available; runtime settings decide which devices receive work and how many model layers are placed in accelerator memory. The project can also combine backends or load backend libraries dynamically, allowing the same general program to be used on machines with different hardware.
That architecture creates a useful separation without promising that all machines behave alike. The common layer describes what inference means: load a model, maintain a context, process a batch, and produce results. The backend describes how particular operations run on a particular device. The operations support table makes the boundary explicit by marking operations as fully supported, partially supported, or unsupported for each backend. A successful build is therefore only the beginning of verification. The selected model graph, operation coverage, driver, and placement still determine the result.
The server adds another boundary rather than replacing these layers. It wraps the same inference capabilities in health checks, streaming responses, API routes, monitoring, slots for concurrent work, and model-loading logic. This is why llama-server can offer OpenAI-compatible and Anthropic-compatible routes while the underlying model remains local. Compatibility is a practical interface claim, not proof that every model, template, parameter, or client behavior is identical to a hosted service. The server documentation itself qualifies the compatibility promise and ties chat behavior to a suitable template.
The layered view also explains why the project can support both a minimal local session and a more elaborate service. A person may use a CLI with one model and one sequence. A product may use the server with continuous batching, multiple slots, prompt-cache reuse, embeddings, or multimodal input. The underlying boundaries remain, but the resource and correctness obligations become larger as more of them are shared.
Four decisions define the practical result
The system map can be reduced to four connected decisions. The first is the artifact decision: which model representation will be loaded, with what metadata, quantization, and optional components. llama.cpp requires the GGUF format in its normal runtime path. Conversion from another model format and quantization are separate stages, and a vision or audio model may need a separate projector or encoder file. This means that “which model are we using?” includes more than a parameter count. It includes the runtime representation and the information that tells the runtime how to interpret the model.
The second is the hardware decision. The project supports many CPU and accelerator paths, but portability is an abstraction plus a capability matrix. The question is not simply whether a machine has a GPU. It is whether the chosen build and backend can execute the relevant operations, whether the artifact fits the available memory, and whether moving work between devices helps or hurts. Partial GPU offload can make a model usable when it exceeds one device’s memory. Multiple GPUs can solve a capacity problem, increase throughput, or reduce latency, but those goals lead to different split strategies and different communication costs.
The third is the resource decision. Model weights are only one part of the budget. Context length and concurrent sequences consume key-value cache memory. Prompt processing and one-token generation have different computational shapes, so batch size and thread count affect them differently. A configuration can load successfully and still be too slow for interactive use, or work for one conversation and fail when a server allocates several slots. Resource-aware inference means treating these settings as a connected operating budget rather than as isolated performance flags.
The fourth is the interface decision. A local terminal session, a streaming HTTP endpoint, an embeddings service, a multimodal request, and a tool-using agent do not impose the same requirements. Chat templates affect whether messages become the input structure the model expects. Structured output can constrain form, but it does not guarantee correct content. Tool use may depend on a native format or a generic fallback, and the project warns that extreme KV-cache quantization can damage tool-calling performance. When a service is exposed beyond one trusted user, API keys, CORS, TLS, metrics, and privilege boundaries become part of whether the deployment is acceptable.
These decisions reinforce one another. A smaller quantized artifact may fit a device but require a quality test. A backend may offer excellent arithmetic throughput but be a poor choice if the model graph lacks full operation coverage. A larger context may support the application’s conversations but consume the memory that would otherwise allow more concurrent users. A server may expose a familiar API but still need a model-specific template. None of these trade-offs can be settled by a single headline such as “runs locally” or “supports GPUs.”
That is the practical judgment this series supports. It is not a promise that llama.cpp erases hardware differences or makes every model interchangeable. It is a way to inspect the chain from artifact to computation to application, ask which boundary owns the problem, and measure the result under the workload that matters. The next boundary is the model artifact itself: how a training checkpoint becomes GGUF, how metadata participates in loading and conversation, and why a sidecar component can be as important as the weights it accompanies.