From Local Session to Shared Service

llama.cpp changes character when a single local inference loop becomes a service for other programs or people. This chapter follows what llama-server adds beyond llama-cli, why familiar API labels remain conditional on templates and model behavior, and how multimodal input and tool use expand the request path and the trust boundary. The aim is to choose an interface that fits the workload without treating convenience as proof of compatibility or safety.

A server adds state, scheduling, and a contract with clients

An interactive CLI can keep its assumptions close to one user: load a model, accept a prompt, generate a response, and show timings. A server must make those same capabilities understandable to a client that does not share its process. llama-server provides a C/C++ HTTP server and web UI with REST routes, so another program can send a request and receive a response without knowing how the model is placed on the underlying devices.

That wrapper is not only transport. The server exposes health, streaming, model information, monitoring, and controls for the state that the performance chapter treated as important. A health request can report that the model is still loading, or that the server is ready. A streaming response can deliver generated tokens as they become available instead of making a client wait for completion. Timing fields can report prompt and generation behavior, while slot endpoints expose the state of concurrent work. These interfaces turn internal runtime state into an external service contract.

Concurrency changes the resource problem. A server can use parallel decoding and continuous batching to handle several users, but each active sequence needs context and KV state, and requests now compete for batch capacity and cache space. Prompt caching can reuse a common prefix across requests, which may reduce repeated prompt processing, but it consumes state and can take different batch paths. The documentation warns that cached and uncached paths are not guaranteed to produce bit-for-bit identical logits when batch sizes differ. A service therefore has to decide whether lower repeated work is worth the reproducibility behavior its clients require.

The same server can run in a router mode that loads and unloads models dynamically and forwards requests to the appropriate model instance. That is useful when one process must expose more than one model without keeping every model resident at once. It also adds another state transition: a request may wait for a model to load, and the service must identify which model a client means. Model aliases and model-information routes help create that external identity, but they do not remove the need to manage model files, memory limits, and per-model settings.

API compatibility is another form of boundary management. The server offers direct completion routes and routes modeled on OpenAI’s completions, chat, responses, and embeddings APIs, as well as an Anthropic-compatible Messages route. These interfaces let existing clients speak to a local runtime, but the project deliberately avoids strong claims that the implementation is identical to the upstream APIs. A route can accept a familiar payload while still differing in supported options, response fields, streaming details, or model behavior.

Chat is especially dependent on the model’s formatting contract. A chat template converts roles and messages into the sequence of tokens expected by the model.