Overview

llama.cpp keeps the model, context, and batch interface common across CPU and accelerator paths, while ggml backends determine which operations can run and at what cost. Build availability is not full model support. Partial offload, layer-based pipeline splits, and experimental tensor splits solve different capacity, throughput, and latency problems, so placement must be checked against the graph, interconnect, memory, and workload.

Learning outcomes

A compiled backend makes a device eligible; the model graph and operations table still determine whether the workload is covered.
Partial offload and multi-GPU placement can solve capacity, but CPU/device transfers and interconnect costs can change performance.
Layer splitting and tensor splitting serve different goals, and tensor split remains experimental with architecture and auto-fit limits.

Keywords

llama.cppBackendsPlacementOffloadMulti-GPUInterconnect

One Inference Core, Many Machines

llama.cpp is designed to carry a common inference path across different processors and accelerators, but it does not make hardware differences disappear. This chapter examines where the common responsibilities end, where backend-specific execution begins, and how to choose device placement when the goal is capacity, throughput, or latency. The practical result is a way to distinguish “the backend is available” from “this model workload is supported and appropriately placed.”

A common contract, with hardware at the edge

The first useful distinction is between what an inference program means to do and how a machine performs it. At the application boundary, llama-cli and llama-server need to load a model, accept input, maintain a sequence, and expose generated results. The public llama.cpp API represents that common responsibility through a model, a context, and batches of work.

The model is the long-lived description of the learned system: its vocabulary, metadata, architecture, and weights. The context is the active runtime around that model. It carries the capacity and changing state required to process a conversation or another sequence. A batch crosses the context boundary into computation. It can contain token or embedding input, positions, sequence identities, and flags identifying which results should be returned. This lets one API describe both a single request and work involving multiple sequences without turning every hardware path into a separate application design.

The distinction also assigns ownership. The application decides what input to submit and what result it needs. The context owns the active inference state and the settings that shape its working capacity. The lower execution layer receives operations produced from that state and must place them on devices that can execute them. When a request is decoded, the context advances; when the application reads logits or embeddings, it consumes the result of that computation. The hardware backend should not have to understand whether a token came from a terminal or an HTTP client. It needs to execute the operation graph that the common layer presents.

That graph is where ggml enters the picture. ggml provides tensor operations and backend implementations. llama.cpp supplies the model-specific and context-specific behavior that assembles those operations into an inference path. A backend then provides the machinery for a CPU or accelerator to run supported operations. This arrangement is an abstraction, not a claim that every operation has the same implementation or cost on every device.

The build process and the runtime process make different decisions. A build can enable CPU, Metal, CUDA, HIP, Vulkan, SYCL, CANN, OpenCL, WebGPU, OpenVINO, and other backend families documented by the project. At runtime, a user can select devices, control accelerator offload, or disable offload entirely. Multiple backends can be built together, and the project documents dynamically loaded backends as a way for the same binary to run on machines with different GPUs. The build makes a capability available; runtime placement chooses whether that capability is used for this invocation.

This separation is valuable because it keeps the application model stable while the machine changes.

Published by DeepMox

FAQ

What is the difference between a llama.cpp build and runtime placement?

A build enables backend families, while runtime placement selects devices, offload, and work distribution for a particular invocation.

When is partial offload useful in llama.cpp?

Partial offload is useful when the full model does not fit in accelerator memory and keeping some layers on the CPU makes it runnable, though transfers can slow it.

How do layer split and tensor split differ?

Layer split assigns contiguous layer slices and usually limits transfers; tensor split divides weights and KV state within layers, increasing communication and offering an experimental latency-oriented path.

Previous10 minllama.cpp Quantization Trade-offs

Next11 minllama.cpp Memory and Token Speed

llama.cpp Backends and Placement

One Inference Core, Many Machines

A common contract, with hardware at the edge

FAQ