The Memory Game Behind Every Inference
Ollama’s scheduler is the part of the runtime that turns a model request into a resource decision: run it here, reuse what is already loaded, wait, unload something else, or fail. This chapter explains that mechanism so the reader can distinguish a model’s stored size from its live memory cost and understand why context length, parallel requests, GPU placement, and keep-alive settings change the result. The goal is not to memorize tuning variables, but to read the conditions under which Ollama can keep inference available.
A model must fit as a workload, not just as a file
A model’s weight file is only the starting point for a load decision. When Ollama prepares a local runner, it reads model metadata and estimates the memory required for the selected context and generation settings. The estimate must account for the weights, the context or key/value cache used to remember tokens, batch-related allocations, optional multimodal projector material, and the native backend’s overhead. A file can therefore fit on disk and still fail at runtime, or fit for a short prompt but not for the larger context an agent or coding task needs.
Context length is the clearest example. It is the maximum number of tokens the model can access in memory while generating the next token. Raising it makes longer conversations and documents possible, but it raises memory demand. The scheduler uses an effective context that respects both the requested value and the model’s trained context limit. When several requests are processed in parallel, the effective context grows with the number of parallel requests. A service configured for one request at a time and a service configured for four requests do not have the same memory profile even when they use the same model and nominal context setting.
Generation batching adds another pressure. Ollama can select an automatic batch size based on context, predicted memory, available memory, Flash Attention support, and the detected devices. Larger contexts may lead to a more constrained batch, while additional headroom can allow a larger one. This is a performance choice inside the load decision, not a universal speed setting. A batch that improves throughput when memory is abundant can make a model fail to load when another model is already resident.
Multimodal models introduce another component into the budget. The text model may be selected successfully, yet its image projector can require additional memory. Ollama’s native runner tracks projector material and can disable projector GPU offload when the devices have limited memory, when only part of the text model is being offloaded, or when the request is CPU-only. Some architectures have the opposite requirement: a projector must remain on the GPU to preserve correct image embeddings. These are model- and backend-specific conditions, so “the model fits” is not a sufficient statement for a vision workload.
The scheduler first considers what devices are available and how much memory remains after accounting for loaded runners. On a multi-GPU system, Ollama prefers a single GPU when the predicted workload can fit there. Keeping the model on one device usually avoids unnecessary transfers across the PCI bus during inference.