Magnitude: The Local Inference System That Knows Your Machine
Magnitude is an open-source local inference engine built for a practical problem: an AI model that exists is not necessarily a model that a particular computer can run well. This opening chapter asks what Magnitude is actually managing when it profiles hardware, recommends a model, loads it, and connects it to an agent. The answer gives the rest of this series its frame: local inference is not just a different place to send a prompt; it is a decision about artifacts, resources, runtime state, and the client that will use the result.
The hidden question behind “run it locally”
“Run it locally” sounds like a location choice. With a hosted model, most of the physical questions are hidden behind an API: the provider owns the accelerators, memory, model files, serving configuration, and capacity decisions. On a personal computer or workstation, those questions return to the user. A model may fit in storage but not in working memory. It may load only with a shorter context. It may technically generate text but do so too slowly for an agent that repeatedly reads files, calls tools, and waits for the next turn. It may lack the vision, tool-use, structured-output, or reasoning capability that the task requires.
That is why Magnitude does not define hardware support with one universal minimum. Its stated approach is to profile the chip, memory, and available acceleration on the current machine, then assess which model configurations fit and estimate how they will behave. The result is not a generic answer to “what can a computer run?” It is a machine-specific set of choices. A compact model may be the right answer when responsiveness and footprint matter; a larger or mixture-of-experts configuration may be worth the resource cost when coding or reasoning quality matters more. The important point is not that one category always wins. It is that the choice is made against the actual machine and the intended workload.
Magnitude's product promise therefore has several linked verbs: know the machine, recommend a model, tune the serving configuration, and connect the result to the agent already in use. The sequence matters. If recommendation ignores memory, it can select a model that fails at load time. If it ignores context, a configuration may appear attractive while becoming impractical as a conversation grows. If it ignores the client, a fast local server can still be awkward to use. Magnitude treats these as parts of one operating problem rather than unrelated features.
The local setting also changes the privacy boundary. The repository describes a path in which prompts, files, and downloaded models remain on the user's machine, and in which an already downloaded model can run without an internet connection. That is a meaningful advantage for work that should not leave the computer. The claim belongs to the local inference path; it should not be generalized to an unspecified provider or route outside that path. The relevant question is always which path is active and where its model and request data go.
One system, two responsibilities
A useful first map separates Magnitude's product control plane from its native inference data plane. The distinction sounds architectural, but it explains what the user experiences. One part decides what is available, what fits, what is installed, and what is connected. Another part performs the resource-sensitive work of loading a model and generating tokens.
The curated catalog is the first side of that boundary. It describes reviewed model choices: identities, variants, context, capabilities, quality-related metadata, and, where applicable, the material needed for speculative decoding. This is the language of a choice the user can understand. A catalog entry can say that a configuration is dense or mixture-of-experts, identify its source model, and provide an intelligence or fidelity signal. But a catalog description alone does not establish that the configuration fits a particular computer.
ICN—the Inference Control Node—is the native side of the boundary. In the repository's architecture, it owns hardware discovery, artifact inspection, fit assessment, model storage and downloads, active runtime state, and live inference. It works with the exact artifacts and serving profiles rather than reasoning from a friendly display name. Its output becomes evidence for the product layer: whether a model is compatible, what memory it requires, which context and placement were selected, and whether it is resident and ready.
ACN—the local daemon—makes those facts usable by clients. It projects local-model state, exposes model operations, coordinates sessions, and connects the product to the private ICN process. The local provider then binds a selected product model to an ICN-backed chat stream. An agent harness does not need to know how native tensors are placed or how a sequence is admitted into the inference engine. It receives a local model boundary with the semantics needed for its own work.
This division also prevents a subtle category error: assuming that the user has one “model state.” In practice, the service can be starting or ready; the model can be not installed, downloading, installed but unloaded, loading, ready, or failed; and an agent session has its own activity. A command that registers a background service has not necessarily downloaded a model. A model that is installed has not necessarily been loaded into memory. A model that has loaded successfully is not the same thing as a request that has completed.
The daemon is valuable precisely because it keeps these transitions in one place. CLI, desktop, web, and harness clients can share a local runtime instead of each launching a native server, inventing a model inventory, or interpreting hardware differently. ACN's readiness opens application admission; its shutdown path closes that admission and cleans up the inference process. Those details will matter later, but the opening distinction is enough for now: Magnitude is coordinating a resource-sensitive service, not merely wrapping a text-generation binary.
Consider what happens when an agent sends its first request after setup. The agent reaches the local service through its selected model identity. ACN can expose the product state and route the request, while the local provider translates it into the native chat boundary used by ICN. If the model is not resident, the request may encounter a loading transition before token generation begins. Once the native engine is ready, it owns the resource-sensitive work and streams the result back through the same local boundary. The client can therefore observe a meaningful progression without having to understand native memory placement or implement its own model loader.
That sequence also explains why a local service should not maintain several competing truths. The product may remember which model slot the user selected, but the native inference node remains authoritative for the artifact, fit assessment, downloads, and active runtime state. The service can project those facts for a friendly interface without replacing them with a second inventory. When something changes—an installation completes, a model is loaded, memory becomes available, or the inference process stops—the visible state is meaningful only if the layers agree about which fact owns the transition.
How to read a Magnitude claim
The first discipline for understanding Magnitude is to ask what kind of statement is being made. Several numbers and labels can appear next to a model, but they do not carry the same authority.
A catalog value is reviewed metadata. For example, an Intelligence score is an external composite index, not the probability that a model answers a question correctly. An Accuracy or fidelity value concerns how well a local quantized artifact is expected to preserve its source model; it is not factual accuracy. Capabilities are closer to a gate than a preference: if a workflow needs vision or structured output, a small score advantage cannot compensate for the missing feature.
A speed value is a hardware-aware estimate of generation throughput under a serving configuration. It is useful for comparing viable options on the same machine, but it is not a benchmark run of the downloaded model and it does not describe the whole request. Loading, queueing, prompt processing, context depth, concurrency, and system activity can all affect what the user feels. Memory is likewise a runtime requirement, not the size of the file that must be downloaded. Context is the selected serving capacity, not automatically the maximum advertised by the model architecture.
A runtime state is a different kind of fact. “Fits” means that a native planning path found a compatible profile under the observed hardware constraints. It does not mean that every other application has left the same amount of memory free, nor that peak runtime allocation will exactly match a planner estimate. “Ready” means that the relevant runtime boundary has reached readiness; it does not mean that the model will be best for every task.
The distinction becomes practical when two choices pull in opposite directions. A smaller configuration may offer a narrower capability set but leave enough memory for a long context and a responsive agent loop. A larger configuration may offer a stronger intelligence signal yet consume so much working memory that loading it requires closing other applications or choosing a more constrained serving profile. Magnitude's recommendation surface is designed to expose that trade-off rather than hide it behind a single “best” badge. The reader still has to supply the workload: quick iteration, difficult reasoning, coding, vision, and tool use do not impose the same priorities.
The same logic applies over time. A model can be a valid catalog choice before it is assessed, a fitting configuration before it is downloaded, an installed artifact before it is loaded, and a ready runtime before it has served a useful request. Each transition answers a different question. “Is this a known choice?” concerns identity. “Can this profile fit?” concerns resources. “Do I have the bytes?” concerns acquisition. “Can the engine admit it now?” concerns current capacity. “Does it serve this task well?” concerns observation and measurement. Magnitude's value is clearest when those answers remain separate and the interface does not pretend that one implies all the others.
Finally, a measured result belongs to a controlled workload. Magnitude's validation design distinguishes primitive correctness, isolated native performance, complete endpoint behavior, and product-level estimates. A generated answer can demonstrate that one request completed, but it cannot by itself prove parity, scheduler quality, or universal throughput. The system's own evidence model asks whether the model bytes, templates, settings, hardware, cache state, and completed work were comparable before accepting a performance conclusion.
These distinctions lead to a simple first judgment. Begin with the task's required capabilities and the machine's hard resource limits. Among the configurations that pass those gates, weigh context, artifact fidelity, expected intelligence, responsiveness, and any reviewed acceleration. Treat the recommendation as a reasoned starting point, then use runtime observation or controlled measurement when the decision matters. Do not replace that process with a single leaderboard number.
Magnitude's central idea is therefore more precise than “local models made easy.” It is an attempt to make a local model selectable, admissible, observable, and usable by an agent while keeping the boundaries between those claims visible. The next question follows directly: how does the system discover candidates, eliminate configurations that cannot run, and rank the remaining choices before asking the user to download anything?