The Meter Is Not the Price Sheet
For the cloud Opus and private quantized GLM configurations, the immediate question is not what one input token costs in isolation, but how a multi-turn coding session turns context, output, and capacity into a bill. Understanding that mechanism lets a decision-maker compare a cache-optimized API with shared or dedicated private inference without confusing an effective unit price with total spend.
A coding-agent request is more than “input plus output”
A conventional API comparison begins with two numbers: the price of input tokens and the price of output tokens. That is a useful starting point, but it is incomplete for an autonomous coding agent. The agent does not send a single short prompt. It repeatedly carries forward system instructions, tool definitions, repository context, specifications, earlier responses, and the results of commands it has run. A later turn often resembles the previous turn at the front and differs mainly in the newest request or tool output.
That distinction creates several economically different token categories. Uncached input is prompt material that the service must process at the ordinary input rate. Cache creation, sometimes called a cache write, is a stable prefix stored so that later requests can reuse it. Cache reads are subsequent uses of that stored prefix. Output tokens are the model’s generated reasoning and response material, including the content that drives another tool call or explains an edit. The invoice depends on the mix, not simply on the total number of tokens appearing in requests.
In the enterprise coding study, the Opus period generated about 15.331 billion total tokens. More than 15.202 billion were cache-read input tokens, producing a reported 99.3% cache-hit rate; only a small fraction of the input was treated as ordinary incremental or newly created context. The local GLM period also had repetitive prefixes: vLLM reported an 86.4% prefix-cache hit rate over about 905.6 million total tokens. The two cache systems are not billed in the same way—the API has token prices while the private deployment is accounted for in hardware hours—but both observations show the same workload property. Agent sessions repeatedly revisit a large, relatively stable context.
The useful accounting boundary is between what remains stable and what changes on each turn. System rules, tool descriptions, repository material, and earlier history may form the reusable portion; a new instruction, test failure, or file result may extend or alter the request. The service must still process the changing material and still generate an answer. A cache hit therefore lowers the cost or work associated with the reusable prefix; it does not make the whole request free, shorten the output, or guarantee that the agent will need fewer rounds to finish a task.
This also explains why a hit-rate percentage cannot be read directly as a savings percentage. The hit rate counts eligible prompt tokens, while the invoice weights cache reads, cache writes, uncached input, and output at different prices. A workload with many inexpensive cache reads can still have a large bill if its output is verbose or its total context volume is enormous. Conversely, a lower hit rate can matter less if sessions are short. The correct question is not “Did the cache work?” in the abstract, but “Which token category did the cache move, by how much, and what other categories grew alongside it?”
The mechanism is easiest to see as a choice between two deployment paths. A real request takes one path, not both; the diagram places the paths side by side to show how the accounting basis changes.
The cache is not magic compression of the whole workload. It is reuse of a portion of the computation and billing treatment when the prefix remains eligible and stable. The changing suffix still has to be processed. The model still has to generate its output. If a tool result changes the context or the serving system cannot reuse the prefix, the expected savings may not appear. Cache-hit rate is therefore a workload measurement that must be collected, not a property to assume from the fact that an agent is being used.
The reported Opus prices make the distinction concrete: $5.00 per million uncached input tokens, $6.25 per million cache-write tokens, $0.50 per million cache-read tokens, and $25.00 per million output tokens. Cache reads are not free, and cache creation is not priced like a normal read. But moving most repeated input into the cache-read category changes the weighted average dramatically. In the study’s accounting, the Opus bill consisted of only a few dollars of ordinary input, hundreds of dollars of cache creation, thousands of dollars of cache reads, and roughly five hundred dollars of output, for a total of $8,785.21.
Without caching, applying the nominal input rate to the same enormous input volume would have produced an estimated bill of about $77,059. The difference is an 88.6% reduction in realized API cost. The important conclusion is not that caching always cuts a bill by that exact amount. It is that the nominal input rate describes a hypothetical workload, while the realized bill reflects the sequence structure the agent actually generates.
Unit cost and total cost answer different questions
The striking result follows when the API bill is divided by total processed tokens. The cached Opus configuration works out to approximately $0.573 per million processed tokens. The study compares that with about $2.83 per million tokens for the shared private allocation and $11.87 per million for the dedicated reservation. On this denominator, the supposedly expensive API is cheaper than either private arrangement.
That is a genuine unit-cost inversion, but it is not a reason to declare the API the cheapest deployment. The Opus configuration processed about 16.9 times as many tokens as the GLM configuration. Its total API bill was consequently larger even though its effective price per processed token was lower. The private numbers also divide a time-billed resource by the tokens generated for one developer. If more developers share the same tray, the denominator grows while the capacity cost is spread across more work. If the tray sits idle or is charged entirely to one user, the unit figure rises.
This is the difference between a marginal or normalized unit question and a budget question. The unit question asks, “How much did this configuration pay for each processed token under this workload?” It is useful for understanding cache economics and utilization. The budget question asks, “How much will the organization actually pay to support the required work over the period?” It includes the absolute volume of tokens, capacity allocation, operations, and—once delivery quality is considered—the human cost of repairing output.
A cloud API with a high effective bill may still be the right choice when it processes much more context because its agent completes tasks with fewer corrective rounds. Conversely, a private model with a lower nominal or allocated compute cost may not lower the budget if it requires more retries, more review, or more dedicated idle capacity. The denominator can make a deployment look efficient while the invoice remains large; the invoice can make it look expensive while the cost per successful task is favorable. Neither view is sufficient alone.
The shared-versus-dedicated distinction makes the allocation problem especially clear. A shared private seat charges a standard slice of capacity to a developer while other workloads can use the remaining resource. A dedicated reservation charges the tray around the clock, whether the one developer has a request in flight or not. The model and hardware have not changed, but the economic denominator has. The same private deployment can move from attractive to unattractive because the organization changed who bears idle time.
Sharing does not make capacity unlimited. As more requests arrive, the cluster can absorb them only until memory, token budgets, or serving throughput become the bottleneck. Before that point, another user can improve utilization and spread the fixed cost. After that point, users compete for the same resource, queues grow, and a nominally cheap request may violate a latency target or spend more time waiting for a usable result. The cost advantage of sharing is therefore conditional on both occupancy and service quality.
This is why a capacity forecast needs at least two views. The first is an accounting view: how many GPU-hours are attributed to the team, and how much of that time is actually charged to active work? The second is an operating view: how many concurrent sessions the allocation can serve while preserving acceptable first-token latency and completion behavior? A private deployment can be well utilized in the accounting sense and still be a poor choice if saturation creates queues. It can also have excellent latency at low load while wasting money through a reservation that few developers use.
This is also why “self-hosting has zero marginal token cost” is an incomplete statement. Once the cluster exists and has spare capacity, another request may have little incremental GPU expense. But the organization still pays for the cluster, its serving layer, electricity and cooling, monitoring, capacity planning, and the people who keep it reliable. Whether a new workload is cheap depends on whether it uses otherwise idle capacity, displaces another workload, or triggers more hardware and operational support. Zero marginal cost at the moment of generation does not mean zero economic cost for the system.
Cache behavior must be interpreted in the same way. A high API cache-hit rate reduces the cost of repeated context, but it does not reduce output charges, remove model errors, or guarantee that a long context is useful. A private prefix cache can reduce repeated prefill work and improve serving efficiency, but the study’s local cost model is still based on GPU hours. The cache may improve how much work a tray can handle; it does not turn the tray’s lease into a token-metered invoice. In both cases, the system’s accounting basis matters.
The measurements that make a comparison credible
Before comparing a cloud API with a private deployment, an organization needs more than a vendor quote. It should establish the workload’s shape in observable terms.
First, measure the token mix. Record total prompt tokens, incremental input, cache creation, cache reads, output tokens, and cache-hit rate separately. A single “tokens used” field hides the main mechanism that made the Opus result surprising. For an agentic workload, also record how much of the context is stable and how often the agent changes the prefix by adding tool output or new repository material.
Second, measure absolute volume and session structure. Two teams can have the same effective price per million tokens and radically different bills if one runs longer sessions, retains more history, or produces more verbose intermediate output. The Opus configuration generated about four times as many output tokens as the GLM configuration while delivering comparable code-only gross insertions. That difference is consistent with a more verbose agent loop, but it also means that unit comparisons cannot substitute for a total-workload forecast.
Third, measure capacity allocation and concurrency for private serving. Record the smallest hardware footprint that can serve the chosen checkpoint, how many requests share it, how much of the period is billable, and whether work arrives in bursts. A four-GPU slice used by one developer is not the same economic object as the same slice serving a team. The organization should also distinguish peak throughput from sustained throughput under the mixture of prompt-prefill and output-decode work its users actually create.
Fourth, measure the work required after generation. A cache-adjusted API bill can look favorable per token while still being too high in total. A local cluster can look favorable per hour while its output creates more test failures and repair rounds. The cost comparison becomes decision-relevant only when it is joined to accepted changes, fix volume, review time, and latency to useful progress. Those terms are not embellishments added after the infrastructure decision; they are the feedback loop through which the infrastructure creates value or waste.
Finally, keep the billing convention visible. The enterprise study uses a shared-seat scenario and a full-reservation scenario to bracket how private capacity might be attributed. That is a useful sensitivity range, not a universal price. The API figures use published list rates, even though negotiated discounts could change the ordering. The local per-token amount is an amortization of hardware time over a particular single-developer workload, not a general marginal cost for every self-hosted request.
The practical rule is simple: never compare a nominal API input rate with a GPU-hour rate and call the result economics. Reconstruct the request mix, cache reuse, total volume, capacity hours, and concurrency first. Then ask how much accepted work the configuration produces and what human repair effort it consumes. The meter becomes meaningful only after its denominator matches the decision being made.
That accounting foundation changes the next question. If the private configuration can generate tokens faster and the API can make repeated context unusually cheap, the remaining difference must be found in the system that turns tokens into software: model capability, agent harness, quantization, serving behavior, and the feedback loop they create together.