From Demo to Service: Evidence for Quality, Operations, and Cost

The repository’s production lessons ask a harder question than “does the model answer?” They ask whether an AI feature behaves usefully under real traffic, within a latency and cost envelope, and with a recovery path when components fail. This chapter connects evaluation, telemetry, inference metrics, cost attribution, and graceful degradation into one operational evidence loop. The goal is to distinguish a convincing demo from a service that can be measured and governed.

That distinction follows naturally from the previous chapter. An outcome gives the system a reason to exist; a metric says what success would look like. Production adds the conditions under which that success must hold: traffic varies, tools time out, prompts change, costs accumulate, and the slowest requests shape user experience. A system that produces good answers only on a quiet happy path is not yet an operable system.

Four kinds of evidence answer different questions

The repository’s LLM engineering route places evaluation before release. An evaluation asks whether a defined input produces an acceptable behavior. It can check correctness, relevance, format, safety, or faithfulness against a rubric or deterministic assertion. A regression suite compares a new prompt, model, or pipeline with a known baseline. The important property is that the test cases and criteria exist before the team sees the new result, so enthusiasm does not become the scoring method.

A trace answers a different question: what happened during one request? It can show which prompt version was selected, whether retrieval or cache lookup ran, which tool was called, how long each component took, and where an error occurred. Structured logs preserve request-level fields that can be aggregated later. Neither a trace nor a log says that the answer was correct by itself. They make the path inspectable so an evaluation failure or user complaint can be investigated.

Inference metrics answer a third question: how does the service behave under load? The repository distinguishes time to first token from the time between later tokens, end-to-end latency from throughput, and averages from tail percentiles. These distinctions reflect different bottlenecks. A long prompt can delay the first token. A crowded decode path can slow later tokens. Queueing can affect some requests while leaving the mean apparently healthy.

The user-visible outcome is a fourth question. A system may have acceptable latency and a green parser while failing to help the person make the intended decision. Conversely, a response that is useful in a prototype may become unacceptable when its cost or tail latency rises under real traffic. The evidence types must therefore be connected without being merged. Evaluation measures behavior, traces explain a path, inference metrics describe service conditions, and outcome measures tell the team whether the system matters.

Release quality is a loop, not a final glance

The repository’s evaluation material rejects “vibes-based” release decisions. Reading a handful of outputs can reveal obvious failures, but it is poor at detecting small regressions, edge-case failures, or changes that affect one category of user. A useful evaluation set contains representative core cases, difficult boundaries, and examples from the actual distribution when those are available. The suite should run when prompts, model settings, retrieval logic, or agent code changes.