Controlling AI Costs in Production Without Degrading Quality

AI costs are easy to underestimate in a pilot. A few hundred carefully chosen prompts make the model bill look trivial. Production is different: users repeat questions, automated workflows run at scale, context windows grow, retries happen, and the most valuable requests are often the most expensive ones. The answer is not to make every response cheaper. It is to spend deliberately where quality creates value and remove waste everywhere else.

The strongest cost controls are architectural. They determine which model handles a request, how much context it receives, whether the same work is repeated, and whether a human or a deterministic rule should handle the task at all.

That matters because cost and quality are not opposites. A system with good routing and measurement can reduce spend while improving response quality, because the model is given a clearer task and less irrelevant context. The work is to identify the decisions that need expensive reasoning and separate them from the work that does not.

Measure the cost of a useful outcome

A model invoice is only one part of the cost of an AI feature. The complete picture includes retrieval, storage, orchestration, retries, observability, human review, and the engineering work required to keep the system reliable. A low token bill can still represent poor economics if users distrust the output and staff rework every answer.

The useful unit of measurement is therefore not cost per request. It is cost per accepted outcome: a correctly classified document, a resolved support case, a completed draft, or a decision that a qualified person can approve. Track model spend alongside acceptance rate, correction rate, escalation rate, and time saved. That tells you whether a cheaper configuration is actually cheaper for the business.

This is the same discipline required in any operational system. The rewards platform UME built and operated for seventeen years served millions of users and processed significant value because operational measures mattered more than an isolated infrastructure line item. AI systems need the same view: cost is a production metric, not just a procurement concern.

Monitoring AI Systems: What to Measure Beyond Uptime →

Route requests to the right model

Using the most capable model for every request is a simple design, but it is rarely a sensible production design. Many tasks do not require open-ended reasoning. A smaller model, a specialised classifier, a cached response, or a conventional rule can handle them with lower latency and lower cost.

Model routing can be based on task type, document complexity, confidence, customer tier, or the consequence of an error. A request to extract a known set of fields from a familiar document does not need the same model as a request to compare conflicting clauses and explain the risk. The routing policy should be explicit enough to test and change, rather than hidden in scattered application code.

Routing also creates a better quality conversation. It lets the team reserve expensive capability for the cases where it changes the outcome, while keeping the common path fast and predictable. If a cheaper model is not good enough for a task, the decision should be visible in evaluation results rather than assumed from a model leaderboard.

Control context before reducing capability

Large prompts are often treated as a quality feature. More context feels safer, so systems retrieve more documents and include longer conversation histories. In practice, irrelevant context increases cost, adds noise, and gives the model more opportunities to follow the wrong detail.

Context discipline starts in retrieval. Return the smallest useful set of passages, remove duplicates, preserve document boundaries, and make the source metadata available without repeating it in every prompt. Summarise older conversation history when the detail is no longer needed, and do not send fields to a model merely because they happen to be available in the request object.

This improves security as well as economics. Every item in a context window is data the model has been exposed to. The retrieval and data-isolation decisions discussed in our AI security work should therefore be treated as cost and quality decisions too: narrower context is usually cheaper, safer, and easier to evaluate.

AI Security: Access Control and Data Isolation for Enterprise Systems →

Remove repeated work with caching and batching

Production systems frequently ask the same question more than once. A user retries because the interface timed out. A scheduled job processes an unchanged document again. Several people request a summary of the same source. If the system cannot recognise repeated work, it pays for it repeatedly.

Cache deterministic or slowly changing results where the business rules allow it. Cache retrieval results separately from generated answers when that makes invalidation easier. Include the model, prompt version, source version, and relevant permissions in the cache key so a fast answer never becomes a stale or unauthorised answer.

For asynchronous work, batching can reduce overhead and smooth demand. It is appropriate for indexing, classification, and other jobs where a response does not need to appear immediately. The key is to make retry behaviour explicit. An uncontrolled retry loop can turn a transient provider failure into a cost incident just as quickly as it turns into an availability incident.

Set budgets at the feature boundary

A single monthly budget for the whole AI estate is too blunt to guide engineering decisions. Set budgets for features, tenants, workflows, and background jobs. Make the owner of each budget clear, and distinguish a hard stop from a degraded mode. A document enrichment job might pause safely; a customer-facing workflow may need to route to a human or a simpler model instead.

Budget controls should be enforced by the service that issues model access, not only displayed on a dashboard. The same server-authoritative principle applies to AI spend as to any other entitlement: clients can request work, but the server decides whether the request is within policy and what fallback is allowed.

Alerting should be based on rate and change as well as absolute spend. A feature that normally costs £20 a day but suddenly costs £200 needs attention before the monthly total becomes meaningful. Record prompt and model versions with the cost data so an increase can be traced to a deployment, a traffic change, or a shift in input distribution.

Spend where intelligence changes the outcome

Production AI cost control is not a race to the lowest token price. It is the design of a system that knows which work needs intelligence, which work needs determinism, and which work needs a person. Measure the outcome, route requests deliberately, control context, eliminate repeated work, and enforce budgets at the boundary where requests are accepted.

That approach gives the business something more useful than a low bill. It gives the team a system whose economics are explainable, whose quality can be protected, and whose costs can grow with value rather than with every accidental implementation detail.