AI SLAs and Error Budgets: How to Set Expectations for AI Features

Conventional service-level agreements are built around availability and response time. Those measures still matter for AI systems, but they do not describe whether an answer is useful, grounded, safe, or worth the cost. An AI endpoint can return a response within 300 milliseconds and be completely wrong. It can be online all day while users quietly stop trusting it.

AI service expectations need to cover both the service and the intelligence it provides. That means defining the task, the quality threshold, the acceptable failure modes, and the recovery path. It also means accepting that AI quality is measured statistically and operationally rather than guaranteed exactly for every request.

The goal is not to pretend that an AI system is deterministic. The goal is to make its behaviour explicit enough that product, engineering, operations, and customers can make sensible decisions when quality changes.

Availability is necessary but not sufficient

An availability target answers a narrow question: could the system accept and process a request? It does not answer whether the response met the task requirement. For an AI assistant, a successful HTTP response may contain an unsupported claim, omit a required field, or fail to cite the source that a reviewer needs.

Keep conventional availability and latency measures, but add task-level quality measures. Depending on the feature, these might include extraction accuracy, answer groundedness, correct routing, reviewer acceptance, citation completeness, or the rate at which a user has to correct the output. The metric should describe the job the system was hired to do.

Quality metrics need a defined measurement method. A sample of production outputs reviewed by domain experts is often more meaningful than a generic benchmark. Automated checks can provide coverage and speed, but high-consequence workflows should retain a human evaluation path for the cases where a score does not capture business risk.

Monitoring AI Systems: What to Measure Beyond Uptime →

Define the task before the target

There is no useful SLA for 'good AI'. There can be a useful service expectation for extracting invoice totals, answering questions from an approved knowledge base, classifying incoming cases, or drafting a response for a trained employee to review. Each task has a different definition of success.

Write the target in operational terms. For example: a specified percentage of sampled answers must be supported by an approved source; a document classifier must route a defined percentage of cases correctly; a draft must contain all mandatory fields before it reaches a reviewer. Include what happens to the remainder. An output below the threshold may be rejected, sent for review, or returned with an explicit uncertainty state.

This level of specificity prevents a common failure in AI programmes: debating model quality without agreeing on the business outcome. It also makes supplier and internal conversations clearer, because the team can test the feature against a shared definition rather than a subjective impression.

Separate quality failure from safety failure

Not every error has the same consequence. A slightly incomplete summary is a quality miss. Revealing information from the wrong tenant, inventing a contractual obligation, or taking an unauthorised action is a safety failure. The latter should not be averaged into a general quality score where it can disappear among thousands of harmless misses.

Define safety invariants separately and enforce them as hard boundaries. Access control must be applied before generation. High-consequence actions should require explicit authorisation and, where appropriate, human approval. Responses that lack sufficient evidence should be withheld or clearly marked rather than presented with fluent certainty.

A good SLA can therefore contain several layers: availability, latency, task quality, and zero-tolerance or low-tolerance safety conditions. The layers need different alerting and response procedures. A quality decline may trigger evaluation and rollback. A data-isolation violation is an incident requiring immediate containment and investigation.

AI Security: Access Control and Data Isolation for Enterprise Systems →

Use error budgets to manage change

An error budget is the amount of failure a service can absorb while still meeting its target over a defined period. For AI, the budget might be a permitted rate of ungrounded answers, incorrect classifications, reviewer escalations, or outputs that need correction. The exact metric depends on the task, but the operating idea is the same: reliability is managed over time, not declared once at launch.

When the budget is healthy, the team can make controlled changes: test a cheaper model, adjust retrieval, revise the prompt, or expand the automation boundary. When the budget is exhausted, the response should be to reduce change and investigate. That may mean rolling back a model version, routing more cases to humans, narrowing the task, or pausing a feature until quality recovers.

This is particularly valuable because AI changes can look harmless in code review while changing the output distribution in production. A prompt edit, a new document source, or a provider update can move quality for one category of request without affecting the overall average. Error budgets make that operationally visible and give the team a shared basis for deciding whether to proceed.

Human in the Loop Design for AI Powered Workflows →

Include latency, cost, and escalation capacity

A quality target that ignores latency will encourage a system nobody wants to use. A quality target that ignores cost can make a feature commercially unviable. A quality target that assumes unlimited human review will fail the moment volume increases. These constraints belong in the service definition from the start.

Record latency by workflow and not only as a single average. A p95 or p99 view shows whether a minority of complex requests is creating an unacceptable experience. Track cost per accepted outcome, not just cost per call, and include the capacity of the review queue in the operational model. If a fallback sends ten times more work to people, that is not a graceful degradation strategy; it is an unpriced failure mode.

These measures also clarify product trade-offs. A slower, more capable path may be appropriate for a legal comparison. A fast, lower-cost path may be right for triaging incoming work. The SLA should describe those paths separately instead of forcing every request into one average that hides the real behaviour.

Controlling AI Costs in Production Without Degrading Quality →

Make trust measurable enough to operate

AI SLAs are not a promise that every output will be correct. They are an agreement about what the system is designed to do, how quality will be measured, what failure is acceptable, and what happens when the limits are reached. That is a much more useful basis for trust than a claim that the model is intelligent.

The production systems that last are the ones whose behaviour can be observed and managed. Define task-level quality, separate safety from ordinary error, budget for change, and include the real constraints of latency, cost, and human review. Then the team can improve the system without losing sight of what users depend on.