On this page
AI API reliability means delivering the product's required behavior for the requests it intends to serve. HTTP success alone is insufficient: a response can arrive on time with invalid structure, unsupported citations or an unauthorized tool proposal.
Evidence type: methodological analysis. This article proposes measurement definitions and a study protocol. It does not report Theaimart uptime, supplier availability, model accuracy or original reliability experiments. All example counts are synthetic.
Begin with a user-visible contract
Define the operation, eligible request population and success conditions. For a text assistant, success might require an authorized request, a complete supported response before a deadline and a valid application schema. For a document assistant, citation correctness and document permissions may require additional evaluation.
Keep transport availability, contract validity and task quality as distinct indicators. Some quality properties require human or delayed evaluation and cannot be known at response time. Label proxies as proxies rather than calling a schema-valid answer correct.
Google's SRE guidance distinguishes an SLI as a measured service property and an SLO as the target for that property. Use that distinction when defining your own service, rather than adopting a numerical target without product evidence. Service-level objectives reference.
Name the denominator
Count logical requests received within the defined eligible population. A retry is an additional attempt for the same logical request, not a new independent user success. Exclude malformed or unauthorized requests only under a predeclared rule, and report excluded counts separately.
logical request success rate = successful eligible logical requests
/ all eligible logical requests
attempt success rate = successful provider attempts
/ all provider attemptsThese indicators answer different questions. A service may recover one failed attempt through a retry and deliver a successful user result, while spending more and adding latency. Reporting only the successful final attempt hides that operating cost.
Use a classification that preserves failure stages
| Class | Example | Measurement implication |
|---|---|---|
| Authentication/authorization denial | Caller cannot access the operation | Report separately under the eligibility rule |
| Admission rejection | Local quota or capacity exhausted | User-visible failure if the request is in scope |
| Transport or provider failure | Network error or upstream non-success | Attribute stage without exposing raw payloads |
| Deadline exceeded | No acceptable result within the allowed time | Failure for the deadline-based contract |
| Invalid response contract | Missing required text or invalid schema | HTTP success does not make it a valid result |
| Task-quality failure | Unsupported answer under the evaluation rubric | Requires declared evaluation coverage |
| Successful completion | All required observable conditions met | Count once per logical request |
Choose one terminal class per logical request for the primary ledger and retain attempt-stage details separately. Avoid double-counting a timeout and its later retry as two user failures unless the product contract truly treats them as distinct tasks.
Define the observation window
Record interval start/end, time zone, deployment revisions, traffic mix and eligibility changes. A rolling window and a calendar month are different reporting conventions. Preserve the convention when comparing periods.
Scheduled maintenance, supplier incidents and dependency failures should not disappear through ad hoc exclusions. If an agreement defines exclusions, show both the contractual measure and a user-visible measure where useful. A favorable denominator chosen after an incident is not a trustworthy reliability practice.
Low traffic creates uncertainty. Report request counts and intervals of observation; do not derive a confident long-term uptime claim from a few successful probes.
A synthetic retry example
Assume 1,000 eligible logical requests. Of these, 950 succeed on the first attempt, 30 succeed after one failed attempt and 20 fail after two attempts. Logical success is 980 / 1,000 = 98%.
There are 1,050 attempts: 950 + 60 + 40. Successful attempts total 980, so attempt success is about 93.33%. The difference is a consequence of the assumed retry policy. These are invented counts, not measured service results.
If the 30 recovered requests exceed the product's deadline, they should not count as success under a deadline-based SLI. Define success before analysis so a late answer cannot be reclassified merely to improve the percentage.
Measure quality with explicit coverage
Create a stable evaluation rubric and identify which traffic it covers. A sampled citation review should report sampling method, sample size and uncertainty. Do not label the unreviewed remainder accurate by default.
Separate deterministic checks from judgment. JSON schema, permission boundaries and required fields can often be tested automatically. Factual support and usefulness may require more careful evaluation. If another model is used as a judge, document its configuration and validate its suitability rather than treating it as ground truth.
Keep sensitive content out of ordinary operational logs. A reviewed evaluation dataset can have a different access and retention policy from aggregate reliability telemetry.
Use probes and production observations appropriately
Synthetic probes can detect a narrow known task failing, but they do not represent every customer workload. Production observations reveal actual traffic but may lack complete labels. Report the two evidence sources separately.
A local mocked-provider test demonstrates application handling of selected failures. A live probe demonstrates one real interaction at a particular time. Neither alone establishes a long-term supplier reliability score. The FastAPI example explicitly limits its claim to mocked tests.
When comparing providers, freeze the required contract, workload and load schedule. A provider receiving shorter tasks or more forgiving timeouts should not be ranked as more reliable without explaining those differences.
Connect indicators to action
Define who investigates a rising failure class and what action is available: disable a bad deployment, reduce admission, stop an unsafe tool or switch to an approved evaluated route. An SLO without an operational response can become a dashboard decoration.
Track error-budget consumption using your chosen target and request population. Distinguish rapid incident detection from slower product-quality review. Avoid selecting numerical targets here without the business and user requirements needed to justify them.
Observe latency and cost alongside reliability. An unbounded retry policy may conceal provider instability while increasing latency and spending. A fallback may improve availability while changing quality or data-processing boundaries.
Keep a reproducibility packet
Store the contract, eligibility rules, event schema, deployment/model revisions, observation window, retry policy, analysis procedure and known gaps. Retain sanitized counts and explain revisions. Publish original reliability findings only after collecting adequate authorized evidence.
Begin with one user-visible operation and a ledger that counts each logical request once. Test the classification using controlled failure fixtures, then compare it against actual service behavior. Reliable measurement starts with a denominator and contract other engineers can reproduce.