On this page
An AI API latency comparison is meaningful only when it defines the work, the timing boundary and the requests that failed to finish. A single fast answer does not establish a provider's typical latency, capacity or user experience.
Evidence type: methodological analysis. This article specifies a reproducible study design. No provider experiment, latency measurement, GPU benchmark or ranking was performed for it. All numerical examples are explicitly synthetic.
State the research question
Choose one question at a time. “Which configuration returns the first useful text sooner for our short support questions under a fixed load?” is testable. “Which API is fastest?” is not sufficiently defined.
Specify the population: task class, input/output bounds, language, model configuration, deployment region and intended load. Define what counts as a valid answer. A fast refusal or truncated result should not silently compete with a complete answer as if they were the same task.
Use the same accepted-task rubric when comparing configurations. If exact model or tokenizer equivalence is impossible, document the difference rather than pretending the workloads are identical.
Define the timing boundaries
| Measurement | Start | Stop | Interpretation |
|---|---|---|---|
| End-to-end latency | Client begins the logical request | Valid complete result or terminal failure | Closest to the application's observed wait |
| Time to first response byte | Transport begins request timing | First response body byte | Can include metadata rather than generated text |
| Time to first useful text | Same documented request boundary | First parsed non-empty user-visible text event | Requires a correct stream parser |
| Completion interval | First useful text | Last required output/termination event | Describe whether final metadata is included |
| Queue delay | Work accepted into your queue | Execution admitted | Separates local scheduling from provider work |
For streaming, do not equate a TCP chunk with a complete event. Chunks may split frames, and heartbeat or role-only events are not generated text. Record parser and protocol versions. If the provider does not expose a quantity, mark it unavailable rather than estimating it from unrelated signals.
Use a monotonic elapsed-time clock on the measuring host. Python documents perf_counter() for performance timing and notes that only differences between calls are meaningful. Python 3.14 timing reference. Do not subtract wall-clock timestamps from unsynchronized machines to invent stage latency.
Freeze the workload and manifest
Create sanitized fixtures with identifiers and documented size distributions. Keep the fixture set private when necessary, but publish enough properties for interpretation. Do not send customer secrets merely to obtain realistic prompts.
The experiment manifest should include:
- Study question, acceptance rubric and predeclared primary metric.
- Fixture revision, task classes, languages and input-size distribution.
- Provider endpoint, model identifier and configuration date.
- Client/runtime versions, hardware, network region and connection policy.
- Output limits, streaming mode, timeouts, retry policy and concurrency.
- Warm-up, cache treatment, order randomization and run schedule.
- Budget ceiling, authorized request rate and stop conditions.
- Analysis rule, quantile convention and treatment of incomplete observations.
A provider can change behind a stable model alias. Record the version information available and acknowledge what cannot be pinned. Reproduction may mean repeating a documented procedure under a later service state, not recreating an identical hidden system.
Design load intentionally
In a closed-loop test, a worker sends another request after the previous one finishes. That measures a particular interactive pattern, but slow responses reduce the arrival rate. In an open-loop schedule, arrivals are scheduled independently; if the generator falls behind, record scheduling delay and rejected work.
Do not hide overload by dropping requests that could not be scheduled. Bound generator resources and respect supplier permissions. Increasing concurrency without controlling input/output size makes results difficult to compare.
Separate cold-start and warm-connection conditions. State whether DNS, TLS and connection establishment are included. If a cache is expected, document the request sequence; do not label a cache-warmed result as an uncached baseline.
Preserve an attempt ledger
Record one row per attempt with logical request ID, attempt index, fixture ID, configuration, scheduled start, actual start, timing observations, terminal status and usage fields where available. Keep authorization headers and content out of the ordinary ledger.
A timeout row should retain its elapsed bound and failure classification. A retry creates another attempt linked to the same logical request. Report logical user experience separately from individual attempt timing so retries cannot make failures disappear.
Clock readings are meaningful within their measuring process. Use duration fields and stable IDs for joins. If distributed tracing is available, document which spans are measured directly and which are inferred.
Report incomplete requests honestly
Suppose a synthetic run schedules 100 requests: 90 complete validly, five time out at ten seconds and five fail immediately. The completion-only median describes the 90 completed requests. It does not describe all 100 requests, and the timed-out requests have unknown completion times beyond the observation boundary.
Report completion rate, timeout threshold, error classes and completion-only latency together. If you estimate a full latency distribution with censored data, specify an appropriate statistical method and its assumptions. Do not replace timeouts with zero or quietly delete them.
If a configuration returns invalid output rapidly, classify it accordingly. Latency and quality must be interpreted jointly; a result that fails the task is not a successful fast answer.
Choose an analysis rule before seeing results
Report sample counts, task mix and a declared quantile convention. For example, a nearest-rank quantile uses the sorted observation at rank ceiling(p × n), for p in (0, 1]. Other tools may interpolate differently. State the method so another analyst can reproduce the number.
Tail estimates from small samples are unstable. Repeated requests from the same fixture, time window or provider incident may be correlated. Report the sampling structure and avoid an unjustified precision claim. A confidence interval needs a method matched to that structure; it is not supplied automatically by printing more decimal places.
Randomize or interleave configurations to reduce time-of-day confounding where permitted. Preserve run order and analyze major conditions separately. One configuration tested during an outage and another tested later is a weak causal comparison.
Connect latency to cost and reliability
Record billable attempts and accepted tasks for cost analysis. An aggressive retry strategy may improve apparent successful-response latency while increasing cost and upstream load. Reliability methodology defines the logical-request denominator separately.
If testing an OpenAI-compatible route, validate the exact contract first. A parser bug or unsupported parameter should not be misreported as a provider performance result.
Publish a reproducibility packet
Include the manifest, sanitized fixture description, harness revision, environment, attempt schema, analysis procedure, exclusions and known limitations. Publish original observations only when actually collected and authorized for release. Separate computed estimates from measured values and interpretation.
The immediate next step is to write the manifest and validate the harness against deterministic local fixtures for success, split stream events, delays and failure. Only then run a bounded, authorized provider study. Until measurements exist, present the work as a method rather than a leaderboard.