LLM Inference Pricing: Tokens, Throughput and Real Costs

LLM inference cost is the sum of billable operations used to produce an outcome. Token rates matter, but so do the number of attempts, the length of outputs and the amount of context repeatedly sent. A low input rate cannot compensate for unlimited retries or an answer that fails the user's task.

This is an engineering analysis with calculated examples, not a provider benchmark. It explains how to read a pricing table, build a workload model and compare throughput without inventing measurements.

Start with the billing unit#

For text generation, a provider may distinguish uncached input, cached input and generated output. Other operations can use images, audio duration, requests or storage as their unit. Check the selected endpoint, model, service tier, context-length conditions and currency before comparing rates.

“Per million tokens” is a unit price, not a subscription allowance. A consumer assistant plan and a developer API bill are separate products. A provider may also report usage categories that are not interchangeable with visible text length. Use documented usage fields and reconcile them against actual invoices.

Keep an “unknown usage” category. Missing telemetry should trigger reconciliation rather than silently becoming free traffic.

A dated price calculation#

Checked on 9 November 2026, OpenAI's Standard short-context pricing lists gpt-6-luna at $0.10 per million input tokens, $0.01 for cached input and $0.50 for output. These are model- and tier-specific published rates, not AIx prices or a promise about future availability. Official pricing.

A hypothetical uncached request with 1,000 input and 200 output tokens therefore costs:

input  = 1,000 × $0.10 / 1,000,000 = $0.00010
output =   200 × $0.50 / 1,000,000 = $0.00010
total  = $0.00020

At 100,000 identical billable attempts, that arithmetic yields $20 before other services, taxes or charges. This is a calculated scenario. We did not send this workload or measure the resulting invoice. A different context band, tier, model or billing category requires recalculation.

Notice that 200 output tokens cost as much as 1,000 input tokens in this particular example. Optimizing only prompt length would miss half the modeled inference spend. That observation follows from these assumptions, not from a general rule about every model.

Distinguish prompt caching from response caching#

Provider prompt caching reuses computation associated with a shared prompt prefix. It does not mean the provider returns a previously generated answer. OpenAI's documentation also distinguishes cache-read and cache-write treatment; billing categories and eligibility must be checked for the selected model. Prompt caching documentation.

Application response caching is a separate design: your service stores a completed result and decides whether a later authorized request can reuse it. That can eliminate a provider call, but it introduces freshness and access-control obligations. A cached answer derived from one customer's documents must not be exposed to another customer simply because their text prompts match.

Do not forecast a cache-hit ratio from repetition alone. Record the provider's actual cache usage separately from your application's cache hits. The caching and observability guide gives an explicit scope and invalidation model.

Build a workload distribution, not one average prompt#

Separate short questions, long document requests and multi-step workflows. For each class, record request count, input/output distribution, retries and success criterion. An average hides the expensive tail, particularly when a small number of document-heavy requests dominate token consumption.

Keep conversation history in the model. If each new turn resends previous messages, later turns have a larger input even when the new user message is short. Retrieval can reduce irrelevant material, but it adds embedding, storage or reranking costs and can omit necessary evidence. Evaluate both cost and answer quality after changing context selection.

Create a scenario table with measured quantities where available and labeled estimates elsewhere. Preserve model version, tokenizer assumptions and source date. A cost model with explicit uncertainty is more useful than a precise-looking total based on undocumented defaults.

Throughput and latency answer different questions#

Latency is elapsed time for a request or stage. Throughput is completed work per time interval under a specified load. Tokens per second can refer to one stream or aggregate output across many concurrent streams. Those measurements cannot be compared without the denominator and workload.

An endpoint can deliver high aggregate throughput while individual users wait in a queue. Increasing concurrency may improve utilization until saturation, after which queueing, timeouts and retries can make both experience and cost worse. Do not infer capacity from a single fast response.

For self-hosting, useful utilization affects cost allocation. A hypothetical machine costing $2 per hour and serving 2,000 accepted tasks per hour allocates $0.001 compute cost per accepted task. At 200 accepted tasks, that becomes $0.01. These assumed numbers demonstrate division, not GPU performance or a hosting quote. Add idle hours, staff, replicas and other infrastructure before comparing against API charges.

Calculate cost per accepted task#

Define an accepted task with a stable rubric. For a citation assistant, acceptance might require a supported answer, correct document permissions and valid citations. Keep the rubric version alongside the bill.

cost per accepted task = attributable cost of all attempts
                       / number of tasks meeting the acceptance rule

If a hypothetical configuration spends $30 and produces 8,000 accepted tasks, its cost is $0.00375 per accepted task. Another spending $24 for 4,000 accepted tasks costs $0.006. The second has a smaller invoice but a higher unit cost under this assumed rubric. No accuracy score or experiment is claimed here.

Include review effort when it changes materially. A model that generates cheap drafts requiring lengthy corrections may shift costs into human labor rather than reduce them.

Compare commercial terms consistently#

Use the same currency, period, workload and service requirements. Separate prepaid credit from consumed cost. Record minimum commitments, expiration terms, capacity reservations and support needs from the actual offer. Check whether batch processing fits the product's delay tolerance; a discount is irrelevant to an interactive workflow that cannot wait.

Compatibility is also a migration cost. An OpenAI-compatible interface can reduce integration work, but endpoint support and model behavior still require validation. A cheaper provider that lacks a required feature may require application changes.

A practical review sequence#

First reconcile a small representative workload with provider usage and billing. Next identify the largest cost category rather than optimizing the easiest counter. Change one lever, rerun the same evaluation and compare cost, accepted-task rate and latency. Finally test peak concurrency and failure behavior before forecasting growth.

Use the AI app budget worksheet for non-inference costs and API latency methodology for a reproducible measurement plan. The useful output is a dated, explainable cost model with clear limits—not a universal cheapest-model ranking.