On this page
Reduce LLM API costs by removing work that does not help the user, then verify that the remaining system still completes the task. Start with a ledger of model usage and accepted outcomes. Without that baseline, a smaller bill can hide lost functionality, shifted human effort or requests that fail before they are counted.
This guide proposes an optimization sequence and release gates. It does not promise a savings percentage, report an experiment or assume that the cheapest model is adequate for your workload.
Establish a baseline you can explain
Record logical user requests and provider attempts separately. Capture the selected model, input/output usage, latency, error class and evaluation result. Keep prompt text out of ordinary telemetry; identifiers and aggregate counts are usually sufficient for cost diagnosis.
Segment by task type. Summarizing a short message, answering a document question and running an agent loop have different cost drivers. Use the pricing guide to distinguish billable categories rather than summing incompatible counters.
Choose an acceptance rule before optimizing: valid citations, a correctly structured result, passing software checks or a reviewed support resolution. Freeze a representative evaluation set that includes difficult and safety-sensitive cases. Preserve its version so later comparisons use the same test.
Identify the expensive stage
| Observation | Hypothesis to investigate | Evidence before changing it |
|---|---|---|
| Input usage grows through a conversation | History is repeatedly resent | Per-turn token and relevance breakdown |
| Long answers rarely get read | Output instruction is too broad | User task requirements and output review |
| Attempts exceed user requests | Retries or agent loops multiply work | Attempt trace with termination reasons |
| Similar requests repeat | A carefully scoped cache may help | Safe reuse boundary and freshness requirement |
| One model handles every task | Some tasks may permit cheaper execution | Per-class acceptance tests and fallback policy |
Treat these as hypotheses. A repeated prompt might still require a fresh answer; a long context might contain the only decisive evidence. Remove content because it is unnecessary for the task, not because it is large.
Reduce context without discarding evidence
Begin with duplicated instructions, redundant history and irrelevant retrieved passages. Keep authorization and safety constraints intact. If documents are retrieved, evaluate whether the selected passages actually contain the answer and preserve source identifiers for citations.
Summaries can control conversation growth, but summarization is another model operation and can lose details. Test multi-turn references, changed user instructions and corrections. A summary that forgets a critical constraint can reduce token cost while damaging correctness.
Version the context-building logic. Keep a rollback path and record when an evaluation failure results from missing evidence rather than generation. In document applications, a retrieval miss is a different problem from a model ignoring a correct passage.
Constrain output to the required deliverable
Specify a useful structure and a bounded amount of detail. If the application needs a short classification, request that instead of an essay later parsed into a label. Validate the returned structure and handle refusal or incomplete output explicitly.
Set an output limit compatible with the task. A limit that truncates valid JSON can create more retries than it saves. Check completion status and verify that the response meets the application's contract before displaying or executing it.
Do not remove explanations users need to assess a consequential result. Cost reduction should follow the product's evidence requirements. A medical, legal or financial workflow may need qualified review rather than a smaller automatically accepted answer.
Eliminate accidental repeated work
Inspect double submissions, reconnect behavior, jobs that are restarted after completion and clients that retry automatically while a server also retries. Assign a logical request identifier and propagate an attempt identifier so the sequence can be reconstructed without logging sensitive content.
Set an explicit retry budget. Retry only failures for which another attempt is appropriate, respect the overall deadline and avoid duplicating external actions. A timeout can leave upstream work in progress. Idempotency must be supported by the operation's real storage or supplier contract; a header name alone does not make a purchase safe to repeat.
For agents, cap steps, tool calls and cumulative cost. Require approval for consequential actions and define a stop state when evidence is insufficient. See agent versus chatbot for why tool workflows need controls beyond ordinary chat limits.
Use caching at the correct boundary
Provider prefix caching and application answer caching solve different problems. Check current provider eligibility and billing; do not forecast every repeated input at a discounted rate. Arrange stable context only when that preserves correct instructions and document boundaries.
An application result cache needs tenant, subject, model and relevant data/prompt revision in its key. Time-to-live alone does not revoke access after a permission change. Invalidate or bypass entries when source permissions or content change. The tested cache example illustrates isolation and expiration without claiming a production distributed cache.
Never cache errors as successful answers. Be cautious with personal, time-sensitive or externally acted-on results. The safest optimization can be deliberately declining to cache a task whose reuse conditions are unclear.
Route by demonstrated task capability
A smaller or less expensive model can be appropriate for constrained tasks, but evaluate the exact configuration. Compare accepted-task cost, latency and failure modes. Include the routing classifier's own cost and errors if one is used.
Define fallback triggers before release. For example, malformed structured output may trigger one bounded retry with an approved alternative. Avoid an unbounded cascade that calls several providers for every difficult request. Report the proportion of requests taking each path.
Routing also changes the data-processing boundary. An inexpensive fallback cannot silently receive documents restricted to another supplier. Keep approved destinations and models on the server, as described in secure AI API gateways.
Consider asynchronous work deliberately
Tasks such as overnight enrichment may tolerate queued processing. Compare current supplier terms for an appropriate non-interactive service rather than applying an advertised discount to every workload. Include scheduling, status tracking and partial-failure recovery in the implementation estimate.
Coalescing duplicate background tasks can avoid repeated processing, but maintain per-customer authorization and cancellation semantics. A shared result must be safe for every recipient, not merely similar in text.
Release an optimization with quality gates
For each change, write the hypothesis, affected task classes, expected mechanism and rollback trigger. Run the same evaluation and compare total cost, accepted-task count, errors and latency. Keep difficult cases visible instead of reporting only an overall average.
During a controlled rollout, monitor both customer outcomes and billing reconciliation. A falling usage counter caused by missing instrumentation is not savings. A cheaper response that shifts correction work to support must be counted honestly.
Start with one observable waste source: duplicate requests, irrelevant context or unconstrained output. Implement the smallest change, retain the previous configuration and compare results. That produces a defensible cost improvement rather than a collection of unverified prompt tricks.