Caching and Observability for LLM Applications

Caching can avoid repeated model work; observability tells you whether that reuse is correct and useful. They belong together because a cache hit can conceal stale data, authorization mistakes or a lower acceptance rate. A fast answer is not a successful answer when it belongs to another customer.

This guide includes a tested process-local cache and a telemetry design. It does not claim a production distributed cache, measured hit rate or benchmark. The implementation is deliberately small enough to inspect and extend.

Choose what can safely be reused#

An exact result cache stores a completed answer for a precisely defined request context. Provider prompt caching reuses internal prefix computation and has different eligibility and billing. Do not count a provider cache read as an application result-cache hit.

Before caching a task, ask whether the result depends on current time, personal state, document permissions or external actions. A public static explanation may be reusable; a private account summary needs a much narrower boundary. Consequential tool actions should not be replayed from an answer cache.

Define freshness and invalidation before selecting a storage engine. A thirty-second time-to-live does not make cross-tenant sharing acceptable, and it does not automatically handle a permission revocation during those thirty seconds.

Build the key from the authorization context#

The example key includes tenant, subject, model, revision and prompt. The revision must represent every relevant data, prompt-policy and configuration change for the specific task. If temperature, tool configuration or retrieval filters affect the result, include them in that revision or in additional key fields.

The subject identifier should come from authenticated context. It is not a caller-chosen namespace. Recheck authorization before returning a cached result, and invalidate or bypass entries after relevant permission changes.

SHA-256 creates a stable compact key here. It does not encrypt the answer, prove authorization or make low-entropy prompt values impossible to guess. Protect the cache and its telemetry as potentially sensitive storage.

The executable cache#

The full module also imports OrderedDict, deepcopy, sha256, dumps, isfinite, Lock and monotonic. These are standard-library dependencies. The implementation below is copied from the tested source:

def cache_key(tenant, subject, model, revision, prompt):
    return sha256(dumps([tenant, subject, model, revision, prompt],
                        ensure_ascii=False, separators=(",", ":")).encode()).hexdigest()
 
 
class ResultCache:
    def __init__(self, max_entries=64, ttl_seconds=30, clock=monotonic):
        if max_entries < 1 or not isfinite(ttl_seconds) or ttl_seconds <= 0:
            raise ValueError("positive capacity and lifetime required")
        self.limit, self.ttl, self.clock = max_entries, ttl_seconds, clock
        self.items, self.lock = OrderedDict(), Lock()
 
    def put(self, key, value):
        with self.lock:
            self.items.pop(key, None)
            self.items[key] = (self.clock() + self.ttl, deepcopy(value))
            while len(self.items) > self.limit:
                self.items.popitem(last=False)
 
    def get(self, key):
        with self.lock:
            record = self.items.get(key)
            if record is None:
                return None
            if self.clock() >= record[0]:
                del self.items[key]
                return None
            return deepcopy(record[1])

The lock protects dictionary operations within one process. Values are deep-copied on insertion and retrieval so a caller cannot mutate a stored result through a returned reference. Expiration uses a monotonic clock. Capacity is bounded by entry count, with oldest-inserted entries evicted first; this is not an LRU policy.

Expired entries are removed when read, and capacity eviction keeps the entry count bounded even if some expired entries have not been read. Entry count is not a byte limit: a production design must also constrain value size and total memory. This cache stores values in plaintext process memory and loses them on restart.

Run and understand the tests#

Use the complete example and pinned environment:

cd examples/blog/infrastructure
.venv/bin/python -m pytest -q test_controls.py

The cache test checks tenant, subject and revision isolation; defensive copying; expiry at the exact lifetime boundary; and capacity eviction. It uses a fake clock instead of sleeping. Four control tests, including token-bucket tests, passed on 9 November 2026 with Python 3.14.4.

These tests establish specific local behaviors. They do not establish a safe production authorization integration, encryption, distributed consistency or deletion across replicas. Treat those as separate implementation requirements.

Avoid a cache stampede#

When many identical misses arrive together, each can call the provider before any result is inserted. A production system may coalesce in-flight work within the same approved authorization scope. Bound waiting callers, propagate deadlines and define what happens if the leader fails.

Do not coalesce across subjects merely because text prompts match. A shared public-result path should be a deliberate separate policy with public inputs and no private dependencies. Cache failures should have an explicit fallback policy rather than creating unbounded repeated provider work.

The sample cache does not implement in-flight coalescing. That omission keeps its testable contract honest and avoids implying that a dictionary alone solves peak-load behavior.

Separate requests from attempts#

Assign a logical request identifier at the application boundary and an attempt identifier for each upstream call. A cache hit may have no provider attempt; a retrying request may have several. Record both so cost and reliability metrics use the correct denominator.

Useful fields include route, task class, model/configuration revision, cache outcome, sanitized error class, elapsed time and provider usage categories when available. Avoid raw prompts, answers, bearer tokens and unbounded customer identifiers as ordinary metric labels.

OpenTelemetry describes traces, metrics and logs as observability signals. Use traces for request-stage relationships, metrics for bounded aggregate trends and restricted logs for diagnostics. A tracing library does not choose your privacy or retention policy for you.

Measure cache usefulness, not just hits#

Define hit rate over eligible lookups, not every app request if only some tasks are cacheable. Count bypasses separately from misses. Record invalidation reasons and distinguish fresh hits from stale or rejected entries.

Compare accepted-task outcomes, latency and attributable cost for cached and uncached paths. A high hit rate can reflect a broken key that incorrectly merges distinct requests. A drop in provider usage can reflect missing instrumentation instead of savings.

Reconcile provider attempts and usage with billing as described in inference pricing. Include cache infrastructure and maintenance costs when evaluating whether the optimization is worthwhile.

Design privacy-aware diagnostics#

Use stable low-cardinality labels such as route and error class. High-cardinality prompt hashes can still expose sensitive correlation and overload telemetry storage. If request identifiers are logged, control access and retention and avoid treating them as anonymous by default.

For debugging a specific answer, prefer a reviewed, limited capture process over enabling content logging globally. Consider support tools, crash reports and tracing exporters in the data-flow review. Redaction needs tests against actual exported payloads, not just the application's visible console.

Operate changes safely#

Version the cache policy and include meaningful revisions in keys. On a model, document or permission change, define whether to invalidate, bypass or migrate entries. Test deletion and revocation across every storage layer used in production.

Use reliability methodology to define success and latency methodology to compare cache paths without mixing workloads. Start with one task whose safe reuse conditions are explicit. Expand caching only when authorization, freshness and measurement are all explainable.