LLM inference is computationally expensive — caching responses for repeated or similar queries can dramatically reduce compute costs and improve response times. This guide covers implementing effective LLM response caching.
Why LLM Caching Is Particularly Valuable
Unlike typical API caching (see How to Cache API Responses Effectively), LLM inference cost per request is often substantially higher — even a modest cache hit rate translates to meaningful compute/cost savings, making caching a particularly high-value optimization for AI workloads specifically.
Exact-Match Caching (Simplest Approach)
import hashlib
import redis
redis_client = redis.Redis()
def get_cached_or_generate(prompt, model_params):
cache_key = hashlib.sha256(f"{prompt}:{model_params}".encode()).hexdigest()
cached = redis_client.get(cache_key)
if cached:
return cached.decode()
response = run_inference(prompt, model_params)
redis_client.setex(cache_key, 3600, response)
return responseCaches based on exact prompt match — simple and effective for genuinely repeated identical queries (common in some applications, like FAQ-style chatbots with predictable common questions).
Semantic Caching (Catching Similar, Not Just Identical, Queries)
from sentence_transformers import SentenceTransformer, util
embed_model = SentenceTransformer('all-MiniLM-L6-v2')
def find_similar_cached_query(new_prompt, cached_prompts, threshold=0.92):
new_embedding = embed_model.encode(new_prompt)
for cached_prompt, cached_embedding in cached_prompts:
similarity = util.cos_sim(new_embedding, cached_embedding)
if similarity > threshold:
return cached_prompt
return NoneSee How to Build a Semantic Search Engine with Embeddings for the underlying concept — catches semantically similar (not just identical) queries, meaningfully increasing cache hit rate for applications where users phrase similar questions differently.
Choosing an Appropriate Similarity Threshold
Too low a threshold risks serving a cached response that doesn't genuinely match the new query's intent; too high a threshold misses legitimate cache opportunities — tune based on your specific use case's tolerance for near-match responses versus guaranteed accuracy.
Setting Appropriate Cache Expiration
Consider whether cached responses should expire — for genuinely time-sensitive information, shorter expiration; for stable factual/reference content, longer caching is appropriate without risking staleness concerns.
Caching at Different Granularities
Beyond full-response caching, consider whether intermediate steps (like retrieved context in a RAG pipeline, see How to Deploy a RAG (Retrieval-Augmented Generation) Pipeline on a VPS) can also be cached separately, providing partial optimization even when the full response isn't cacheable.
Excluding Genuinely Dynamic/Personalized Content from Caching
Don't cache responses containing user-specific or genuinely time-sensitive information inappropriately — be deliberate about what's genuinely cacheable versus what must be generated fresh for correctness.
Monitoring Cache Hit Rate
Track cache hit vs miss ratio to understand whether your caching strategy is genuinely delivering value — a low hit rate suggests either your threshold/matching approach needs tuning, or your actual query patterns have less genuine repetition than assumed.
Balancing Cache Storage Cost Against Compute Savings
For very large-scale caching (many stored responses), consider the storage cost/complexity trade-off against the compute savings achieved — usually still strongly favors caching given LLM inference cost, but worth being deliberate about for very large-scale deployments.
Common Errors
Cached responses feel stale or wrong for some queries — review your semantic similarity threshold; it may be too permissive, matching genuinely different queries as "similar enough" when they actually warrant distinct responses.
Continue Reading
- How to Cache API Responses Effectively
- How to Optimize LLM Inference Speed on Limited Hardware
- How to Monitor and Log AI Model Usage and Costs
Browse more articles in AI & Machine Learning on a VPS.
