Caching and reducing spend

What’s actually repeated

Before caching anything, know what your service actually repeats. Log a week of queries and analyse:

• Exact repeats. Same query, same context. Common in FAQ-style bots.
• Near-repeats. Slightly different phrasings of the same question. Common in customer-service bots.
• Repeated system prompt. The system prompt is identical across every request.
• Repeated retrieved chunks. Popular documents get returned to many queries.

Different repetitions get different caching strategies. Understand the pattern first, then apply the right tool.

Technique 1 — Response caching

The most obvious: if a user asks the same question twice, don’t re-generate. Return the cached answer.

import hashlib
import redis
import json

r = redis.Redis()
CACHE_TTL = 24 * 3600   # 1 day


def cache_key(question: str, context: str) -> str:
    """Hash the full input to produce a stable cache key."""
    payload = f"{question}::{context}"
    return "llm_cache:" + hashlib.sha256(payload.encode()).hexdigest()


def get_or_generate(question: str, context: str) -> str:
    key = cache_key(question, context)

    cached = r.get(key)
    if cached:
        return json.loads(cached)["answer"]

    # Cache miss — call the model
    answer = call_model(question, context)

    r.setex(key, CACHE_TTL, json.dumps({"answer": answer, "cached_at": time.time()}))
    return answer

For a Nepali FAQ bot where “How do I renew my citizenship?” is the top query and gets asked 500 times a day, response caching cuts those 500 calls to 1. That’s a 99.8% reduction on that query.

When response caching works well:

• FAQ-style Q&A over a stable corpus.
• Deterministic tasks (translation, formatting) where the output depends only on the input.
• Content generation where “the same request should give the same answer” makes sense.

When it doesn’t:

• Personalised responses (each user gets a tailored answer).
• Time-dependent answers (“what’s today’s price?”).
• Conversational context (the exchange evolves per user).

Cache key design

The cache key determines what counts as “the same request.” Get this wrong and you either miss real cache hits or serve wrong answers.

For simple Q&A: hash the normalised question + retrieved context.

def normalise_question(q: str) -> str:
    return q.strip().lower()

def cache_key(question: str, context: str) -> str:
    payload = normalise_question(question) + "::" + context
    return "llm_cache:" + hashlib.sha256(payload.encode()).hexdigest()

For conversational context, include the last N turns:

def cache_key(question: str, history: list) -> str:
    recent = json.dumps(history[-4:])  # last 2 exchanges
    payload = normalise_question(question) + "::" + recent
    return "llm_cache:" + hashlib.sha256(payload.encode()).hexdigest()

Semantic caching

A step beyond exact-match: use embeddings to find similar previous questions and serve cached answers when the similarity is high enough.

# Store each question's embedding alongside its cached answer
def semantic_cache_lookup(question: str, threshold: float = 0.92):
    q_vec = embed(question)
    hits = vector_store.search(q_vec, top_k=1)
    if hits and hits[0]["score"] > threshold:
        return hits[0]["answer"]
    return None

If a user asks “How do I renew my citizenship?” and the cache has “How can I renew my citizenship certificate?” at similarity 0.94, serve the same answer.

Careful: semantic caching can silently serve wrong answers if the threshold is too low. Test rigorously. A safe threshold for Nepali FAQ data is often 0.90-0.94 — but validate on your own queries.

Technique 2 — Prompt caching (Anthropic)

Anthropic’s API supports prompt caching: mark parts of your prompt as cacheable, and Anthropic charges much less to re-send them on subsequent calls (about 10× cheaper on the input side for cached tokens).

Use case: your system prompt and retrieved documents are the same across 20 calls in a conversation. Marking them cacheable means Anthropic only fully processes them once.

response = client.messages.create(
    model="claude-3-5-sonnet-latest",
    max_tokens=500,
    system=[
        {
            "type": "text",
            "text": SYSTEM_PROMPT,
            "cache_control": {"type": "ephemeral"},
        },
        {
            "type": "text",
            "text": f"Documents:\n{retrieved_context}",
            "cache_control": {"type": "ephemeral"},
        },
    ],
    messages=[{"role": "user", "content": user_question}],
)

When prompt caching pays off:

• Long, static system prompts (>1,024 tokens for Anthropic’s threshold).
• Long, static retrieved context reused across many calls (e.g., a whole document as context).
• Multi-turn conversations where the history keeps growing.

When it doesn’t:

• Short prompts (the caching overhead exceeds the savings below 1,024 tokens).
• Every-call-is-different workloads.

For a RAG bot with a 400-token system prompt and 3,000 tokens of retrieved context, prompt caching cuts input cost by 8-10× on cache hits. If the same context is reused across a 10-turn conversation, that’s a huge saving.

Technique 3 — Embedding caching

Embeddings are cheap ($0.02 per million tokens for text-embedding-3-small), but they add up. If you re-embed the same query many times a day, cache it.

def embed_cached(text: str) -> list[float]:
    key = "emb:" + hashlib.sha256(text.encode()).hexdigest()
    cached = r.get(key)
    if cached:
        return json.loads(cached)
    v = openai_client.embeddings.create(
        model="text-embedding-3-small",
        input=text,
    ).data[0].embedding
    r.setex(key, 7 * 86400, json.dumps(v))   # cache 7 days
    return v

More important: cache your document embeddings on disk, permanently. You embed a document once when ingesting it, then never again. If your ingestion pipeline re-embeds the same content on every deploy, you are paying twice for the same work.

The economics of caching

Order-of-magnitude cost impact on a typical Nepali policy Q&A bot:

Layer	No caching	With caching	Savings
Model calls (LLM)	$2400	$960	$1,440 (60% off)
Embedding calls	$50	$10	$40 (80% off)
Vector store queries	$0	$0	0
Redis cache infra	$0	$10
Net monthly cost	$2450	$980	$1,470/month

Rs ~190,000/month saved on a Rs ~320,000/month baseline. The Redis cost is a rounding error.

Rule of thumb: if your service handles more than a few hundred queries per day and has any pattern of repetition, response caching alone will more than pay for itself the first month.

Cache invalidation

The hard part. Two rules:

1. Time-based expiry (TTL) always. No cache entry lives forever. A 24-hour TTL on FAQ answers means yesterday’s cached wrong answer is gone by today. Choose the TTL to match how often your underlying data changes.

2. Manual invalidation on document updates. When you update the underlying documents (a new HR policy, a corrected FAQ), invalidate all cached answers derived from them. If your cache keys hash in the document IDs, this is straightforward. If not, add a “cache version” prefix and bump it on updates.

Do not skip either. A stale cache serves confidently wrong answers, which is worse than making a fresh call.

What to cache vs. what not to

A useful checklist:

Cache freely:

• FAQ / Q&A responses over stable documents.
• Deterministic transformations (translation, summarisation of static content, classification).
• Document embeddings (permanently, on disk).

Cache carefully (with tight TTLs or semantic thresholds):

• Answers that combine document data with a small amount of user context.
• Popular queries in a conversation.

Do not cache:

• Personalised outputs (user’s account, transactions, portfolio).
• Time-sensitive answers (stock prices, weather, news).
• Creative content where each request should be fresh.
• Anything the user should never see recycled from another user.

The caching audit

Once a month, look at your cache stats:

• Hit rate. Of all requests, what fraction was served from cache? Aim for 30-60% for FAQ-style products.
• Cost saved. Multiply cache hits by average cost per call to estimate savings.
• Wrong-answer complaints on cached responses. These are the strongest signal of a caching bug.

Cache health is a metric worth watching, alongside error rates and latency.

What comes next

You’ve squeezed cost. But now you have a system that is often silent — you don’t know when it’s serving well, when it’s failing, or when a user is having a bad time. Chapter 3 is about seeing your system: logs, metrics, and the discipline of knowing what your service is actually doing right now.