Handling errors, limits, and bad responses

The three families of failure

Every failure your LLM code will see falls into one of three categories:

1. Transient failures — the network is slow, the API is briefly overloaded, a 5xx error comes back. Fix: retry with backoff.
2. Rate limit failures — you’ve called the API too fast; the provider is telling you to slow down. Fix: back off exponentially and respect their guidance.
3. Semantic failures — the call succeeded but the response is wrong. Fix: validate, log, and (sometimes) retry with a stronger model or a different prompt.

Each has different code and different economics. Learn them separately.

Transient failures and retry

For transient failures, the standard pattern is exponential backoff with jitter:

import time
import random
import anthropic
from anthropic import APIError, APIConnectionError, APITimeoutError

RETRIABLE_ERRORS = (APIConnectionError, APITimeoutError)


def call_with_retry(fn, max_attempts: int = 5, base_delay: float = 1.0):
    """Call fn(), retrying transient errors with exponential backoff."""
    for attempt in range(1, max_attempts + 1):
        try:
            return fn()
        except RETRIABLE_ERRORS as e:
            if attempt == max_attempts:
                raise
            # Exponential backoff: 1s, 2s, 4s, 8s, ... with jitter
            delay = base_delay * (2 ** (attempt - 1)) + random.uniform(0, 0.5)
            print(f"Attempt {attempt} failed with {type(e).__name__}: {e}. "
                  f"Retrying in {delay:.1f}s...")
            time.sleep(delay)

Usage:

def call_llm():
    return client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=400,
        temperature=0.0,
        messages=[{"role": "user", "content": "Say hello in Nepali."}],
    )


response = call_with_retry(call_llm)

Three key ideas in the retry code:

• Retry only transient errors. Retrying a 400 (bad request) or 401 (auth failure) is pointless; those need code fixes, not more calls.
• Exponential backoff. Each retry waits longer than the last, so you don’t hammer a struggling API.
• Jitter. Adding a small random delay prevents all your instances from retrying at exactly the same moment, which would make a bad situation worse.

Rate limits, specifically

When the API returns 429 Too Many Requests, the response usually includes a retry-after header telling you exactly how long to wait. Anthropic’s SDK exposes this:

from anthropic import RateLimitError

def call_with_rate_limit_handling(fn, max_attempts: int = 5):
    for attempt in range(1, max_attempts + 1):
        try:
            return fn()
        except RateLimitError as e:
            if attempt == max_attempts:
                raise
            # Prefer the API's suggested wait time; fall back to exponential backoff
            retry_after = float(getattr(e, "retry_after", None) or 2 ** attempt)
            print(f"Rate limited. Waiting {retry_after:.1f}s...")
            time.sleep(retry_after)

Respect the retry-after header when it’s there. If it’s not, back off exponentially. If you’re hitting rate limits routinely, that’s a signal to either upgrade your API tier, add caching, or use a smaller model with higher throughput limits.

Timeouts

The Anthropic SDK has a default timeout, but you should set an explicit one for anything user-facing:

client = anthropic.Anthropic(timeout=15.0)   # 15 seconds default for all calls

# Or per-call:
response = client.with_options(timeout=30.0).messages.create(...)

For interactive UIs, 15-30 seconds is generous but not so long that a stuck request wedges your app. For background jobs, longer timeouts (60-120 seconds) are reasonable — but always finite.

Semantic failures

The call returned. The JSON parsed. The pydantic validation passed. But the response is wrong — the model hallucinated, misclassified, or refused politely. This is the hardest failure mode because your code has no direct signal.

Three defences:

1. Log everything. For production LLM calls, log the request, response, and any validation results. Later, when a user reports a bad answer, you can look at what actually happened. Without logging, LLM bugs are almost impossible to debug.

logger.info(
    "llm_call",
    prompt=prompt[:500],   # truncate for privacy
    response_text=response.content[0].text[:500],
    input_tokens=response.usage.input_tokens,
    output_tokens=response.usage.output_tokens,
    stop_reason=response.stop_reason,
    duration_ms=elapsed_ms,
)

2. Validate against sanity checks. If your extraction returns amount_npr = 0 on 5% of receipts, something is wrong — receipts don’t have zero amounts. Sanity checks catch this:

if fields.amount_npr == 0:
    logger.warning("suspicious_extraction", raw_text=receipt_text)

3. Have a fallback. For critical paths, if the small model fails or produces a suspicious output, retry with a stronger model:

def extract_receipt_robust(text: str) -> Optional[ReceiptFields]:
    result = extract_receipt(text, model="claude-haiku-4-5-20251001")
    if result is None or result.amount_npr == 0:
        # Escalate to a stronger model
        result = extract_receipt(text, model="claude-sonnet-4-6")
    return result

This is expensive (Sonnet is 12x pricier than Haiku), so use it sparingly — but for cases where correctness matters, escalation is the standard pattern.

The graceful refusal case

Sometimes the model politely refuses to answer — usually for safety reasons, but occasionally because your prompt was ambiguous:

"I'd be happy to help, but I need more context. Could you..."

Your JSON parser will fail. Your pydantic validator will fail. And the user, in production, will see nothing.

The fix is to detect refusals explicitly in your parser:

REFUSAL_MARKERS = [
    "I'd be happy to help",
    "I'm not able to",
    "I cannot",
    "I don't have information about",
]

def looks_like_refusal(text: str) -> bool:
    return any(m.lower() in text.lower() for m in REFUSAL_MARKERS)

def parse_response(text: str):
    if looks_like_refusal(text):
        raise ModelRefusalError(text)
    return parse_llm_json(text)

Now refusals are a named failure mode you can handle intentionally — usually by escalating to a human or explaining to the user that the model can’t help with this specific request.

A wrapper that ties it all together

For every serious LLM application, you’ll want a call_llm function that handles retries, timeouts, rate limits, and returns clean results:

import time
import logging
from typing import Callable
from anthropic import APIError, APIConnectionError, APITimeoutError, RateLimitError


logger = logging.getLogger(__name__)


class LLMError(Exception): pass
class ModelRefusalError(LLMError): pass
class LLMBudgetExceeded(LLMError): pass


def safe_call(fn: Callable, max_attempts: int = 5) -> object:
    for attempt in range(1, max_attempts + 1):
        try:
            start = time.time()
            result = fn()
            duration = time.time() - start
            logger.info(f"LLM call ok (attempt {attempt}, {duration:.2f}s)")
            return result

        except RateLimitError as e:
            retry_after = float(getattr(e, "retry_after", None) or 2 ** attempt)
            logger.warning(f"Rate limited. Sleeping {retry_after}s.")
            time.sleep(retry_after)

        except (APIConnectionError, APITimeoutError) as e:
            delay = 2 ** (attempt - 1)
            logger.warning(f"Transient error: {type(e).__name__}. Retrying in {delay}s.")
            time.sleep(delay)

        except APIError as e:
            # Non-retriable API error (auth, bad request, etc.)
            logger.error(f"Non-retriable API error: {e}")
            raise

    raise LLMError(f"Failed after {max_attempts} attempts.")

Every production LLM function you write should call the API through safe_call. Once, at the boundary. Everything downstream can assume the call either worked or raised a specific exception it knows how to handle.

Cost as a hard failure

For any production system, put a hard cap on cost. A runaway loop or a stuck script can burn hundreds of dollars in minutes:

class BudgetGuard:
    def __init__(self, monthly_budget_usd: float):
        self.budget = monthly_budget_usd
        self.spent = 0.0

    def charge(self, cost: float):
        self.spent += cost
        if self.spent > self.budget:
            raise LLMBudgetExceeded(f"Spent ${self.spent:.2f} of ${self.budget:.2f}")

guard = BudgetGuard(monthly_budget_usd=200)

def call_and_track(fn):
    response = safe_call(fn)
    cost = compute_cost(response)
    guard.charge(cost)
    return response

The budget guard catches runaway usage before it becomes a real bill. Combined with the provider’s console alerts, it’s cheap insurance.

Check your understanding

Quick check

—

A production LLM app started returning garbage responses last night, but the code hasn't changed. Investigation shows the calls succeeded — no exceptions raised. What's the missing habit that would have caught this earlier?

1. ABetter exception handling.
2. BLogging of LLM calls (prompt, response text, tokens, duration). Without logs of the actual model responses, you can't tell whether the model started returning wrong output or the downstream code silently changed. Every production LLM call should log its request and response (redacted for PII) so you can debug 'what did the model actually say?' after the fact. This is the single highest-leverage operational habit in LLM engineering.
3. CSwitch models immediately.
4. DAdd more retries.

Check answer→

Quick check

—

A junior engineer reports 429 rate-limit errors and asks whether they should rotate API keys to avoid them. What's the right response?

1. AYes — rotating keys is a standard technique.
2. BNo — rate-limit rotation is against the provider's terms and will get the account banned. The correct responses are: (1) implement exponential backoff and respect the retry-after header; (2) reduce the call rate through caching, batching, or smaller models with higher throughput limits; (3) if the workload legitimately requires higher throughput, upgrade the API tier — it's much cheaper than the reputational cost of getting flagged for abuse.
3. CIgnore the errors.
4. DSend calls from multiple IP addresses.

Check answer→

What comes next

Chapter 2 is done. You can write production prompts, get structured JSON, and handle the real failure modes. Chapter 3 addresses the next big architectural piece: memory. LLMs forget between calls. Multi-turn conversations require you to explicitly manage the history. And building a real chatbot means confronting the token budget of a growing conversation. By the end of Chapter 3 you’ll have a working Nepali chatbot with careful memory management.