Handling failure gracefully

The four failure patterns

Every LLM service needs four defenses:

1. Timeouts — the guarantee that no call takes forever.
2. Retries with backoff — the pattern for handling transient failures.
3. Fallbacks — what to do when the primary path is broken.
4. Honest user messages — the small kindness of telling users what happened.

Together they turn “the API failed” from a P0 incident into a temporarily degraded experience.

Pattern 1 — Timeouts

The single most-skipped pattern. Without a timeout, a slow model call can hold your worker for 60 seconds, tying up capacity and delaying every other user’s request. A dead API can block requests forever.

Set a hard timeout on every external call:

import httpx
from anthropic import Anthropic

# Anthropic SDK accepts a timeout parameter
client = Anthropic(timeout=30.0)  # 30-second timeout

response = client.messages.create(
    model="claude-3-5-sonnet-latest",
    max_tokens=500,
    messages=[...],
    # per-call override:
    # timeout=15.0,
)

For most LLM calls, 30-60 seconds is a reasonable cap. If your service needs longer (very long documents, image analysis), lift the cap for that endpoint but keep it bounded.

Also set a whole-request timeout in your web server. For FastAPI + uvicorn:

uvicorn service:app --timeout-keep-alive 60

Pattern 2 — Retries with backoff

Some failures are transient. The API returned a 503, or the request timed out. Retrying often succeeds. But not every failure should be retried:

• Retry: 429 (rate limit), 500-599 (server errors), 408 (timeout).
• Don’t retry: 400 (bad request — retry won’t help), 401/403 (auth), 404 (not found), 422 (validation).

The right way to retry — with exponential backoff and jitter (from Course 04 Section 2.3):

import random
import time
from anthropic import APIError, APIStatusError

def call_with_retry(fn, max_retries=3):
    for attempt in range(max_retries):
        try:
            return fn()
        except APIStatusError as e:
            if e.status_code not in {429, 500, 502, 503, 504}:
                raise  # non-retryable
            if attempt == max_retries - 1:
                raise
            wait = 2 ** attempt + random.uniform(0, 1)
            time.sleep(wait)
        except APIError:
            if attempt == max_retries - 1:
                raise
            wait = 2 ** attempt + random.uniform(0, 1)
            time.sleep(wait)

Anthropic and OpenAI SDKs both have built-in retry — but they use fixed retry counts, and mixing your own retry with the SDK’s often produces surprising delays. Choose one. If you use SDK retries, set max_retries explicitly.

Retries add latency. A single call that took 2 s becomes a 2 s + 4 s + 8 s = 14 s call after two retries. Users notice. Combine retries with a per-request timeout so the total time is bounded.

Pattern 3 — Fallbacks

When the primary path is broken, what do you offer instead?

Provider fallback. Anthropic is down, so route to OpenAI. Requires two SDKs and equivalent prompts, but it protects you from provider outages:

def call_llm(prompt: str) -> str:
    try:
        return call_anthropic(prompt)
    except (APIError, TimeoutError):
        log_event("provider_fallback", from_="anthropic", to="openai")
        return call_openai(prompt)

Model fallback. Sonnet is timing out; degrade to Haiku (faster, cheaper, slightly lower quality). Users get an answer; you accept a small quality hit during the incident.

Static fallback. For pure Q&A over a small FAQ, when the model is down, fall back to keyword search over the FAQ and return the top match. Users get useful information; you get zero cost.

Explicit degradation message. If none of the above are viable, return a clear message: “Our AI is having trouble right now. Please try again in a few minutes.”

Every service benefits from at least the last one. Provider and model fallbacks are worth it for revenue-critical products.

Pattern 4 — Honest user messages

When something fails and you can’t fully recover, tell the user what happened. Compare:

Bad:

Error 500. Something went wrong.

Bad in a different way:

An error occurred while processing your request. Please contact
support with error code: NUL-4192-OP-8283-FAIL-99.

Good:

Our AI assistant is running slowly right now. Please try again in
a minute. If this keeps happening, let us know at
support@nepse-mitra.np and we'll fix it fast.

Three parts of a good failure message:

1. What happened, in plain language.
2. What the user should do (try again, wait, contact us).
3. A way to reach a human, if it matters.

For Nepali users, offer the message in both Nepali and English:

हाम्रो AI सहायक अहिले बिस्तारै चलिरहेको छ। एक मिनेटमा फेरि प्रयास गर्नुहोस्।

Our AI assistant is running slowly right now. Please try again in a minute.

The bilingual message signals care. It costs nothing to produce and it retains users.

The layered defence

Putting the four patterns together for a real endpoint:

@app.post("/summarise")
def summarise(req: SummariseRequest, user_id: str = Depends(get_current_user)):
    request_id = str(uuid.uuid4())

    # Boundary checks (from Section 1.2)
    if len(req.text) > MAX_INPUT_CHARS:
        raise HTTPException(400, "Text too long.")

    if not check_and_record_usage(user_id):
        raise HTTPException(429, "Daily quota reached.")

    # The main path, with defence
    try:
        summary = call_with_retry(
            lambda: call_llm_with_timeout(build_prompt(req), timeout=30)
        )
    except (APIError, TimeoutError) as e:
        log_event("primary_failed", request_id=request_id, error=str(e))

        # Fallback: try a lighter model
        try:
            summary = call_llm(build_prompt(req), model=FALLBACK_MODEL)
            log_event("fallback_used", request_id=request_id, model=FALLBACK_MODEL)
        except Exception:
            # Full failure — return an honest error
            raise HTTPException(
                status_code=503,
                detail="Our AI is having trouble right now. Please try again in a few minutes.",
            )

    return SummariseResponse(summary=summary, input_length=len(req.text))

Every layer has a purpose. Every failure has a recovery path. The user only sees the honest error message if every fallback has been tried.

Common failure modes and their fixes

A short catalogue:

Model returns garbled or empty output.

• Fix: validate before returning. If the output is empty or fails a basic sanity check, treat it as a failure and retry (or fall back).

Retrieval returns nothing.

• Fix: if top_score is below a threshold, tell the user “I don’t know” instead of feeding an unrelated document to the LLM.

Prompt is longer than the context window.

• Fix: check token count before the call; truncate history or documents; log the truncation so you can look for chronic offenders.

User’s IP address is somewhere the provider doesn’t allow.

• Fix: proxy the request through your service (which is on an allowed IP). Never expose provider API keys to the client.

Provider deprecates the model you were using.

• Fix: monitor provider deprecation notices; test candidate replacements ahead of time; version your model choice (Chapter 5).

Each of these has hit real projects. Each is preventable with a small amount of defensive design.

The reliability mindset

At its root, graceful failure is a mindset: assume things will go wrong, and make sure the failure mode is okay. Not perfect. Not invisible. Just okay.

• Slow API? Timeout, retry, fall back, honest message.
• Bad input? Reject cleanly at the boundary.
• Model down? Fall back to a smaller model; if that fails, explain and move on.
• User confused? Log the interaction and improve the FAQ.

None of these prevent failure. All of them prevent failure from being a catastrophe. Users tolerate momentary degradation. They do not tolerate mysterious silence, confusing errors, or systems that just disappear.

Check your understanding

Quick check

—

A team ships a chatbot without an LLM timeout. The provider has a slow morning, and calls that used to take 2 seconds now take 45 seconds. What is the systemic consequence?

1. ANothing — users just wait a bit longer.
2. BWorkers get held on slow calls, so concurrency drops sharply. With 4 workers each held for 45 seconds, the service can only handle ~5 requests per minute per worker — new users see growing queue latency, then eventually 502 gateway errors when the queue is full. The absence of a timeout turns 'the provider is slow' into 'the service is down.' The fix is a hard timeout (30 s) on every model call, combined with retries and a fallback, so no single slow call can cascade into a service-wide outage.
3. CThe service will crash immediately.
4. DThe provider will refund the wasted time.

Check answer→

Quick check

—

Which of these is the correct policy for retrying a failed LLM call?

1. ARetry every failure immediately, up to 10 times.
2. BRetry only *transient* failures (429 rate limit, 500-504 server errors, timeouts), with exponential backoff and jitter, capped at 2-3 retries. Non-retryable errors (400 bad request, 401/403 auth, 422 validation) should not be retried — a retry won't fix a bad input. Bound the total time budget with a per-request timeout so retries don't turn a 2-second call into a 30-second one. The pattern is: identify the error class, apply the right recovery strategy, bound the total time.
3. CNever retry; return an error to the user immediately.
4. DRetry with the same request every second forever.

Check answer→

What comes next

Your service is reliable. It stays up under bad conditions. Now the harder question: is it good? Chapter 4 begins the discipline of evaluating your AI system in production — knowing whether your model is actually giving people the right answers, and knowing quickly enough to fix it when it’s not.