Logging every interaction · Shipping AI Systems

Every mystery in a live AI system — “why did it say that?” “why is it slow?” “why did this user get a wrong answer?” — is solvable if you have the logs and unsolvable if you don’t. Logging is not glamorous; it is the substrate every other operational discipline is built on. This section is the small, unremarkable habit that separates services that improve over time from services that quietly degrade.

The minimum log

For every request that hits your service, log at least:

Timestamp (with timezone).
Request ID — a UUID that ties together everything about this one call.
User ID — who is asking.
Endpoint — which route.
Input — the user’s question, plus any relevant parameters.
Retrieval — which chunks were retrieved (IDs + scores).
Model call — which model, tokens in, tokens out, cost estimate.
Output — what your service returned.
Latency — how long it took, broken into stages if possible.
Status — success, error, refusal, quota hit.

If you have all of that, you can answer almost every operational question after the fact.

Structured logs, always

Never dump lines of print() in production. Log as JSON:

import logging
import json
import time
import uuid


logger = logging.getLogger("service")
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("%(message)s"))
logger.addHandler(handler)
logger.setLevel(logging.INFO)


def log_event(event: str, **fields):
    payload = {
        "event": event,
        "ts": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
        **fields,
    }
    logger.info(json.dumps(payload, ensure_ascii=False))

In your endpoint:

@app.post("/summarise")
def summarise(req: SummariseRequest, user_id: str = Depends(get_current_user)):
    request_id = str(uuid.uuid4())
    started = time.time()

    log_event("request_start",
              request_id=request_id,
              user_id=user_id,
              endpoint="/summarise",
              input_length=len(req.text))

    try:
        result = do_the_work(req)
    except Exception as e:
        log_event("request_error",
                  request_id=request_id,
                  error_type=type(e).__name__,
                  error_message=str(e))
        raise

    log_event("request_success",
              request_id=request_id,
              latency_ms=int((time.time() - started) * 1000),
              output_length=len(result.summary),
              tokens_input=result.tokens_input,
              tokens_output=result.tokens_output,
              cost_usd=result.cost_usd)

    return result

Structured JSON means you can query later: logs | filter user_id=nishan | count in a log aggregator (Datadog, Grafana Loki, or even grep and jq).

What NOT to log

Just as important as what to log:

Never log unhashed PII in plaintext. If your service receives phone numbers, national ID numbers, or full names, do not log the raw values.

Instead: hash the sensitive fields (hashlib.sha256) so you can group by user without exposing identity.
Or: log only a truncated/masked version ("9841***456").

Never log full API keys or auth tokens. If an auth token appears in the request, log a truncated version ("tok_abcd...").

Never log raw payment data. Regulated in most jurisdictions; risky in all.

Be careful with the user’s full question if it may contain personal data. For a general chatbot, log it. For a mental health app, consider redacting or hashing.

Retention and storage

Two axes: how long you keep logs, and where.

Retention.

Real-time logs (for debugging): 24 hours to 7 days, in-memory or hot storage.
Structured event logs (for analytics, evaluation): 30-90 days in cheap storage.
Regulated logs (financial, health): whatever local regulations require. For NEPSE/BFI-facing products, consult a compliance advisor.

Storage.

Small scale: stdout piped to a log aggregator (Fly.io, Railway, Render all include one).
Medium scale: managed service like Datadog, Better Stack, Grafana Cloud.
Large scale: self-hosted Grafana + Loki, or Elasticsearch.

At Nepal-scale (thousands to tens of thousands of interactions per day), the managed platform’s built-in logging is usually enough for months. Do not over-engineer.

Tracing across the request

For AI systems with multiple stages (retrieval → model → post-processing), a trace connects them. Every stage logs the same request_id, so you can reconstruct the full life cycle of one call.

def do_the_work(req, request_id):
    t0 = time.time()

    log_event("retrieval_start", request_id=request_id)
    hits = search_index(req.text, top_k=3)
    log_event("retrieval_done",
              request_id=request_id,
              latency_ms=int((time.time() - t0) * 1000),
              top_score=hits[0]["score"],
              doc_ids=[h["doc_id"] for h in hits])

    t1 = time.time()
    log_event("model_call_start", request_id=request_id)
    response = client.messages.create(...)
    log_event("model_call_done",
              request_id=request_id,
              latency_ms=int((time.time() - t1) * 1000),
              model="claude-3-5-sonnet-latest",
              tokens_input=response.usage.input_tokens,
              tokens_output=response.usage.output_tokens)

    ...

Now, for any given request_id, you can see: how long retrieval took, which docs it returned, how long the model took, what the model produced. Debugging a slow request becomes a matter of finding the stage that took long.

The audit log

For AI systems that make decisions on behalf of users (a policy Q&A bot, a financial assistant, a medical triage tool), keep a permanent audit log — a separate append-only store of every interaction, retained longer than operational logs.

Fields:

request_id, user_id, timestamp
Full input
Retrieved documents (IDs + scores)
Full output
Model version, prompt version

Purpose: when a user says “the bot gave me wrong information,” you can look up exactly what happened. This protects the user, and it protects you.

An audit log lives outside the ephemeral 30-day retention window. Store it in cheap, append-only storage (S3, GCS, or a local WORM store). Six months to a few years, depending on your product’s risk profile.

The five things logs give you

Well-structured logs enable everything downstream:

Debugging. “This user got a wrong answer” becomes a fact you can investigate, not a claim you have to guess about.
Performance. “Requests were slow at 2 PM” becomes a query, not a mystery.
Evaluation. Logs are the raw material for the continuous-evaluation loop (Chapter 4).
Cost analysis. “Which endpoint costs the most?” is answerable from log-level cost fields.
Product improvement. “What are users actually asking?” is answerable from input logs.

Every one of these is impossible without logs. Every one of them compounds over months of accumulated data.

Log discipline

Three habits worth building:

Log at every important boundary. Every request start, every stage transition, every failure. Not one every function call — but every meaningful event.
Log dimensions, not narratives. {"latency_ms": 1240, "endpoint": "/summarise"} is queryable. "took a long time to summarise" is not.
Look at your logs weekly. Even a scan of 20 recent requests reveals patterns. What questions are users asking? What’s slow? What’s failing? The habit of reading logs is where operational insight lives.

Check your understanding

Quick check

—

A user reports their chatbot gave them completely wrong information about a specific policy an hour ago. Which of these enables you to actually investigate?

Trust the user and update the FAQ.
Structured logs that include the request ID, user ID, timestamp, full input, retrieved documents (with IDs and similarity scores), full output, and model version. Given those, you can reconstruct exactly what happened: which chunks were retrieved, why the model latched onto them, and where in the pipeline the failure lived. Without those, you're guessing. Logging every interaction with structured fields turns 'someone said the bot was wrong' from a mystery into a solvable investigation.
Ask the user for a screenshot.
Run the same query and see if it fails again.

Quick check

—

Which of the following should you NOT log in plaintext for a Nepali chatbot?

Request timestamps.
The user's raw national ID number, phone number, or auth token. These are PII (personally identifiable information) or credentials, and logging them in plaintext creates a compliance risk and a leak surface. The right pattern is: hash sensitive fields for grouping (sha256), or mask them ('9841***456'), or leave them out of logs entirely. Every log line is a small privacy decision; PII belongs in the encrypted data store, not the operational log stream.
Endpoint names.
Model version.

What comes next

You have logs. Great. But logs are a haystack — needles hide in them. The next section is metrics and dashboards: the small number of aggregated signals that let you see the health of your system at a glance, and set alarms for when something goes wrong.