Metrics, alerts, and dashboards

The four core metrics

For an LLM service, four dimensions cover most of what you need to know:

1. Latency — how long is each request taking, at p50 and p95?
2. Error rate — what fraction of requests fail?
3. Cost — what are we spending per hour, per user, per endpoint?
4. Quality — are the answers good?

Latency and errors are technical. Cost is economic. Quality is what the user actually cares about — and the hardest to measure well.

Latency

For every endpoint, track:

• p50 (median) — the typical experience.
• p95 — the slower 5% of requests. Where “the app feels laggy” lives.
• p99 — the tail. Rare, but the worst case a user might hit.

For an LLM Q&A service, healthy targets:

• p50: 1-2 s
• p95: 3-5 s
• p99: <10 s

If p95 starts creeping above 5 s, something has changed — retrieval got slower, the model provider is having a rough day, or your service is under load. Investigate.

Latency, broken down by stage, is even more useful. If total latency is 4 s and retrieval was 200 ms and the model was 3.6 s, the model is the bottleneck. If retrieval was 3 s, your vector store is the problem.

# collect stage-level latencies
log_event("latency", request_id=id, stage="retrieval", ms=210)
log_event("latency", request_id=id, stage="model", ms=1840)
log_event("latency", request_id=id, stage="post", ms=25)

Aggregate these into per-stage p50/p95 dashboards.

Error rate

Every request either succeeds, errors, or is rejected (quota, validation, rate limit). Track the fractions:

• Success rate — should be 95%+ for healthy services.
• 5xx rate — server errors. Should be <1%. Any spike is an incident.
• 4xx rate — client errors (bad input, quota, auth). Depends on your traffic; watch for sudden changes.
• Refusal rate — model refused to answer, or the assistant said “I don’t know.” Track separately; a spike often means retrieval broke.

log_event("outcome", endpoint="/summarise", outcome="success")
log_event("outcome", endpoint="/summarise", outcome="5xx", error="APIError")
log_event("outcome", endpoint="/summarise", outcome="quota_exceeded")
log_event("outcome", endpoint="/summarise", outcome="refusal")

Then aggregate: outcome | count by outcome, endpoint, 5-minute-window. Chart it. Alert on 5xx rate > 1% for 5 minutes.

Cost

Cost is measured by summing your per-request cost estimates:

• Total cost per hour — is it what you expected?
• Cost per user per day — Rs 10 vs. Rs 100 is a whole different economic model.
• Cost per endpoint — where is the money going?
• Cache hit rate — how much are you saving from caching?

An unexpected cost spike is often the first sign of abuse, a bug, or a viral moment. Alert on any hour where cost exceeds 3× the recent baseline.

Quality — the hardest one

You can’t easily measure “is the answer good?” from logs alone. Proxies:

• User thumbs up/down rate. If you have a feedback button, thumbs-down rate is a direct quality signal.
• “I don’t know” rate. The rate at which your assistant falls back to a refusal. A rising rate usually means retrieval is breaking or user queries have shifted.
• Retrieval confidence. Distribution of top-score across all queries. A downward drift means your semantic search is degrading.
• Session-length. For chat products, session length can signal engagement. Very short sessions on a Q&A bot may mean the first answer was unhelpful.

Chapter 4 covers evaluation in production more thoroughly — this is the shortcut set of proxies for the dashboard.

Dashboards

Every AI service should have one dashboard, on one screen, that the team can glance at daily. Suggested layout:

┌──────────────────────────────────────────┐
│ nepse-mitra — production dashboard       │
├──────────────────────────────────────────┤
│ Latency (p50 / p95, per endpoint)         │
│ Error rate (%, past hour / past 24h)     │
│ Cost (Rs, this hour / this day / month)  │
│ Refusal rate (%, past 24h)               │
│ Cache hit rate (%, past 24h)             │
│ Thumbs down rate (%, past 24h)           │
│ Active users (past hour / 24h)           │
└──────────────────────────────────────────┘

Seven numbers. All visible at once. Anyone on the team can look, in 30 seconds, and say “we look healthy” or “something is off.”

Tooling options:

• Managed: Datadog, Grafana Cloud, Better Stack. Point them at your log stream; use their dashboard builder.
• Self-hosted: Grafana + Loki (logs) + Prometheus (metrics). More setup, more control.
• Minimal: a small Python script that reads yesterday’s logs from S3, aggregates, and emails the team every morning. Zero infrastructure. Surprisingly effective for a small project.

Start minimal. Upgrade only when you feel the pain of the current tool.

Alerts

An alert is a rule that fires a notification when a metric crosses a threshold. The rules for a good alert:

• Actionable. Someone can do something about it. “Latency is high” without any recovery path is noise.
• Specific. “5xx rate > 2% for 5 minutes on /summarise” is specific. “Something is wrong” is not.
• Rate-limited. One notification per incident, not one per minute.
• Routed sensibly. During work hours: Slack. Overnight: only page for genuine emergencies (service down, wallet fire).

Starter alert set:

• 5xx rate > 1% for 5 min → warning to Slack.
• 5xx rate > 5% for 5 min → page on-call.
• Cost rate > 3× baseline for 30 min → page on-call (potential wallet fire).
• p95 latency > 10 s for 15 min → warning to Slack.
• Thumbs-down rate > 15% for 24h → warning to Slack (quality regression).

Five alerts. All actionable. No noise for the sake of noise.

The Nepal-specific dashboard

For a Nepal-facing product, add a few local signals:

• Devanagari vs. Latin input ratio. How many queries are in Nepali vs. English? Trends here signal language-audience shifts.
• Regional user distribution (if geo data is available). Kathmandu vs. Pokhara vs. rural. Quality often varies by region because training data does.
• Time-of-day usage in NPT. When are your users active? Match capacity and cost caps to actual usage windows, not global averages.

These aren’t universally applicable, but for products where language / region / time-of-day matter, they surface problems specific to Nepali audiences that generic metrics miss.

The daily habit

Every morning, one person spends 5 minutes on the dashboard:

1. Are the numbers where they should be?
2. Are there any spikes or dips?
3. Any alerts overnight that need follow-up?

The person changes weekly. Everyone builds intuition for what “normal” looks like. When something shifts, the team notices before users do.

This one habit — 5 minutes of dashboard-reading each day — is the difference between a service that quietly rots and a service that stays healthy for years.

What comes next

You can see your system, and you can alarm when it breaks. Next: what to do inside your code when things break. Timeouts, retries, fallbacks — the patterns that turn a fragile service into a graceful one, so users see friendly errors instead of confusing stack traces.