Feedback loops and improvement

The feedback signals

Two categories: explicit and implicit.

Explicit signals — what users tell you on purpose:

• Thumbs up / thumbs down.
• Star ratings (1-5).
• A “report an issue” button with a text field.
• Direct emails or Slack messages.

Implicit signals — what users tell you by their behaviour:

• Session length and re-engagement rate.
• Rephrasing rate (user asked again in different words = first answer was bad).
• Copy-paste rate (user copied the answer = they trusted it enough to use).
• Bounce rate (user closed the tab after one query = something was wrong).

Both are useful. Explicit is precise but low-volume; implicit is high-volume but noisy. Collect both.

Thumbs — the basic instrument

The single highest-ROI feedback instrument is a thumbs-up/thumbs-down button. Small, easy to click, actionable.

@app.post("/feedback")
def feedback(req: FeedbackRequest, user_id: str = Depends(get_current_user)):
    log_event("feedback",
              request_id=req.request_id,
              thumbs=req.thumbs,       # "up" or "down"
              comment=req.comment,     # optional
              user_id=user_id)
    return {"ok": True}

In the UI, place the buttons alongside every AI-generated answer. Do not gate them behind a modal. Do not require an account (if you can help it).

Add an optional comment field on thumbs-down. Most people won’t fill it. The few who do give you gold.

Reading feedback well

A pile of thumbs-down is useless if no one reads it. The routine:

Weekly review, 30 minutes.

One person on the team spends half an hour going through the thumbs-down responses from the past week. For each:

1. Read the original query.
2. Look at the retrieved documents.
3. Look at the model’s answer.
4. Categorise the failure:
   • Retrieval got the wrong document.
   • Retrieval got the right document but the model ignored it.
   • Correct document, correct answer, user was still unhappy (why?).
   • Model refused when it shouldn’t have.
   • Prompt injection or abuse.

Log the categorisation in a simple spreadsheet. Over a month, patterns emerge. If 40% of thumbs-down are “retrieval got the wrong document,” you have a retrieval problem. If 40% are “correct but tone was wrong,” you have a prompt problem.

Fix in batches.

Every 2-4 weeks, take the top failure category and fix it. Ship the fix behind an A/B test. Watch the thumbs-down rate.

This loop — collect, categorise, fix, measure — is the mechanism by which AI products actually get better. Skipping it means shipping the same product forever, mediocre, no matter how carefully you launched.

Slice-by-slice patterns

Once you have a few weeks of feedback, slice it:

• Language. Thumbs-down rate on Nepali vs. English. Wide gaps signal language-specific quality issues.
• Time of day. Thumbs-down higher at certain times? Might be latency-driven, or usage-pattern-driven.
• User cohort. New users vs. returning users have different thumbs-down rates. Investigate why.
• Query category. Categorise queries (FAQ, complex, out-of-scope). Which category gets the most complaints?

Sometimes the pattern is unexpected — e.g. “engineering-role queries have 3× the thumbs-down rate of HR queries” — and reveals a gap in retrieval or coverage.

Handling direct user reports

When a user emails or messages you about a bad answer, it deserves personal follow-up:

1. Look up the request_id. From your logs, reconstruct exactly what happened.
2. Reply within 24 hours. “We looked at your query. Here’s what happened. Here’s what we’re fixing.”
3. Add the query to the eval set. So the fix protects against future regression.
4. Thank them. Users who take the time to report a bug are giving you a gift. Treat them as such.

Response quality here has outsized reputation effects. A user who complained and got a thoughtful reply becomes an advocate. A user who complained into the void becomes an ex-user.

The Nepali user in the feedback loop

Nepali-language feedback is often thinner in volume but disproportionately valuable in signal:

• Users writing feedback in Nepali are often more engaged than English-only users.
• Their pain points reveal Nepal-specific issues (dialect coverage, cultural context, script normalisation) that English-only testing would miss.
• Nepali-language complaints deserve Nepali-language replies — a small courtesy that dramatically improves the response experience.

Make sure your feedback intake accepts Devanagari input. Make sure your reply flow can send Devanagari back. These small technical details are often surprisingly overlooked in Nepal-facing projects.

The retention lens

Thumbs are lightweight. The deeper feedback signal is whether users come back.

Track for each user:

• Days since first use.
• Total sessions.
• Sessions per week.
• Time to first repeat use.

If a user tries the product once and never returns, something was wrong on the first try — even if they never left explicit feedback. Bucketing users by “one-visit” vs. “returning” and looking at their queries reveals implicit dissatisfaction.

For a Nepal-facing bot, watch retention among Nepali speakers separately from English speakers. Different retention curves reveal different audience experiences.

The improvement backlog

All the feedback, all the failure categories, all the reported bugs — they go into one backlog. The backlog has three columns:

Frequency  |  Difficulty  |  Fix idea
   40%     |    Low       |  Improve chunk boundaries in HR policy PDFs
   25%     |    Medium    |  Add a retrieval reranker
   15%     |    High      |  Retrain the embedding model on Nepal-specific data
   10%     |    Low       |  Fix the "I don't know" over-refusal on ambiguous queries
   10%     |    Medium    |  Tone adjustment for Nepali replies

Sort by expected impact (frequency × severity) divided by difficulty. Work top-down. Ship one improvement at a time, each behind an A/B test where feasible. Measure the effect on the thumbs-down rate.

This is not a research plan. It is a small ordered list that the team executes over months.

The virtuous loop

When it’s working, the loop looks like this:

Users interact
   ↓
Feedback (thumbs, comments, behaviour)
   ↓
Weekly review, categorise failures
   ↓
Prioritise fixes
   ↓
A/B test the fix
   ↓
Ship if metrics improve
   ↓
Users interact with the better version
   ↓
(loop)

Every rotation of the loop makes the product measurably better. Every skipped week is a compounding tax on the product’s future. The teams that ship enduring AI products are the ones that treat this loop as sacred, not the ones that ship a great launch and hope.

What comes next

You have the feedback loop. But every change you ship is also a change to a real, running system with real users. Chapter 5 is about the discipline of safely rolling changes out — versioning prompts, models, and data; testing in staging; rolling back when something breaks; and handling the special class of failures where the model provider itself changes underneath you.