4.2 Classification and tagging

Single-label classification

Classify the customer message into exactly one of:
- payment_failure
- kyc_or_verification
- login_or_auth
- app_crash
- other

Rules:
- Return only the slug. No preamble, no explanation.
- If the message matches more than one, pick the one that is causing the customer the most pain right now.
- If the message does not match any, return "other."

<message>
App keeps crashing when I try to send money to my brother's Khalti.
</message>

The pieces that make it work:

• Closed label set. Named explicitly, in a list, as slugs (not sentences).
• Return-only rule. “Return only the slug” — otherwise you get “The category is: app_crash.”
• Tie-breaker. Real messages often fit two labels. Say which wins.
• Explicit “other.” Without this, the model shoehorns edge cases into a wrong label.

Multi-label classification (tagging)

Same task, but each input can have multiple labels:

Tag the news article with every applicable tag from this set:
- politics
- economy
- education
- health
- sports
- climate
- kathmandu_metro
- infrastructure

Rules:
- Return a JSON array of tags. Do not invent tags outside the set.
- If no tag applies, return an empty array [].
- Return only the JSON. No preamble.

<article>
The Kathmandu Metropolitan City announced a NPR 240 million road-widening
project along Ring Road, funded from this year's municipal budget.
</article>

Expected output: ["kathmandu_metro", "infrastructure", "economy"]

The difference from single-label: politics and infrastructure might both apply; you want both.

Few-shot: the accuracy lever

For non-obvious labels, add examples:

Classify each ticket. Examples first, then the ticket to classify.

Ticket: "OTP not arriving on my phone even after 5 minutes."
Label: login_or_auth

Ticket: "Uploaded my citizenship photo but status still says pending after 3 days."
Label: kyc_or_verification

Ticket: "Sent money to wrong Khalti number by mistake, need refund."
Label: payment_failure

Ticket: "App keeps freezing after I open the transaction history."
Label: app_crash

Now classify:
<ticket>
Login screen shows a white page and never loads.
</ticket>
Label:

Two to five examples per label class is the sweet spot. More rarely helps.

Confidence and abstention

For anything you’ll route to a human, return not just the label but the confidence:

Return JSON with two fields:
- label: one of the slugs above
- confidence: "high" | "medium" | "low"

Rules:
- "high" = the message matches one label and nothing else.
- "medium" = the message probably matches one label but could plausibly be another.
- "low" = the message is unclear or matches nothing well.

Then in your routing code, “low” confidence goes to a human, “medium” gets logged for review, “high” goes to auto-resolution. Confidence from a model is not calibrated — you can’t trust “0.87” as a real probability — but coarse buckets like these are useful for routing.

Testing your classifier

You should treat a classification prompt like any classifier: build a small labelled test set (Chapter 5.1), run the prompt across it, and measure accuracy per label. Two symptoms and their fixes:

• One label is over-predicted. Its examples are too broad, or the description is too generic. Tighten it.
• The other bucket is either empty or huge. The prompt’s threshold for “does not match” is off — clarify what makes something not fit any specific label.

Check your understanding

Quick check

—

Which single change is most likely to improve accuracy on a classification prompt with four unusual project-specific labels?

1. AAdding "Please be accurate!" at the top.
2. BAdding 2–3 labelled examples per label, plus a tie-breaking rule.
3. CRemoving the "other" label so the model must pick one of the four.
4. DAsking the model to invent additional labels if none fit.

Check answer→