When not to use it

Most of this course has been about how to use generative AI well. This section is the counterweight. There are categories of use where these tools are the wrong tool, no matter how carefully you prompt them, and no amount of verification can rescue a fundamentally inappropriate use.

Knowing this list — being able to say “this is not what this tool is for” — is one of the more valuable things AI literacy can give you.

Use 1 — As the source of facts in high-stakes contexts

We discussed this in the hallucination section, but it deserves naming as a top-level rule.

Do not use generative AI as a primary source of fact when:

• The fact will be published or attributed to you.
• The fact will guide a decision with real consequences (medical, legal, financial).
• The fact concerns a specific named person, place, or event.

You can use generative AI to help you organise facts you have already verified. You should not rely on it to be the source of those facts.

The temptation is large because the output sounds authoritative. The cost of a wrong fact, attributed to you, is even larger.

Use 2 — For medical, legal, or financial advice that bears on a real decision

The general principle: generative AI is trained to produce text that sounds like advice, not to be qualified advice. For consequential decisions in regulated domains, this is exactly the wrong shape of tool.

• A model can explain general concepts about diabetes, depression, or cancer. It cannot replace a doctor evaluating you.
• A model can explain general concepts about contract law or property rights in Nepal. It cannot replace a lawyer evaluating your specific case.
• A model can explain general concepts about investing or budgeting. It cannot replace a financial advisor working with your specific situation.

For information-gathering and concept-learning, generative AI is fine. For the actual decision, see a qualified human.

Use 3 — For emotional support in distress

A grey area, with growing usage. Many people use ChatGPT or Claude to talk through emotional difficulties. This is not categorically bad — for some, it helps. But there are specific situations where it is the wrong tool:

• Suicidal thoughts or acute crisis. Reach for a real human — friend, family, crisis hotline (Nepal: TUTH Suicide Prevention Helpline 1166, or local equivalent). Generative AI is not trained as a crisis counsellor, even when it tries to play one.
• Symptoms of severe mental illness. A professional is essential. The model is not a therapist.
• Decisions about a relationship that the other person hasn’t consented to discussing with an AI. Practical and ethical.

For everyday venting, journaling, or thinking-out-loud, conversation with a model can be useful. For real distress, the model is an extra tool, not a substitute for human and professional care.

Use 4 — When the input is confidential and you don’t trust the provider

We treat this in detail in the next chapter, but the principle is worth flagging:

• Do not paste confidential business information (client data, financials, source code with trade secrets) into a consumer chatbot unless you’ve read its data-handling terms and they support your use.
• Do not paste personal data of others (medical records of patients, financial details of clients) into a tool with unclear data handling.
• Do not paste government-classified or contractually-confidential material into anything you don’t fully control.

Many organisations have specific policies. Find yours. If you don’t have one, default to caution — assume what you paste may be logged, stored, used for training, or seen by a human reviewer at the provider. Use a private deployment for confidential work, or don’t use generative AI at all.

Use 5 — When you don’t have the skill to verify the output

A subtle but important case. A model is most useful when you have just enough expertise to verify its output. A junior accountant can use a model to draft a balance sheet, because the junior accountant knows what a balance sheet should look like. A complete beginner cannot — they have no way to catch the model’s errors.

The same pattern applies across domains:

• A new programmer using a model to write code they don’t understand will introduce bugs they can’t catch.
• A history student using a model to explain a topic they have no background in will absorb confident wrong claims without realising.
• A new manager using a model to write a “diplomatic but firm” email may not catch when the model’s tone is not the tone they intended.

The fix is not to avoid generative AI when you’re a beginner. It is to use it as a teaching tool — to learn the domain, with a human teacher or text as the ground truth — until you have enough expertise to verify the output. Generative AI works best as an amplifier of skill you already have, not as a substitute for skill you haven’t acquired.

Use 6 — When the failure mode is asymmetric

In some tasks, the cost of being wrong is enormously higher than the cost of being right. A 95% accurate model is great for sorting customer complaints into departments — the 5% wrong are easily corrected. The same 95% accuracy is unacceptable for, say, evaluating which prisoners to release on parole, where the 5% wrong might be a person harmed.

For asymmetric tasks, the test is not “is the model usually right?” but “what happens when it is wrong, and who pays?”

Useful asymmetric-failure red flags:

• A wrong output can hurt a real, identifiable person.
• A wrong output is irreversible.
• A wrong output will be acted on without further human review.

When all three apply, generative AI is almost never the right tool unless deployed with extensive guardrails.

A test you can apply yourself

When deciding whether to reach for generative AI for a task, ask:

1. What does the model actually do here — produce text/image/audio, or judge truth/value/correctness? If the latter, be very careful.
2. What is the cost of being wrong? Map it explicitly.
3. Will I verify before acting? If not, why not?
4. Is there a qualified human whose role this would replace? If yes, is that an appropriate replacement?
5. If this output were posted publicly tomorrow, would I be comfortable that it was generated, not authored? If not, what’s the disclosure plan?

These five questions are not a perfect filter, but they are usually enough to tell you whether you are using the tool well or using it wrongly.

Check your understanding

Quick check

—

A friend asks ChatGPT whether their persistent chest pain could be heart disease. ChatGPT provides a thoughtful, detailed response covering common causes. The most accurate description is:

1. AChatGPT is now their doctor
2. BChatGPT can usefully explain general concepts about chest pain — but is not a substitute for a qualified medical professional evaluating *this specific person*. The friend should see a doctor.
3. CThey should believe whatever ChatGPT says without further consultation
4. DThey should ignore the pain entirely

Check answer→

What comes next

We’ve covered the limits. Chapter 6 turns to the operational side of using generative AI well over time — privacy, honesty about your use, and integrating these tools into your work without losing your skill. We close the course there.