Chapter 03 · Section III · 16 min read
Naive Bayes classification
A genuinely useful AI technique that fits on one whiteboard — and powered spam filters for two decades.
Naive Bayes is the simplest practical AI classifier. It applies Bayes’ rule under an assumption that is obviously wrong: that each piece of evidence is independent of the others. Surprisingly, this naive assumption works well enough to filter spam, classify documents, triage support tickets, and tag the language of a tweet.
It is also a perfect first model to learn. You can implement it in twenty lines of Python. You can explain it to a regulator. You can audit it on a notebook. Most “AI” deployments in 2026 would be better served by Naive Bayes than by a large language model — they just wouldn’t get the press release.
The setup: classifying SMS spam
A small Nepali telecom carrier wants to flag spam SMS messages. They have a dataset of historical messages, each labelled “spam” or “ham” (not spam). When a new message arrives, the system should output the probability it is spam.
The Bayes rule statement is:
P(spam | message) is proportional to P(message | spam) × P(spam)
The prior P(spam) is easy: count the fraction of historical messages that are spam. Suppose 20% of all messages are spam, so P(spam) = 0.2 and P(ham) = 0.8.
The likelihood P(message | spam) is the hard part. A message is a sequence of words. There are an astronomical number of possible messages. We cannot have seen each one often enough to estimate its probability.
This is where “naive” enters.
The naive assumption
Naive Bayes assumes that, given the class, every word in the message is independent of every other word. The word “winner” appearing in a spam message tells you nothing about whether “lottery” also appears. They are independent.
This is obviously, even insultingly, wrong. Real messages have grammar; words depend on each other. “Winner” and “lottery” co-occur in spam far more than chance would predict.
But the assumption is so convenient that we use it anyway. With independence, the likelihood factors into a product of single-word probabilities:
P(message | spam) ≈ P(word₁ | spam) × P(word₂ | spam) × … × P(wordₙ | spam)
And those single-word probabilities are easy to estimate by counting words in spam messages from the training set.
A worked example
Suppose our training set tells us:
- 25% of words in spam messages are “winner”. (P(winner | spam) = 0.25.)
- 1% of words in ham messages are “winner”. (P(winner | ham) = 0.01.)
- 30% of spam messages contain “lottery”. (P(lottery | spam) = 0.30.)
- 0.5% of ham messages contain “lottery”. (P(lottery | ham) = 0.005.)
A new message arrives: “You are the winner. Claim your lottery prize.”
We want P(spam | message) vs. P(ham | message). Under naive independence:
- P(message | spam) ∝ 0.25 × 0.30 × (other words) = 0.075 × (other words).
- P(message | ham) ∝ 0.01 × 0.005 × (other words) = 0.00005 × (other words).
Multiplying by the priors (P(spam) = 0.2, P(ham) = 0.8):
- Score for spam: 0.075 × 0.2 = 0.015.
- Score for ham: 0.00005 × 0.8 = 0.00004.
Spam wins by a factor of ~375. The classifier confidently calls this spam, and is right. The other words in the message — “you”, “are”, “the”, “claim”, “your”, “prize” — adjust the numbers, but the words winner and lottery already settled the verdict.
That is the whole algorithm. Count word frequencies in each class. Apply Bayes’ rule with naive independence. Pick the class with the highest score.
Why it works in practice
Naive Bayes has three properties that make it shockingly useful.
Fast. Training is counting. Inference is multiplying. Both are linear in the size of the data. A Naive Bayes filter on a million SMS messages trains in seconds on a laptop.
Transparent. You can inspect every word’s contribution. If the classifier flags a message, you can ask which words pushed it toward spam, and how much. This is impossible with a deep neural network.
Robust. Naive Bayes performs reasonably even when the training set is small or noisy. It does not need millions of examples to start being useful.
For a Nepali telecom rolling out spam filtering on Devanagari SMS, this matters enormously. They have a few hundred thousand labelled messages, not a few billion. They need to audit the system for a regulator. They need to retrain it when spammers change tactics, in minutes, not weeks. Naive Bayes is the right tool, and a transformer is not.
Where Naive Bayes breaks
Three failure modes are worth knowing.
-
Strongly dependent features. If the model has to classify medical records, where “fever” and “high temperature” are essentially the same feature, naive independence double-counts. Naive Bayes will over-weight the joint presence.
-
Unseen words. A new word in the test set that never appeared in training has zero probability under the naive estimate. Zero kills the entire product. The fix is smoothing: add a tiny pseudo-count to every word, so nothing has probability exactly zero.
-
Subtle classes. Naive Bayes will not learn sarcasm, sentiment, or anything that needs understanding context across many words. For those, you need richer models — but you should still benchmark them against Naive Bayes, because it is so easy to deploy that the bar to beat is high.
In the larger picture
The reason we spent a whole section on Naive Bayes is not nostalgia. It is that Naive Bayes is a complete AI system you can hold in your head — every step from data to deployment fits on a single page. Once you understand it, you understand the shape of every classifier we will meet, and you have a baseline against which to judge the more impressive ones.
A common mistake in AI procurement is to assume the latest, biggest model is the right one. The right model is the smallest one that does the job. For a great many real Nepali use-cases — spam filtering, language identification, helpdesk routing, sentiment on customer feedback — Naive Bayes is the smallest one.
Check your understanding
Quick check
—Naive Bayes is called 'naive' because it assumes that:
Quick check
—A Nepali telecom team has 200,000 labelled SMS messages and needs a spam filter they can audit, retrain quickly, and explain to a regulator. Which is the most defensible first choice?
What comes next
We close out the probability chapter here. Chapter 4 reframes everything we have learned under a more general name: machine learning. You will see that Naive Bayes is one of many ways to learn from data, and that the choice of method depends on what kind of data you have and what kind of question you are asking.