Grounding a model in your own documents · Building with Language Models

An LLM knows a lot about the world in general. It knows almost nothing about the specific documents you care about — your company’s policies, your ministry’s circulars, your team’s Notion, your bank’s product sheets. Retrieval-Augmented Generation — RAG — is the pattern that closes this gap. It combines the semantic search from Chapter 4 with an LLM reader from Chapters 1-3. This section is the mental model: why RAG exists, what it solves, and what its limits are. The next two sections build one.

The problem RAG solves

Recall the “hallucination” limit from Chapter 1: LLMs sometimes confidently produce text that is factually wrong. This gets much worse when you ask about specifics:

“What does the NEPSE listing bylaw say about disclosure timelines?” — the model has vague training-era knowledge, guesses.
“What’s our company’s parental leave policy?” — the model has no idea, invents plausible-sounding text.
“Which of the Ministry of Education’s 2025 circulars covers scholarship eligibility?” — again, guessing.

None of this is malice. The model is doing what it was trained to do: produce plausible text given the prompt. It has no built-in signal for “I don’t actually know this.” The result: confident-sounding fabrications, which are terrible for anything a real user will act on.

RAG solves this by providing the answer in the prompt. Instead of asking the model to answer from memory, you first retrieve the relevant documents (using semantic search) and include them in the prompt. The model reads the retrieved text and answers from it.

The picture

     User: "What does the NEPSE bylaw say about disclosure timelines?"

           │
           ▼
     ┌─────────────────────────┐
     │  Semantic Search        │  → returns top 5 passages from
     │  (embeddings, Chapter 4)│    the NEPSE bylaw corpus
     └─────────────────────────┘
           │
           ▼
     ┌────────────────────────────────────────────────┐
     │  LLM Prompt:                                   │
     │  "Answer using these passages ONLY.            │
     │   If the passages don't contain the answer,    │
     │   say so.                                      │
     │                                                │
     │   Passages:                                    │
     │   [1] ...retrieved passage 1...                │
     │   [2] ...retrieved passage 2...                │
     │   ...                                          │
     │                                                │
     │   Question: What does the NEPSE bylaw say      │
     │   about disclosure timelines?"                 │
     └────────────────────────────────────────────────┘
           │
           ▼
     LLM reads the passages, produces an answer with citations.

Two components, one flow:

Retrieval: find documents that might contain the answer (semantic search).
Generation: hand the LLM the found documents plus the question; let it write the answer.

Why this works so much better

Three things happen when you switch from “ask the LLM from memory” to RAG:

1. The answer is grounded. The model answers from text you gave it, not from vague memory. Hallucination rates drop dramatically.

2. You can cite sources. Because you know which passages the model was reading, you can show the user where the answer came from. This is huge for trust — the difference between “the AI says…” and “here’s the exact bylaw section that says…”.

3. The model can say “I don’t know.” When the retrieved passages don’t contain the answer, the LLM can (if prompted to) say so honestly. Compare with plain-LLM Q&A, where the model has no principled way to detect it doesn’t know.

When to use RAG (and when not)

RAG is right when:

You have a specific corpus the model needs to answer from (policies, docs, articles, your own writing).
You want citations and traceability.
The corpus is too large or too current to fit in the model’s training data.
The corpus updates over time, and you want the LLM to always see the latest.
The user’s questions are answerable if you can find the right passage.

RAG is wrong when:

The user’s question is genuinely general (creative writing, brainstorming, general knowledge already in the model).
You need real-time reasoning over live data (databases, APIs, calculations) — that’s tool use, Chapter 6.
The corpus is small enough to fit entirely in the context window — just paste it in the system prompt directly and skip the retrieval overhead.
The user’s question requires multi-hop reasoning across many documents at once — RAG helps but isn’t the whole answer.

The most common misuse is applying RAG when the user’s actual need is different. A “chat with our website” bot that returns paraphrased marketing copy is often solving the wrong problem — the user wanted a live product query, which needs tool use, not RAG.

The three real limitations

RAG works well, but is not magic. Three honest limits:

1. Retrieval quality caps quality. If the semantic search misses the relevant passage, the LLM has nothing to work with and either says “I don’t know” or hallucinates from general knowledge. The upstream of RAG is the retrieval; investing in retrieval quality matters more than swapping to a bigger LLM.

2. Chunking affects everything. Chunks that split a key passage awkwardly across two chunks will each lose half the meaning; retrieval degrades. Chunking discipline (Chapter 4) is real engineering.

3. Multi-hop questions are hard. “What percentage of MoU-related circulars from 2025 also mention scholarship eligibility?” requires reading many documents and reasoning across them. Basic RAG (retrieve top-k, feed to LLM) does this poorly. Advanced RAG techniques (query decomposition, iterative retrieval, agentic loops) exist but are Course-05 territory.

The template

Every RAG system, at its core, is a prompt template:

"You are an assistant answering questions about {domain}.

Use ONLY the following retrieved passages to answer the question.
If the passages do not contain the answer, respond with 'I don't have
information about that in my knowledge base' — do not guess.

When you give an answer, cite the source(s) you used, using the
[Source] tags from the passages.

Passages:
[Source 1: {source_name_1}]
{passage_1}

[Source 2: {source_name_2}]
{passage_2}

...

Question: {user_question}"

Two rules for a good RAG prompt:

Explicitly forbid hallucination. “Answer ONLY from these passages” and “say I don’t know” are non-negotiable. Without them, models will confidently fill in gaps.
Explicitly request citations. Otherwise the model produces a coherent-sounding answer with no traceability, which defeats one of RAG’s main benefits.

The next section builds this template into a working pipeline.

A little honesty about scope

Some things RAG cannot do, that people wish it could:

Automatically stay current. RAG uses whatever’s in your corpus. If you don’t ingest new documents, the answers stay stale.
Answer questions the corpus doesn’t cover. If the bylaws don’t mention disclosure timelines, RAG cannot invent them. This is a feature (no hallucination), not a bug.
Reason across documents in one pass. Basic RAG stitches together what came back from retrieval; it doesn’t proactively look for supporting evidence across the whole corpus.
Replace domain expertise. RAG can surface the right passage; understanding what to do with it still often needs a human.

Building good RAG systems is 80% good corpus curation and retrieval, 20% good prompt engineering. The LLM is the least important part.

Check your understanding

Quick check

—

A developer builds a 'company knowledge base assistant' by feeding user questions directly to Claude with no retrieval, no documents. Users report it makes up policies that don't exist. What's the specific fix, in the language of this section?

Switch to a bigger model.
Adopt a RAG architecture: index the company's actual policy documents into a vector store, retrieve the passages relevant to each user query, and put those passages in the LLM prompt with an instruction to answer *only* from them. The current design is 'ask the LLM to remember policies it never saw' — a classic hallucination setup. RAG grounds the model in real content and lets it truthfully say 'I don't know' when the docs don't cover the question.
Add a warning to responses.
Reduce the temperature to 0.

Quick check

—

A team's RAG system returns confident but wrong answers. Investigation shows the LLM does read the retrieved passages, but the passages are usually not the right ones — the semantic search is missing what the user actually needs. What's the right investment?

Switch to Claude Opus.
Invest in the retrieval side: better chunking, better embedding model, better handling of Nepali/English mixed queries, maybe hybrid search (semantic + keyword), maybe reranking. Retrieval quality caps end-to-end quality — the LLM is only as good as the passages it receives. Upgrading the LLM without fixing retrieval is like buying a better cook and still handing them the wrong ingredients.
Use bigger prompts.
Reduce the number of passages retrieved.

What comes next

You have the mental model. The next section is the code: a complete RAG pipeline, step by step. Load documents, chunk, embed, retrieve, prompt, cite. About 200 lines of Python, and by the end of Section 2 you’ll have a working Q&A system you can point at any folder of documents.