5.5 When a prompt is not enough: retrieval, tools, workflows

The four signals you have outgrown prompting

1. The model needs facts it doesn’t have. If the answer requires the current price of rice, the latest company policy, or a document written last week, no prompt tweak fixes that. The information genuinely isn’t in the model.

2. The model needs to act, not just answer. Sending an email, updating a database, calling an API. Prompting produces text; text alone can’t change the world.

3. The task is too big for one call. A 200-page contract that needs three different kinds of analysis, or a batch of 10,000 tickets that need triage. One prompt-per-call won’t cut it.

4. The failure rate is unacceptable and eval-driven iteration has plateaued. Chapter 5.2 says stop iterating after 3–4 consecutive changes that move the eval by less than a percent. If you’re plateaued at 78% and you need 99%, prompting alone won’t get you there.

What each signal points to

Signal 1 → Retrieval

For “the model needs facts it doesn’t have,” the answer is retrieval-augmented generation (RAG):

1. Store your source-of-truth documents (policy PDFs, product docs, past tickets, an internal wiki) in a searchable index — usually a vector database, sometimes just a good full-text search.
2. At query time, retrieve the top few relevant passages.
3. Feed those passages into a grounded prompt (Chapter 3.3) and let the model answer.

The model does what models are good at (reading, summarising, quoting). The retrieval layer does what models are bad at (recalling specific facts). Combined, they answer questions about your specific documents.

You don’t need to build a RAG system to prototype the idea — for small corpora, just paste the relevant document into a grounded prompt manually. If you’re pasting the same handful of documents into every call, you probably want retrieval.

Signal 2 → Tools

For “the model needs to act,” the answer is tool use (also called function calling):

1. Define a set of functions the model can call — send_email(to, subject, body), create_ticket(title, priority), look_up_customer(customer_id).
2. The model outputs a structured request to call a function; your code actually executes it.
3. The result gets fed back into the model for the next step.

Tool use is how a chatbot becomes an assistant. It’s also where the security posture from Chapter 5.3 becomes non-negotiable — a model that can call tools is a model that can cause real damage if injected.

Signal 3 → Workflows

For “the task is too big for one call,” the answer is a workflow:

• Split the task into stages, one prompt per stage. First prompt extracts. Second prompt classifies. Third prompt drafts a response. Each stage does one job.
• Loop over batches. Ten thousand tickets is a for-loop, not a single prompt.
• Use conditional routing. High-confidence outputs go to auto-resolve; low-confidence outputs go to a human.
• Save intermediate state so you can retry failed stages without re-running the whole pipeline.

Workflows turn a fragile 20-step monster-prompt into a chain of five reliable single-purpose prompts. Cheaper to debug, easier to test, and each stage can be improved independently.

Signal 4 → Fine-tuning, or accept the ceiling

If the eval has plateaued and prompting won’t move it further, options are:

• Fine-tune a model on your task. Expensive, slow to iterate, but sometimes the only way past a stubborn ceiling — especially for classification into project-specific labels.
• Use a stronger model. A frontier model will often outperform a smaller one on hard tasks even with the same prompt. Cost vs quality trade-off.
• Accept the ceiling and design around it. Route ambiguous cases to a human, add a confidence gate, or narrow the scope of what the model handles.

Not every task should be automated to 100%. A well-designed 85%-model-plus-15%-human workflow can beat a 95%-model-only workflow on both accuracy and user trust.

The end of this course, and what’s next

You’ve now seen the whole surface of the craft — the mental model, structure, reliability, applied patterns, and the boundaries of what a prompt alone can do. Prompt engineering is not a static discipline. Models change, defaults shift, new failure modes appear. What stays constant is the habit: write down what you meant, close off the wrong interpretations, test it against real inputs, and iterate one change at a time.

If you want to go further:

• The Generative AI track covers what’s actually happening inside the model.
• The Building with Language Models track covers the API, SDK, and system-level side of shipping LLM features.
• Claude Code for Developers covers agentic coding — a specialised application of everything in this course.

The best next step, though, is to take a prompt you’ve been meaning to improve and run it through Chapter 5.1’s eval habit. Fifteen minutes, fifteen examples, one number. That’s where craft becomes discipline.

Check your understanding

Quick check

—

Your prompt needs to answer employee questions about your company's HR policy. The prompt gets facts wrong and cites made-up policy sections. You have rewritten it seven times. What is the correct next move?

1. ARewrite the prompt an eighth time with stronger warnings against hallucination.
2. BBuild a retrieval layer: index the actual HR policy documents, retrieve the relevant passages, and feed them into a grounded prompt that quotes before it answers.
3. CAsk the model to try harder.
4. DAccept that this task cannot be automated and delete the prompt.

Check answer→