3.1 Chain-of-thought and "think step by step"

The idea in one paragraph

Language models are next-token predictors. When you ask for an answer directly, they produce the answer directly — with no intermediate scratchpad to check their work. Prompting for a reasoning trace (“let’s think step by step”) gives the model tokens to plan on. For any task that involves multi-step arithmetic, logic, or comparison, this reasoning trace routinely improves accuracy.

When it actually helps

Chain-of-thought pays off for:

• Arithmetic and unit conversions. “If a bus fare is NPR 25 and I pay with NPR 500, how many NPR 20 notes and NPR 5 coins would make correct change?” — the model that thinks aloud gets it right much more often.
• Multi-step logic puzzles. “If A is taller than B, and B is taller than C, but D is shorter than C, order them by height.”
• Comparisons and rankings. “Which of these four job offers is best if I optimise for take-home pay after tax?”
• Anything that requires holding two or three constraints in mind at once.

When it doesn’t (or costs more than it earns)

• Pure lookup / recall. “What’s the capital of Bhutan?” doesn’t need thinking. Chain-of-thought here just burns tokens.
• Style tasks. “Rewrite this in a friendlier tone.” Reasoning traces don’t make prose warmer.
• Anything where the model is already correct. Extra reasoning on top of a working prompt often introduces new errors.

The two ways to do it

1. The bare “think step by step” nudge.

A shop sells cooking oil at NPR 340 per litre and rice at NPR 130 per kg.
If I have NPR 2000 and buy 3 litres of oil, how many kg of rice can I buy?

Think step by step, then give the final answer on a new line prefixed "ANSWER:".

You’ve done nothing but add one sentence, and accuracy on math-word problems typically jumps.

2. Structured scratchpad.

Solve the problem below. Show your work inside <scratchpad> tags,
then give the final answer inside <answer> tags.

<problem>
A shop sells cooking oil at NPR 340 per litre and rice at NPR 130 per kg.
If I have NPR 2000 and buy 3 litres of oil, how many kg of rice can I buy?
</problem>

The structured version is easier to parse — you extract the <answer> block for the user and keep the <scratchpad> for debugging or logging.

Hiding the reasoning from users

Chain-of-thought is useful for the model; it’s usually not useful for the user. Two ways to keep it out of the final output:

• Ask for both. “Show reasoning inside <scratchpad> tags, then the final answer inside <answer> tags.” Strip the scratchpad in code before showing anything to the user.
• Use a native “thinking” feature. Modern models (Claude, GPT-5, Gemini) offer a built-in extended-thinking mode where the reasoning stays server-side and only the final answer reaches you. Prefer this when available — it’s faster and cleaner than prompting.

The failure mode: reasoning theatre

A prompt that says “think step by step” doesn’t guarantee real thinking — the model can produce reasoning that looks deliberate but is post-hoc rationalisation of an already-wrong guess. Symptoms:

• Steps that don’t logically build on each other.
• Arithmetic that skips digits.
• A “conclusion” that ignores the reasoning above it.

The best safeguard is a small eval set (see Chapter 5.1) where you have known-correct answers and can measure whether chain-of-thought actually raises accuracy on your specific task.

Check your understanding

Quick check

—

For which of the following tasks is adding "think step by step" LEAST likely to improve results?

1. ACalculating monthly savings given income, rent, and grocery estimates.
2. BRewriting a paragraph in a warmer, more casual tone.
3. COrdering four candidates by score across three criteria.
4. DSolving a logic puzzle with three constraints.

Check answer→