Temperature, and controlling the output · Building with Language Models

Every LLM call has one small argument that most beginners never touch: temperature. It controls how deterministic or how creative the model’s output is. Setting it correctly for the task at hand is the difference between an extraction pipeline that reliably returns the same result every time and one that returns three different answers to three identical calls. This section is the careful version of what temperature does, why it matters, and the small set of habits that keep your outputs predictable.

What temperature actually is

Under the hood, an LLM generates one token at a time. At each step, the model computes a probability distribution over the next possible token — “given the words so far, what’s likely to come next?” — and then samples from that distribution to pick the actual token.

Temperature controls how that sampling happens:

Temperature 0.0 — always pick the most likely next token. The output is deterministic: same prompt, same result every time (or as close to it as makes no difference).
Temperature 1.0 — sample directly from the probability distribution. More varied, more creative outputs. Same prompt can produce different results.
Higher (1.5+) — flatten the distribution further, making unlikely tokens more likely. Rarely useful; produces weird or incoherent text.

The dial goes from 0 to 2 in most APIs, but you’ll basically only ever use values between 0 and 1.

When to use what

A short, honest guide:

Temperature 0.0 for:

Extracting structured data from text (names, amounts, dates).
Classifying text into categories.
Following instructions that have a single correct outcome.
Generating code where correctness matters.
Any task where you want to log a result and be able to reproduce it later.

Temperature 0.3-0.5 for:

Summarising text (a small amount of variation is fine).
Answering factual questions (mostly deterministic, tiny room to vary phrasing).
Rewriting text in a different style (variety helps).

Temperature 0.7-1.0 for:

Creative writing (poems, stories, marketing copy).
Brainstorming multiple options.
Chatbot personalities where varied responses feel more natural.
Any task where “the model surprising me” is a feature, not a bug.

The demonstration

To see the difference concretely:

prompt = "Give me a one-sentence description of a good NEPSE trader."

for t in [0.0, 0.5, 1.0]:
    print(f"\n=== temperature={t} ===")
    for i in range(3):
        response = client.messages.create(
            model="claude-haiku-4-5-20251001",
            max_tokens=100,
            temperature=t,
            messages=[{"role": "user", "content": prompt}],
        )
        print(f"  {i+1}: {response.content[0].text.strip()}")

You will see something like:

=== temperature=0.0 ===
  1: A good NEPSE trader is disciplined, patient, and understands
     Nepal's market fundamentals before acting on any tip.
  2: A good NEPSE trader is disciplined, patient, and understands
     Nepal's market fundamentals before acting on any tip.
  3: A good NEPSE trader is disciplined, patient, and understands
     Nepal's market fundamentals before acting on any tip.

=== temperature=0.5 ===
  1: A good NEPSE trader is patient, does their own research, and
     avoids acting on hype from social media.
  2: A good NEPSE trader is disciplined about position sizing and
     reads company filings, not just chart patterns.
  3: A good NEPSE trader balances short-term movements with a
     long-term view of Nepal's economy.

=== temperature=1.0 ===
  1: A good NEPSE trader combines local market knowledge with the
     humility to admit when a thesis has broken.
  2: A good NEPSE trader treats capital preservation as sacred and
     lets the compounding do its slow work.
  3: A good NEPSE trader reads the room — knowing when SEBON
     decisions matter more than technical charts.

Same prompt. Three temperatures. At 0.0 the answer is fixed. At 0.5 there’s meaningful variety in framing but the same overall character. At 1.0 the responses feel more distinct, more essayistic.

Neither is objectively better. The right choice depends on your application.

The subtle case: extraction

The most common mistake beginners make is leaving temperature at the API default (often 1.0) for extraction tasks. The result is a “structured output” that varies between calls — sometimes with different field values, sometimes with different JSON structure entirely.

Compare:

# Bad: extraction at temperature=1.0
extract_prompt = """
Extract the vendor, amount, and date from this receipt:

"KHALTI PAYMENT — Rs. 1,240 at Bhatbhateni on 2026-06-15"

Return JSON with fields "vendor", "amount", "date".
"""

# Call it three times at temperature=1.0
for _ in range(3):
    response = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=200,
        temperature=1.0,
        messages=[{"role": "user", "content": extract_prompt}],
    )
    print(response.content[0].text)
    print("---")

You might see three different JSON structures — one has "vendor": "Bhatbhateni", another has "vendor": "Khalti", a third wraps it all in a nested object. The task has one correct answer, but the model is sampling from possibilities and giving you inconsistent output.

The fix: temperature=0.0.

response = client.messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=200,
    temperature=0.0,   # deterministic
    messages=[{"role": "user", "content": extract_prompt}],
)

Now you get the same output every call. If it’s wrong, it’s consistently wrong — and you can debug and fix the prompt. If it varies between calls, you can’t tell what’s the prompt’s fault and what’s the sampling’s.

Other output-shaping knobs

Temperature is the biggest, but there are two more worth knowing:

top_p — nucleus sampling. Like temperature, but instead of controlling probability shape, it truncates the distribution to only the most likely tokens whose cumulative probability adds up to top_p. Sensible values are 0.9-1.0. Most of the time, tune temperature and leave top_p alone.

stop_sequences — a list of strings that, if generated, make the model stop immediately. Useful when you want the response to end at a specific delimiter (e.g. stopping at \n\n for a single-paragraph response).

response = client.messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=400,
    stop_sequences=["\n\nHuman:", "\n\nAssistant:"],
    messages=[...],
)

You’ll use stop_sequences mostly when generating structured content where you know exactly what the delimiter is. Temperature is the daily-driver knob.

A recommended default

For most of Course 04, unless we’re doing something specifically creative, we’ll use:

temperature=0.0
max_tokens=800

temperature=0.0 gives us reproducible outputs we can debug. max_tokens=800 is generous enough for most responses (up to ~600 words) without inviting unnecessary rambling. Set these once in your utility function and forget about them.

When we do want creativity (a chatbot, an idea generator), we’ll raise the temperature deliberately — and I’ll say so explicitly. Everything else defaults to determinism.

Check your understanding

Quick check

—

A teammate reports that their receipt-parsing pipeline sometimes returns `'vendor': 'Khalti'` and sometimes `'vendor': 'Bhatbhateni'` for identical input receipts. They're using temperature=1.0 with GPT-4. What's the first fix to try?

Switch to Claude.
Set temperature=0.0. At temperature=1.0 the model is *sampling* from possible outputs — including from ambiguity in the receipt string about which name is the vendor. At temperature=0.0 the output becomes deterministic: same receipt → same result, always. Now if the model still picks the wrong field, it's the prompt's fault (and you can fix the prompt), not sampling variance.
Send each receipt three times and take a majority vote.
The problem is the receipt; ask the user to reformat.

Quick check

—

You're building a Nepali chatbot. The lead engineer suggests running everything at temperature=0.0 for consistency. What's the honest response?

Yes — always use temperature=0.0.
For a chatbot, temperature=0.0 tends to make responses feel robotic and repetitive — literally the same greeting to the same 'hi' every time. Use 0.6-0.8 for chat: enough variation to feel natural, but not so much that responses drift into weirdness. Use 0.0 for the extraction, classification, and structured-output pieces of the chatbot's pipeline, but not for the conversational turns themselves.
No — always use temperature=1.5 for chatbots.
Temperature doesn't matter for chatbots.

What comes next

Chapter 1 is done. You know what an LLM API is, how to call it, what it costs, and how to control its output. Chapter 2 goes deep on prompting — writing prompts that survive contact with production, getting structured JSON output you can validate, and handling the very real errors and edge cases that show up in real systems. This is where LLM engineering starts to feel like software engineering, not just chat.