Why models forget, and context windows

The forgetful function

An LLM API is, technically, a stateless function. You give it a prompt, it gives you a response. Between calls, nothing is remembered. The model has no server-side database of previous conversations, no persistent identity of who “you” are, no idea that this call is the tenth in a conversation vs. the first.

This is the same shape as almost every web API. When you call requests.get("https://api.example.com/weather?city=kathmandu"), the weather API has no memory of your previous calls; it just answers each one on its own merits. LLMs are the same.

The illusion of memory in a chat interface like Claude.ai or ChatGPT is entirely maintained by the client, not the server. When you send message #10 in a chat, the client is actually sending the full history of messages 1-9 along with your new message. The model reads all ten, and its response takes the whole conversation into account.

The mechanic, plainly

In Anthropic’s API, memory is exposed through the messages list:

# Turn 1: user asks a question
response1 = client.messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=400,
    messages=[
        {"role": "user", "content": "My name is Nischal. What's a good name for a NEPSE-tracking bot?"},
    ],
)
# → "Nice to meet you, Nischal. How about 'NepseNepal' or ..."

# Turn 2: user asks a follow-up — you must send the history
response2 = client.messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=400,
    messages=[
        {"role": "user", "content": "My name is Nischal. What's a good name for a NEPSE-tracking bot?"},
        {"role": "assistant", "content": response1.content[0].text},
        {"role": "user", "content": "Which of those would you actually pick?"},
    ],
)
# → "For your bot idea, I'd pick 'NepseNepal' because ..."

If you had omitted the first two messages from the second call, the model would have no idea what “those” referred to, or that you’re Nischal. The whole state of the conversation lives in the client and gets re-sent every turn.

Context window: the upper bound

The reason you can’t just send infinite history is the context window — the maximum number of tokens the model can process in a single call. As of 2026:

• Claude Haiku 4.5: 200,000 tokens (~150,000 words)
• Claude Sonnet 4.6: 200,000 tokens
• Claude Opus 4.7: 200,000 tokens
• GPT-4.1: 200,000 tokens
• GPT-5: 400,000-1M tokens depending on variant
• Gemini Pro: 1,000,000+ tokens

These are large — a 200k-token window fits roughly a novel’s worth of text. But they are finite. Once your conversation history plus the response exceeds the window, the API returns an error.

More subtly, using more context costs more per call (you pay per input token) and models tend to get slightly worse as the context gets very long — they attend less carefully to details buried deep in the middle. This is called “lost in the middle” and it’s a real effect in production systems.

What this means for building

Three design implications flow from all of this:

1. Every meaningful conversation needs a memory strategy. Whether you keep the full history (simple), summarise old turns (cheaper), or fetch relevant past messages on demand (RAG-style, Chapter 5), you have to decide.

2. Context is a budget. In a chat that runs for hours, the conversation history grows. At some point you must trim, summarise, or scroll. Ignoring this leads to gradually rising costs and eventual API errors.

3. Long-term memory is a separate system. If you want a chatbot that remembers a user across sessions (weeks, months), you cannot rely on the LLM’s context window. You need a database of user facts that you inject into the system prompt each session. The LLM is not the memory; the LLM is the reasoner.

The three memory strategies, briefly

We cover each in more depth in the next section, but a preview:

Strategy A — full history. Keep every message and send it all. Simple, correct, expensive at scale. Works for short conversations.

Strategy B — sliding window. Keep only the last N turns. Simple, cheap, forgets old context. Works when only recent context matters (customer support, quick Q&A).

Strategy C — summarised history. As the conversation grows, periodically summarise old turns into a short block that gets prepended. Complex, tuning-heavy, but scales to arbitrarily long conversations. Works for long-lived assistants.

Almost every real chatbot uses some combination. A common recipe: sliding window of the last 10 turns, plus a running summary of everything before that.

Nepali chatbots specifically

Nepali text uses roughly 2x the tokens of English, so a context window that fits 200k tokens of English fits only ~100k tokens of Nepali. If your bot is bilingual (Nepali/English), budget accordingly.

Also: models occasionally lose track of language switches in long histories. A conversation that started in Nepali and drifts to English can end up with the model responding in a weird mix. Keeping the system prompt explicit about language (“respond in the same language as the most recent user message”) helps.

A tiny worked demonstration

To feel the statelessness concretely, run this:

# Two calls with the same client, but no shared history

response1 = client.messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=200,
    messages=[
        {"role": "user", "content": "Remember: my favourite Nepali dish is momo. Now respond with just 'ok'."},
    ],
)
print("Call 1:", response1.content[0].text.strip())

response2 = client.messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=200,
    messages=[
        {"role": "user", "content": "What's my favourite Nepali dish?"},
    ],
)
print("Call 2:", response2.content[0].text.strip())

Output:

Call 1: ok
Call 2: I don't have any information about your favourite Nepali dish. Could you tell me what it is?

The second call has no idea what happened in the first. Nothing was carried across. This is not a Claude limitation; it’s how the API works.

Now the version with memory:

history = [
    {"role": "user", "content": "Remember: my favourite Nepali dish is momo. Now respond with just 'ok'."},
]

response1 = client.messages.create(model="...", max_tokens=200, messages=history)
history.append({"role": "assistant", "content": response1.content[0].text})

history.append({"role": "user", "content": "What's my favourite Nepali dish?"})
response2 = client.messages.create(model="...", max_tokens=200, messages=history)

print(response2.content[0].text.strip())
# → "Your favourite Nepali dish is momo."

Same two questions. The only difference is that we manually maintained the history list. The next section is the full version of this pattern.

Check your understanding

Quick check

—

A teammate claims that Claude 'remembers' facts about users because it responded correctly to a follow-up question in their chatbot. What's the accurate description of what's happening?

1. AClaude has a per-user memory database.
2. BThe chatbot code is sending the full conversation history (or a curated subset) as the `messages` list on every call. The model reads that history and responds accordingly. There is no server-side memory — if the code stopped including history, the model would answer as if it had never talked to the user before. Memory in LLM chat is a client-side pattern maintained by the developer, not a server-side feature.
3. CThe model was fine-tuned on the specific user's data.
4. DThe API stores conversations for 24 hours automatically.

Check answer→

Quick check

—

A Nepali customer-support chatbot is going to run 100 conversations a day, each averaging 30 turns. What's the most immediate practical concern from this section?

1. AThe model will run out of memory.
2. BCost per conversation grows quadratically if we resend the full history every turn — each new turn re-pays for all previous input tokens. Over 30 turns of a Nepali conversation (Nepali uses ~2x tokens vs English), that adds up. A sliding window (last 10-15 turns) or summarised history strategy is needed before scaling. This is a design decision to make before shipping, not after.
3. CThe API will crash.
4. DYou need a bigger context window.

Check answer→

What comes next

You understand why models forget. The next section is the practical craft of keeping conversation state across turns — the sliding-window pattern, the summarised-history pattern, and the code that makes multi-turn chat work reliably. By the end of Section 2 you’ll have a reusable Conversation class you can drop into any chatbot you build.