Conversations: keeping state across turns

Strategy A — full history

The simplest correct implementation: keep every message, send it all every call.

import anthropic

client = anthropic.Anthropic()


class Conversation:
    def __init__(self, system_prompt: str, model: str = "claude-haiku-4-5-20251001"):
        self.system = system_prompt
        self.model = model
        self.messages: list[dict] = []

    def ask(self, user_message: str, max_tokens: int = 800) -> str:
        self.messages.append({"role": "user", "content": user_message})

        response = client.messages.create(
            model=self.model,
            max_tokens=max_tokens,
            temperature=0.7,
            system=self.system,
            messages=self.messages,
        )

        assistant_text = response.content[0].text
        self.messages.append({"role": "assistant", "content": assistant_text})
        return assistant_text

Usage:

conv = Conversation(
    system_prompt=(
        "You are a friendly Nepali-speaking assistant. Respond in the "
        "same language the user writes in. Keep responses under three "
        "sentences unless asked otherwise."
    )
)

print(conv.ask("नमस्ते! मेरो नाम निश्चल हो।"))
print(conv.ask("मेरो नाम याद छ?"))
print(conv.ask("के मैले खल्ती अलर्ट स्क्रिप्ट बनाउन सक्छु?"))

Twenty-five lines. Works. For short conversations (under 20 turns), this is all you need.

When full history breaks down

Two problems appear at scale:

Cost. Every turn re-sends all previous input tokens. A 30-turn conversation where each turn is 200 tokens ends up sending 90,000 input tokens over the course of the conversation (200 + 400 + 600 + … + 6000 = 90k). At Haiku prices ($0.25/M) that’s about $0.023 for the conversation — not catastrophic, but multiplied across many users and many long sessions it adds up.

Context window. For extremely long conversations (300+ turns, or ones with large uploaded documents), you eventually hit the 200k token ceiling and the API returns an error.

Both problems have the same solution: don’t send the full history. Trim it.

Strategy B — sliding window

Keep only the last N turns:

class Conversation:
    def __init__(self, system_prompt: str, window: int = 10, model: str = "claude-haiku-4-5-20251001"):
        self.system = system_prompt
        self.window = window
        self.model = model
        self.messages: list[dict] = []

    def ask(self, user_message: str, max_tokens: int = 800) -> str:
        self.messages.append({"role": "user", "content": user_message})

        # Only send the most recent `window` messages (each user/assistant pair is 2)
        recent = self.messages[-(self.window * 2):]

        response = client.messages.create(
            model=self.model,
            max_tokens=max_tokens,
            temperature=0.7,
            system=self.system,
            messages=recent,
        )

        text = response.content[0].text
        self.messages.append({"role": "assistant", "content": text})
        return text

With window=10, the model always sees the last 10 pairs of user+assistant messages, regardless of how long the conversation is. Old messages are silently forgotten.

This works beautifully for:

• Customer support (recent context matters, ancient context doesn’t).
• Q&A bots (each question mostly stands alone).
• Task-focused chatbots where the current task is what matters.

It fails for:

• Anything where the user might reference something from 20 turns ago (“as I said earlier…”).
• Long chats where the personality/preferences you set up early keep mattering.

For those, you need summarisation.

Strategy C — summarised history

The pattern: as the conversation grows, periodically summarise older turns into a compact block, and prepend that summary to the recent window.

class SummarisingConversation:
    def __init__(
        self,
        system_prompt: str,
        keep_recent: int = 6,
        summarise_after: int = 12,
        model: str = "claude-haiku-4-5-20251001",
    ):
        self.system = system_prompt
        self.keep_recent = keep_recent          # last N turns to keep verbatim
        self.summarise_after = summarise_after  # when to trigger summarisation
        self.model = model
        self.messages: list[dict] = []
        self.summary: str = ""

    def _summarise_old_messages(self):
        """Compress everything older than the recent window into a summary."""
        to_summarise = self.messages[: -(self.keep_recent * 2)]
        if not to_summarise:
            return

        summary_prompt = (
            "Summarise the following conversation history in 3-5 sentences. "
            "Focus on facts about the user, their goals, and any decisions made. "
            "Preserve names, preferences, and constraints.\n\n"
            + "\n".join(f"{m['role'].upper()}: {m['content']}" for m in to_summarise)
        )

        response = client.messages.create(
            model=self.model,
            max_tokens=400,
            temperature=0.0,
            messages=[{"role": "user", "content": summary_prompt}],
        )
        new_summary = response.content[0].text.strip()

        # Merge with previous summary if present
        if self.summary:
            self.summary = f"{self.summary}\n\nMore recent: {new_summary}"
        else:
            self.summary = new_summary

        # Drop the summarised messages from the live history
        self.messages = self.messages[-(self.keep_recent * 2):]

    def ask(self, user_message: str, max_tokens: int = 800) -> str:
        self.messages.append({"role": "user", "content": user_message})

        # Summarise if we've grown too long
        if len(self.messages) > self.summarise_after * 2:
            self._summarise_old_messages()

        # Build the system prompt with any summary prepended
        system = self.system
        if self.summary:
            system = f"{system}\n\nCONVERSATION SO FAR (summary):\n{self.summary}"

        response = client.messages.create(
            model=self.model,
            max_tokens=max_tokens,
            temperature=0.7,
            system=system,
            messages=self.messages,
        )

        text = response.content[0].text
        self.messages.append({"role": "assistant", "content": text})
        return text

The pattern:

• Below summarise_after turns: send everything.
• Above it: summarise older turns into a compact block, keep the last keep_recent verbatim, prepend the summary to the system prompt.
• The summary grows over time as new segments get summarised.

For chats that run for hundreds of turns, this pattern keeps costs bounded and context relevant. It’s the recipe most serious production chatbots use.

Token accounting

For any real chat application, track tokens and cost per conversation:

class Conversation:
    # ... previous code ...
    def __init__(self, ...):
        # ...
        self.total_input_tokens = 0
        self.total_output_tokens = 0

    def ask(self, user_message: str, max_tokens: int = 800) -> str:
        # ... build messages, call API ...
        self.total_input_tokens += response.usage.input_tokens
        self.total_output_tokens += response.usage.output_tokens
        return text

    def cost_so_far_haiku(self) -> float:
        return (
            self.total_input_tokens  / 1_000_000 * 0.25 +
            self.total_output_tokens / 1_000_000 * 1.25
        )

A one-line conv.cost_so_far_haiku() tells you exactly what a conversation has cost. In production, log this per session. Look at the distribution weekly — if the 95th-percentile conversation costs 20x the median, you have a runaway-user problem worth fixing.

The system prompt is memory too

An underused technique: put stable facts in the system prompt, not the message history.

system = f"""You are a Nepali-speaking assistant.

Known facts about this user:
- Name: {user.name}
- Location: {user.city}
- Preferred language: {user.language}
- Signed up: {user.created_at}
- Interests: {', '.join(user.interests)}

Guidelines: [...]
"""

Facts injected here are always visible to the model, don’t get trimmed by sliding windows, and don’t need to be repeated. They’re the model’s “long-term memory” of who this user is.

This is how real chatbots remember users across sessions: the app stores facts in a database keyed by user, injects them into the system prompt at the start of each session, and the LLM treats them as part of its identity of the user.

A short honest note

None of these strategies are perfect. Sliding windows genuinely forget old context. Summaries lose detail. Even 200k-token full-history chats occasionally lose track of things mentioned in the middle. Building good chatbot memory is an engineering discipline of choosing what to lose — because you always lose something. The strategies in this section are the honest, well-tested defaults; production applications usually blend them.

Check your understanding

Quick check

—

A teammate's customer-support chatbot works well for the first 10 turns, then starts hallucinating names and confusing users. Investigation shows they're using a full-history strategy and conversations often run 30+ turns. What's the most likely cause?

1. AThe model is running out of parameters.
2. BAt 30+ turns, the conversation history is large — likely 8,000-15,000 tokens. Even though this is well below the context window, LLMs get slightly worse at attending to details buried in the middle of long inputs ('lost in the middle' effect). A sliding-window strategy (last 10-15 turns) or a summarised-history strategy would keep the relevant context tight and improve accuracy. Full history isn't always better.
3. CAdd more retries to the API calls.
4. DSwitch to a smaller model.

Check answer→

Quick check

—

A production Nepali chatbot needs to remember user preferences across sessions (weeks or months apart). Which architectural component solves this, according to this section?

1. AA larger context window.
2. BA user-facts database, keyed by user, injected into the system prompt at the start of each session. The LLM's context window is short-term memory; long-term memory requires a separate storage layer (a database of user preferences, past decisions, personal facts). At the start of each session, load the relevant user facts, format them into the system prompt, and the model treats them as its identity of the user. This is the standard architecture.
3. CSliding-window chat history.
4. DThe LLM's own memory.

Check answer→

What comes next

You have the tools to build stateful chat. The last section of Chapter 3 puts them together: a complete, honest Nepali chatbot with system prompt, sliding-window memory, cost tracking, and a working command-line UI. By the end of Section 3 you’ll have something you could actually run in your terminal and chat with in Nepali.