Chapter 01 · Section II · 10 min read
1.2 How the model reads you: tokens, context, memory
Just enough about tokens, context windows, and cross-call memory to make sense of why prompts work the way they do. Model-agnostic; no maths.
Everything you can do with a prompt is shaped by three mechanical facts about how a language model reads text. You don’t need to know how transformers work — but you do need to know these three, or you’ll write prompts that fight the machine.
1. The model reads in tokens, not words
Text is chopped into tokens before the model sees it. A token is usually a common word, a piece of a word, or a punctuation mark. Rough rule of thumb for English: 1 token ≈ 4 characters ≈ 3/4 of a word. For Nepali and other Devanagari scripts, tokens are often shorter — a single Nepali word can spend several tokens because the tokenizer was trained mostly on English.
What this means for you:
- Word count is a bad proxy for prompt size. A JSON blob with lots of quotes and braces spends more tokens than the same content in prose.
- Non-English text costs more. A prompt in Nepali will use noticeably more tokens than the same prompt in English. This affects both cost (if you’re paying per token) and how much fits in the context window.
- Whitespace and punctuation count. They are their own tokens.
You don’t need to count tokens by hand. You just need to know they exist so you understand why very long prompts get expensive and why asking for output in Nepali costs more than the same request in English.
2. The context window is finite
Everything the model can see in one call — the system message, the user message, previously-attached files, and its own draft response — has to fit in the context window. Different models have different windows (thousands of tokens on the low end, millions on the high end).
When the window fills up:
- Older content gets truncated or summarised, depending on the tool.
- Instructions at the very start of a long prompt can get “forgotten” — not literally, but the model weights recent content more heavily.
- The response you’re generating also eats into the window.
3. There is no memory between calls (unless someone built one)
Each API call to a raw language model is independent. Nothing about your previous call is available in the next one — no chat history, no preferences, no files you shared before.
Chat apps like ChatGPT, Claude, and Gemini fake memory by re-sending the previous turns as part of each new prompt. That’s why long chat conversations get slower and eventually hit context limits. Some products now add a separate “memory” feature that summarises facts about you and injects them back into future prompts — but this is a product feature layered on top, not a property of the model.
For you, the practical rule: if you want the model to know something, it has to be in the current prompt. Do not assume it “remembers” from yesterday.
Why this matters for prompting
Take a moment with these three facts and a lot of prompt-engineering advice stops being magic:
- “Put important instructions at the top and the bottom” — because the middle of a long prompt gets less attention.
- “Keep prompts short” — because tokens cost money and time, and shorter prompts leave less room for noise.
- “Give it examples in the prompt” — because you can’t rely on it having seen your specific style before.
- “Be explicit about who the audience is” — because the model has no idea otherwise.
Check your understanding
Quick check
—