Tokens and context windows

By the end, you can

• Explain what a token is and estimate roughly how many tokens a piece of English text will use (LA-6).
• Explain why a context window is a single shared token budget that every part of a conversation draws from, not a limit on how many messages fit (LA-6).
• Look up a real local model's context window and explain why the number a runtime actually gives you by default can be smaller than the model's own maximum (LA-6).

Before you start

This is Lesson 1 of 3 in The model underneath, the module that comes right after Module 1's mental model and before the course moves on to bounded task design. It assumes Module 1's vocabulary — agent, boundary, tool — but nothing about local setup yet; that is the Running models locally module's job, once this module has given you the vocabulary for what you are actually setting up. This lesson does not require a running model. It does ask you to look at real numbers from real model documentation, so a browser is useful but no installation is.

A token is not a word

Every model in this course reads and writes in tokens, not words or characters. OpenAI's own developer documentation defines the unit plainly: "Text generation and embeddings models process text in chunks called tokens. Tokens represent commonly occurring sequences of characters." A common word like "the" is usually one token; a less common or compound word can split into several — "tokenization" itself commonly splits into pieces like "token" and "ization." Punctuation, spaces and partial words all consume tokens too.

For rough estimation in English, that same documentation gives a working rule of thumb: "1 token is approximately 4 characters or 0.75 words for English text." That ratio is not universal — every model family ships its own tokenizer, trained on its own data, so the same sentence can tokenize to a different count on a different model. Treat the 4-characters/0.75-words figure as a planning estimate for English prose, not an exact conversion, and re-check it against a specific model's own tokenizer when a precise count matters.

The context window is one shared budget, not a message limit

A model's context window is the maximum number of tokens it can hold in a single request — every system instruction, every prior turn, every tool result and every retrieved document, all counting against the same pool, together with whatever the model is about to generate in reply. It is easy to picture a "window" as something that scrolls to show only the newest few messages. It does not work that way: every token in scope adds to the same running total, and once that total would exceed the model's maximum, something has to give — the oldest turns are dropped or summarized by whatever is managing the conversation, or the request fails outright.

Real numbers vary by model and change as new ones ship, but one concrete, verifiable example: Meta's own model card for Llama 3.1 states that its 8B, 70B and 405B variants all support "128k" tokens of context — a large, genuinely useful budget for a long-running local agent conversation. Treat that number as one worked example, not a ceiling for every local model — check a specific model's own card before relying on its figure, since this territory moves quickly.

Your runtime's window is not automatically the model's maximum

Here is the detail that catches builders who have only read the model card: a model's maximum context length and the context length your local runtime actually gives it are two different numbers, and the second one is frequently smaller unless you set it.

Ollama's own documentation is explicit about this. It defaults the context length it allocates by the available VRAM — a graphics card's own dedicated memory, separate from the machine's main RAM — not by the model's own ceiling: "< 24 GiB VRAM: 4k context, 24-48 GiB VRAM: 32k context, >= 48 GiB VRAM: 256k context." A Llama 3.1 model capable of 128k tokens can quietly be handed only 4,000 of them by default on a modest GPU — a thirty-second of what its own card advertises. Ollama documents an override, an environment variable set before the server starts: `OLLAMA_CONTEXT_LENGTH=64000 ollama serve`. Its own guidance is direct about when to raise it: "Tasks which require large context like web search, agents, and coding tools should be set to at least 64000 tokens" — exactly the workload this course builds toward.

The practical habit this lesson wants you to leave with: never assume a model is receiving the context window advertised on its card. Check what your specific runtime configuration actually allocates, especially for any bounded task — the course's bounded task design module's territory — that depends on the model seeing everything you gave it.

A worked example: the assistant that forgot its own instructions

A builder sets up a local research assistant on a laptop with a mid-range GPU. Its system prompt lists five careful constraints — cite sources, flag uncertainty, never fabricate a page number, and two more. Early in a session it follows all five. Deep into a long working session — several pasted documents, a handful of tool results — its behavior drifts: fabricated citations start appearing, uncertainty flags stop showing up.

The model was not "getting worse." The laptop's GPU falls in Ollama's under-24-GiB tier, so the session was running at the 4k-context default the whole time. As the pasted documents and tool results filled that small budget, the earliest tokens — the five-constraint system prompt — were the first to be pushed out to make room for newer content. The model was still working correctly; it simply could no longer see its own instructions. Raising the context length for that session, well past what the working documents plus the system prompt would need, would have kept the constraints in view for the whole run.

Accessibility notes

This lesson is text-first, with no images, audio, video or downloadable artifacts. The practice exercise's model answer sits behind a native disclosure control that is reachable and operable by keyboard and correctly announced by screen readers. The knowledge check uses native radio-button inputs with a visible question and options, and posts its result to a live status region so assistive technology announces the outcome without a page reload.

Sources and limits

This lesson synthesises the sources below into a practical learning model. It is not a security standard, legal advice or a guarantee that any particular agent design is safe.

1. Key concepts — Tokens — OpenAI. Defines a token as a commonly occurring sequence of characters and gives the rule of thumb that one token is approximately 4 characters or 0.75 words of English text.
2. Llama 3.1 Model Card — Meta. States that the 8B, 70B and 405B Llama 3.1 models all support a 128k-token context length, released 2024-07-23.
3. Context length — Ollama. Documents Ollama's default context-length tiers by available VRAM and the OLLAMA_CONTEXT_LENGTH override, independent of a model's own maximum.