What Is a Context Window?

A context window is the maximum number of tokens a model can consider in a single request. Everything competes for that space: your system prompt, the conversation so far, any documents you retrieved, the tool definitions you exposed, the tool results that came back, and the model's own response.

Deliberately absent from this page: a table of current model context sizes. Any such table is wrong within a couple of months, and every article that publishes one is quietly serving stale numbers a year later. Check the provider's own documentation for the model you are using. What follows is the part that does not change.

Tokens, briefly

Models do not read characters or words. They read tokens — subword fragments produced by a tokeniser. For ordinary English, one token averages roughly four characters, so 1,000 words is somewhere around 1,300 tokens. That ratio is a rule of thumb, not a rule: code, JSON, non-Latin scripts, and unusual proper nouns all tokenise less efficiently, sometimes far less.

Two consequences follow immediately. Billing is per token, with separate input and output rates, so context is literally money. And the window is shared — a long system prompt is permanently subtracting from the space available for actual work on every single call.

Why the window has a limit at all

The constraint comes from the architecture. In the transformer, self-attention compares every token to every other token, so cost scales with the square of the sequence length. Double the input and the attention computation roughly quadruples.

Modern implementations do a great deal to soften that — sparse and windowed attention, better kernels, KV caching — and the practical ceiling has moved from a few thousand tokens to a million and beyond. But the underlying pressure explains why long context is expensive, why time-to-first-token climbs with input length, and why just put everything in the prompt has a cost even when it fits.

The finding that should change how you use it

Here is the part most explanations omit, and it matters more than the size.

In Lost in the Middle: How Language Models Use Long Contexts (TACL, 2023), Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang found that:

performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts.

A model having a large context window is not the same as a model using it evenly. Bury the one paragraph that matters in the middle of a hundred pages and the model may simply not act on it — while still producing a fluent, confident answer, because a missed detail does not announce itself.

This is why fits in the window is the wrong success criterion. The right question is whether the material the model actually needs is positioned where it will be used.

What this means in practice

Put the important things at the edges. Instructions and the immediate task at the start or the end, bulk reference material in the middle. If one retrieved document is far more relevant than the rest, do not let it land in position 47 of 60.

Retrieve less, better. The instinct when a model misses something is to add more context. Usually the correct move is the opposite: fewer, more relevant chunks beat a larger pile. This is the practical argument in RAG vs tools vs long context — a big window makes retrieval quality less forgiving, not less important, because a poor retriever now has room to bury the right answer in noise.

Treat tool definitions as context you are paying for. Every tool you expose contributes its name, description, and full JSON schema to every call. A server with sixty tools can consume a startling share of the window before the user has typed anything, and it makes the model's selection problem harder at the same time — see how to design tools your AI agent can actually use and how to build an MCP server.

Watch what agent loops do to it. An agent accumulates every tool call and every result. Long-running agents hit the window not because any single input was large but because twenty steps of history added up — which is why compaction, summarisation, and external memory exist. See the agentic loop explained and memory for AI agents.

Keep the stable part of your prompt stable. Prompt caching can dramatically reduce the cost and latency of a long, unchanging prefix — but only if it genuinely does not change. Interpolating a timestamp at the top of a system prompt invalidates the cache on every call.

Context window versus context engineering

These get conflated. The context window is a hard limit set by the model. Context engineering is the discipline of deciding what goes inside it — what to retrieve, what to summarise, what to drop, and in what order.

The window is the budget. Context engineering is how you spend it. Larger budgets have made the discipline more important rather than less, because the failure mode moved: it used to be a hard error when you exceeded the limit, and now it is a silent degradation when you fill the space with material the model does not use well.

The short version

The context window is working memory measured in tokens, shared by everything in the request, priced per token, and used unevenly — strongest at the beginning and end, weakest in the middle. Do not treat filling it as the goal. The best-performing prompt is usually not the biggest one that fits.

For the broader picture, the AI development pack collects the rest, and Choose the Right Model for a Task covers picking a model on the hardest case it must handle rather than on headline specs.

Sources

• Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang, Lost in the Middle: How Language Models Use Long Contexts, Transactions of the Association for Computational Linguistics, 2023 — performance is highest when relevant information sits at the beginning or end of the context and degrades significantly in the middle
• Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin, Attention Is All You Need, NeurIPS 2017 — the transformer architecture whose self-attention gives sequence length its quadratic cost