Budget an Agent's Context Window
Treats context as a budget to spend rather than a container to fill, allocating tokens across system prompt, tools, history, and retrieved content, then setting a compaction strategy. Grounded in the finding that every frontier model degrades as input grows.
0 likes
0 dislikes
Sign in to rate this prompt
Prompt
You are an engineer who has learned the hard way that more context is not better context. Chroma tested 18 frontier models and every one of them lost accuracy as input length grew — commonly 20 to 50% degradation between 10k and 100k tokens, with cliffs well before the advertised window limit. Treat the context window as a budget I am spending, not a container I am filling.
What the agent does: {{agent_purpose}}
Model and stated context limit: {{model_and_limit}}
What currently goes into the window: {{current_context_contents}}
Typical run length in turns: {{run_length}}
What the agent must still know at the end of a long run: {{must_persist}}
Produce the following.
1. **An allocation table.** For each category — system prompt, tool definitions, conversation history, retrieved documents, tool outputs, scratchpad — give a target token budget, the actual current usage if I supplied enough to estimate it, and a verdict of keep, shrink, move, or drop. Make the total add to a working budget well below the model's limit, and say what number you targeted and why.
2. **Sort everything into three classes.** Always loaded (small, universal, earns its place every single turn), retrieved on demand (large, only sometimes relevant), and generated then discarded. Most context problems are things in class one that belong in class two.
3. **A compaction strategy for long runs.** When to summarize, what to summarize, what must survive summarization verbatim, and how to keep the agent from losing a constraint it was given in turn three. State the trigger — token threshold or turn count — rather than leaving it to judgment.
4. **Tool output discipline.** Tool results are usually the largest and least curated thing in the window. What should be truncated, what should be summarized before it enters context, and what should be written to a file or store and referenced by handle instead.
5. **The ordering question.** Where the critical instructions sit relative to the bulk content, and what to restate near the end of a long context. Note that the Chroma work found models did better on shuffled haystacks than on logically coherent documents, so intuitions about "natural" ordering are not reliable here.
6. **A test.** How I would confirm the trimmed version is not worse: what to measure, on what inputs, at what context lengths.
Rule: for every item you tell me to keep, say what breaks if it is removed. Anything without an answer should be dropped.