Context Budget and Output Buffer Tuning
Learning Headroom
From Precedence to Practice
**Builds on ← 6.1 Headroom Configuration Scopes and Precedence**
You already know settings can come from request, session, or default scope, and the most specific one wins.
**Today's question:** once a setting IS resolved, what number should it actually be?
- How many tokens can the input actually use?
- How big should the output buffer be?
- How many turns should be protected no matter what?
**Where this leads →** 6.3 Simulation-Driven Configuration Comparison will run these exact numbers through simulated conversations to compare configs side by side.
The Core Formula: Context Budget
**Definitions**
- \(model\_context\_limit\): total tokens the model can hold in one call (input + output combined)
- \(output\_buffer\_tokens\): tokens reserved for the model's reply (config key: `headroom_output_buffer_tokens`)
- \(context\_budget\): tokens available for the input (system prompt + tools + conversation history)
\[ context\_budget = model\_context\_limit - output\_buffer\_tokens \]
**Why subtraction, and why first?**
Input and output share ONE window. Every token given to output is a token taken from input. If you don't reserve the buffer *before* filling input, a full input leaves zero room to reply — the model gets truncated mid-answer.
```mermaid
flowchart LR
A["model_context_limit ("e.g. 128000")"] --> B{"Split"}
B --> C["output_buffer_tokens: reserved for reply"]
B --> D["context_budget: available for input"]
```
Worked Example: Calculating the Budget
**Task:** a coding-assistant agent, model context limit \(= 128{,}000\) tokens, expects long responses (code diffs), so `headroom_output_buffer_tokens` is set to \(8{,}000\).
**Step 1 — identify the inputs**
\(model\_context\_limit = 128{,}000\)
\(output\_buffer\_tokens = 8{,}000\)
**Step 2 — apply the formula**
\[ context\_budget = 128{,}000 - 8{,}000 \]
**Step 3 — subtract**
\[ context\_budget = 120{,}000 \]
**Result:** system prompt + tools + conversation history must fit inside 120,000 tokens. If it needs more, Headroom's live-zone-only compression (3.3) kicks in on the newest turn.
**Contrast case — chat assistant, short replies:**
\(model\_context\_limit = 128{,}000\), \(output\_buffer\_tokens = 1{,}500\)
\[ context\_budget = 128{,}000 - 1{,}500 = 126{,}500 \]
Smaller buffer → more room for input, because short replies don't need much reserved space.
Turn Protection: headroom_keep_turns
**Rule:** the last \(N\) user/assistant turns are always preserved untouched, regardless of budget pressure.
- Config key: `headroom_keep_turns`
- Default: \(N = 2\)
- System messages: always protected too (never dropped, any \(N\))
**Why it exists:** the model needs immediate conversational context to stay coherent — if the last thing the user said got compressed away, the reply won't make sense.
```mermaid
flowchart TD
subgraph History["Conversation history, oldest to newest"]
T1["Turn 1 (old)"] --> T2["Turn 2"] --> T3["Turn 3"] --> T4["Turn 4 (protected)"] --> T5["Turn 5 ("protected, newest")"]
end
T1 -. eligible for compression .-> Note1((( )))
T2 -. eligible for compression .-> Note1
style T4 fill:#166534,color:#fff
style T5 fill:#166534,color:#fff
```
**Trade-off:**
- Higher `keep_turns` → more guaranteed recent context, but less room in the budget for compression to save space elsewhere.
- Lower `keep_turns` → more flexible budget, but risk of losing near-term context on volatile tasks.
Tuning Both Together for a Task
**Task:** a customer-support agent. Model context limit \(= 32{,}000\). Replies are short (a few sentences). Users often reference something said 2–3 messages ago, so recent turns matter a lot.
**Step 1 — choose output buffer**
Short replies → small buffer is enough: \(output\_buffer\_tokens = 1{,}000\)
**Step 2 — compute context budget**
\[ context\_budget = 32{,}000 - 1{,}000 = 31{,}000 \]
**Step 3 — choose keep_turns**
Users reference the last 2–3 exchanges → set \(keep\_turns = 3\) (above the default of 2) to guarantee that context survives.
**Step 4 — sanity check**
Assume each turn (user + assistant) averages 400 tokens:
\[ 3 \times 400 = 1{,}200 \text{ tokens protected} \]
Compare against the budget:
\[ \frac{1{,}200}{31{,}000} \approx 0.039 \approx 4\% \]
Only about 4% of the budget is locked up by protected turns → plenty left for system prompt, tools, and older history to be compressed if needed. Config is safe.
**If instead** the task were long-document analysis (huge single tool outputs, replies referencing only the current document): lower `keep_turns` toward 1–2 and raise `output_buffer_tokens` for longer analytical replies.
Edge Cases and What's Never Touched
**What compression never touches, at any settings:**
- System messages (persona, instructions, tool descriptions) — never dropped
- Tool definitions — never dropped
- The last `headroom_keep_turns` turns — never compressed
- Older turns and the cache hot zone — forwarded byte-for-byte, only the **live zone** (newest user message + latest tool result) is ever compressed
**Boundary condition:** if \(output\_buffer\_tokens \geq model\_context\_limit\), then \(context\_budget \leq 0\) — no room for input at all. Invalid configuration.
**Common tuning mistake:** setting `output_buffer_tokens` too small for the task (e.g. 200 tokens for an agent that writes long reports) → replies get cut off mid-sentence, not because compression failed, but because the reserved reply space was too small from the start.
**Where this leads →** 6.3 will let you simulate configs like these against real conversations and directly observe truncation or over-compression before deploying.
Back to course