Your Tools Are Blowing the Context Window: Budgeting Tool Output Before It Reaches the Model

Run a typical CRM search tool and look at what comes back: 50 records, each with nested ownership metadata, audit fields, and empty custom properties, landing at around 30,000 tokens. The system prompt you spent three weeks tuning is 2,000 tokens. One tool call just outvoted every instruction you wrote, 15 to 1, and it will keep outvoting them on every subsequent turn because that JSON blob stays in the conversation until the session ends.

This is the most common failure pattern we see in tool-heavy agents, and it is rarely diagnosed correctly. Teams see the agent get vague, forget instructions, or fail mid-task, and they respond by tuning prompts or upgrading models. The problem lives in the tool layer. The fix does too, and it is one of the cheapest reliability improvements available in most agent stacks.

One tool call can outweigh everything else in context

When we instrumented our own agent pipelines at OpenNash, the distribution was lopsided: across tool-heavy sessions, tool results accounted for roughly 60-80% of total context consumption. System prompt, user messages, and the model's own reasoning combined were a minority share. The single largest context item in a typical failed session was almost always one unpaginated API response or log dump.

Some representative costs, from payloads we see regularly:

Two or three of these in one session and a 200K window is functionally gone, because the model also needs room for its own output and reasoning. The agent did not fail at planning. It drowned in payloads it never needed to read in full.

The asymmetry worth internalizing is that the agent usually needed 200-500 tokens of signal from each of those payloads. The other 95% is transport framing, null fields, and detail relevant to no current decision. You are paying full price, in both dollars and attention, to ship noise into the model.

The model degrades long before it overflows

If context exhaustion only mattered at the hard limit, you could treat it as an edge case. It is worse than that. Chroma's context rot study tested 18 models and found performance degrades as input length grows even on tasks the models handle easily at short lengths, and that degradation is uneven and hard to predict. The earlier Lost in the Middle work from Liu et al. showed the now-familiar U-shape: models attend well to the start and end of context and poorly to the middle, which is exactly where your third tool result from six turns ago is sitting.

Vendors advertise the window's ceiling, not its usable range. Elvex's comparison of advertised versus real context windows found effective task performance falls off well before the marketed limit on most frontier models. Treat the advertised number as a hard safety limit and budget against something much smaller.

The practical consequence is that a 30,000-token JSON dump costs you twice. Once in the tokens themselves, and again in degraded accuracy on every decision the model makes afterward, because the signal it needs now competes with noise it ingested four turns ago. Filling the window is not a neutral act that only matters at the boundary.

Give every tool a declared output budget

The fix we ship on nearly every engagement is the same shape: every tool gets a declared output budget, enforced in the tool layer, not requested in the prompt.

Asking the model to "keep results concise" does nothing, because the model is not the one producing the payload. The API is. Enforcement has to happen server-side, between the API response and the model, where you have deterministic control. This is the same argument we made for output contracts and schema validation: reliability mechanisms belong in code, where they always run, not in instructions, where they are sometimes followed.

A budget declaration for a tool looks like this:

tool: crm_search
max_output_tokens: 3000
overflow_policy: truncate_ranked   # head of ranked list carries signal
detail_handle: true                # return record IDs for follow-up fetch
projection: [id, name, stage, owner, last_activity]

For allocation, the heuristic we use: no single tool result over ~10% of usable context, and all tool results in a turn under 50% combined. Against a 200K window with realistic effective limits, that puts most tools at 2,000-5,000 tokens. Search-style tools can go lower. Document readers earn a bigger allowance, paired with an aggressive summarization policy.

When a payload exceeds its budget, the tool layer returns three things: the trimmed content, an explicit marker that trimming happened ("showing 20 of 412 results"), and a handle to get more. That marker matters. An agent that knows results were truncated can decide to paginate. An agent handed a silently truncated list will confidently report that only 20 records exist.

Four ways to spend less per call

Budgets need enforcement mechanisms. Four cover almost every case, and they compose.

Field projection. The cheapest win. Most API responses are 80-95% fields no agent decision ever touches. Strip them in the tool layer before the payload reaches the model. A Stripe invoice needs id, amount, status, customer, and due_date for nearly every agent task; the raw object carries dozens more. Projection alone often cuts payloads 5-10x with zero information loss for the task at hand.

Pagination. The MCP specification defines cursor-based pagination for exactly this reason: return a page, return a cursor, let the client ask for more. The agent-facing version is the same idea. Return the first 20 ranked results plus a "next page" affordance, and let the agent decide whether page two is worth the tokens. Agents rarely need it; ranked heads carry most of the signal.

Server-side summarization. For payloads where signal is diffuse (logs, transcripts, long documents), truncation destroys information. Instead, run the payload through a cheap, fast model or a deterministic reducer before it reaches the agent. An 80,000-token pod log becomes a 400-token summary: error classes, first occurrence timestamps, restart count. The Manus team's context engineering writeup describes a related discipline of keeping context compressible and restorable rather than letting raw observations accumulate, learned across millions of agent sessions.

Reference handles. The technique that ties the others together. Store the full payload outside the context (filesystem, object store, or MCP resources) and return a summary plus an identifier. The agent pulls detail only when a decision requires it. Cloudflare's Code Mode pushes this to its logical end by having the agent write code that calls tools and filters results in a sandbox, so intermediate payloads never enter model context at all. You do not need that full architecture to benefit; a get_detail(id) tool next to every search tool captures most of the value.

Start with projection and pagination. They are deterministic, testable, and cheap. Add summarization where signal is diffuse, and handles once you have somewhere to park payloads.

MCP servers are the worst offenders by default

Most MCP servers are thin wrappers: they take an API response and forward it to the model verbatim. The spec gives servers the primitives to do better (resources for out-of-context storage, cursors for pagination), but a wrapper generated from an OpenAPI schema uses none of them. Connect three such servers and you have imported three firehoses, each pointed at your context window. The tool definitions themselves consume context too, a problem we covered in tool selection for MCP-connected agents, and definitions plus unbudgeted results compound each other.

Anthropic's guidance on writing tools for agents makes the underlying point well: a tool's response is prompt engineering, and high-signal, concise results outperform complete ones. Their teams found response format and verbosity controls among the highest-impact levers when optimizing tools against agent evals.

So when evaluating an MCP server, ask three questions before connecting it. Does it paginate list operations? Does it filter or project fields, or return raw payloads? Does it expose large content as resources rather than inline results? A server that fails all three is not disqualified, but it needs a budgeting proxy between it and your agent. We now put that proxy layer in front of third-party MCP servers by default, because "well-behaved payload sizes" is not a property you can assume from someone else's wrapper.

Measure first, then cap: where this fits in an agent audit

You cannot budget what you have not measured, and teams consistently guess wrong about which tool is the offender. The instrumentation is small: log token counts per tool result in your traces, then look at the per-tool distribution, especially p95. The distribution is usually violently skewed. One or two tools produce the majority of context consumption, and they are frequently tools the team considered boring, like a list endpoint or a log fetcher, not the sophisticated retrieval pipeline anyone worried about.

That skew is what makes this the highest-return fix available. Capping one tool is a day of work and routinely recovers 30-50% of context consumption in the sessions that were failing. Compare that with the downstream alternative, compacting context mid-session, which is lossy, model-mediated, and harder to test. Compaction is triage for context you already spent. Budgets stop the spend upstream, deterministically. You likely need both eventually, but budgets come first because they are cheaper and their behavior is unit-testable.

This measurement step is also where output budgeting slots into how OpenNash runs agent audits: pull traces, rank tools by token consumption, then design the budget, overflow policy, and detail handle per tool before touching prompts or models. Clients own the resulting tool layer outright, and the budgets live in code review like any other contract. If your agent is degrading on long tasks and you suspect the tool layer, book a call and bring one failing trace; the per-tool token breakdown usually settles the diagnosis in the first half hour.

If you are fixing this yourself, the sequence for this week: pull your last 100 agent traces, compute tokens per tool result, and find the p95 offender. Give that one tool a projection list and a token cap with a truncation marker. Rerun your eval set. Then do the next tool. Most stacks only have two or three that matter.