AI agents turn one request into dozens of API calls. That fan-out multiplies two costs that were small with human traffic. The first is work your system repeats on every call. The second is locks that make calls wait in line. Remove those costs before you add servers. monday.com cut per-call overhead at its API edge and now handles 2.5x the traffic on 92% less CPU. GitHub is narrowing a write lock so that its busiest repositories can use new capacity.
Why Agent Traffic Changes the Load on Your API
A person who uses an app makes a few requests per task. An agent makes many. monday.com gives a direct example. A user asks for one change to a project. Then the agent that carries it out makes "dozens of calls to the monday.com API". Its steps are to "find the board, read every item, check the dates, update each task, notify each owner".
The volume numbers from the sources show the scale:
| Organization | Traffic fact | Source type |
|---|---|---|
| monday.com | Weekly API edge requests grew from about 1 billion to about 2.5 billion in one year | Engineering blog |
| GitHub | Its Git storage layer, Spokes, handles "well over 100 billion operations per day" | Engineering blog |
| AWS | Claims that "AI agents now represent 30–60% of their total traffic," with no stated method | Vendor security blog |
The AWS figure is from the AWS WAF AI traffic analysis announcement. AWS also says its Bot Control now tracks "more than 650 unique bots and agents." Treat the 30-60% range as AWS's claim. The announcement does not say how AWS measured it.
monday.com first responded the usual way. Its engineers wrote that "keeping up meant repeatedly increasing CPU and memory resources". They added that "the edge started taking a growing share of our on-call time". The rebuild came after that approach stopped scaling well.
Three Costs That Multiply With Fan-Out
A fixed cost per call is the time or compute your system spends on every request, whatever the request does. Examples are parsing, authentication, and network hops between processes. With fan-out, the user pays that cost once per agent call.
A serialization point is any place where requests must run one at a time. A lock is the most common example. With fan-out, more requests arrive at that point together, so the queue gets longer.
Both companies found versions of these three patterns:
| Pattern | What it looks like | Where it appeared |
|---|---|---|
| Late rejection | A request that will fail a rate limit is parsed and routed before the system stops it | monday.com |
| Repeated work across hops | The same request crosses process boundaries and is parsed again at each one | monday.com |
| Lock wider than the consistency rule | Writes that do not conflict still wait for each other | GitHub |
A worked example of per-call overhead
This is a hypothetical example. It uses monday.com's published median edge times and assumes an agent task of 30 calls that run one after another.
- Take the old median edge time: about 80 ms per call.
- Multiply by 30 calls: 30 × 80 ms = 2,400 ms of edge time for one task.
- Take the new median edge time: 9 ms per call.
- Multiply by 30 calls: 30 × 9 ms = 270 ms.
- Subtract: the task spends about 2.1 seconds less in the edge.
| Calls per task | Edge time per call | Edge time per task |
|---|---|---|
| 1 (human click) | 80 ms | 80 ms |
| 30 (agent task, old edge) | 80 ms | 2,400 ms |
| 30 (agent task, new edge) | 9 ms | 270 ms |
For one human click, 80 ms of overhead is hard to notice. For an agent task, the same overhead adds more than two seconds. It also costs compute 30 times.
monday.com: Decide at the Gateway, Then Route
monday.com's old edge was an Apollo Router with a Node.js coprocessor. A coprocessor is a separate process that the router calls to run custom logic. All the business rules lived in that coprocessor.
The design had two problems. First, the router made repeated hops. "At several points in every request, the router had to stop, send the request to the coprocessor over HTTP, and wait for its answer". Because of these hops, "the same request was parsed multiple times". Second, rejections came too late. Some requests were ones "that a rate limit should have rejected immediately". These "were only stopped deep inside the router's lifecycle". By then, they "had already been parsed and passed back and forth".
The new edge splits the work into two layers. A Go gateway runs authentication, authorization, rate limits, complexity budgets, versioning, agent detection, and observability. "If a request fails a check, the gateway rejects it on the spot, and the router never sees it". Behind the gateway, monday.com runs its own fork of WunderGraph Cosmo, an open-source GraphQL router written in Go.
The router keeps one rule. "The one rule it enforces is the complexity budget the gateway set, because estimating a query's cost needs the query plan". A complexity budget is a limit on how much work one query may ask for. The gateway sets the budget, and the router checks the query against it after it plans the query.
The results
monday.com reports that "the edge now handles 2.5x the traffic on 92% less CPU and 80% less memory". The source gives percentages only, so the chart uses an index where the old value is 100.
Here is the per-request view of the same numbers. It is our calculation. It uses monday.com's own pairing of 2.5x traffic with the CPU and memory reductions. It also assumes that the traffic ratio holds in the September weeks when CPU and memory were measured.
- Formula: CPU per request after ÷ CPU per request before = (CPU after ÷ CPU before) ÷ (traffic after ÷ traffic before).
- CPU: 0.08 ÷ 2.5 = 0.032. Each request uses about 3% of its old CPU, about 31x less.
- Memory: 0.20 ÷ 2.5 = 0.08. Each request uses about 8% of its old memory, about 12.5x less.
monday.com states a related headroom claim: "Even at 10x today's traffic, the edge would use less CPU than the old edge needed a year ago."
The p99 time the edge adds fell from about 490 ms to 44 ms. The p99 is the time that 99% of requests stay under. 490 ÷ 44 = 11.1, which matches monday.com's "11x lower" figure. The drop came in two steps:
| Step | p99 added latency |
|---|---|
| Old edge | about 490 ms |
| After cutover to the new edge | about 200 ms |
| After authorization rework (Sep 2026) | roughly another 150 ms off |
| Current (measured Oct 7, 2026) | 44 ms |
The step sizes are monday.com's rounded figures, so they do not add up exactly to the current 44 ms.
monday.com started this latency metric in March 2026. Therefore the latency "before" values come from spring 2026, and the CPU and memory "before" values come from September 2025.
Many small fixes added up
The gateway split was the largest change, but many smaller fixes also contributed:
- File uploads are streamed. The edge no longer buffers the full file in memory.
- "Each response is held in memory once, not three times."
- Federated tracing data is parsed from the wire format. This "halved its share of router CPU usage."
The migration was careful. monday.com copied only read-only traffic to the new edge as shadow traffic. "Writes were never copied." Feature flags controlled the rollout. A second call to the old router filtered out noise, in the style of Twitter's Diffy comparison tool. The real differences were in "validation, null handling, error shapes, and query cost calculation."
GitHub: Narrow the Write Lock to One Reference
GitHub's problem is on the write path. Git data at GitHub is stored on several replicas, which are full copies of a repository on different machines. Most repositories have five replicas. A quorum is the group of replicas that must agree before a write commits.
The quorum "only needs a majority to commit a write". But GitHub adds that "it waits for every healthy replica to respond (up to a timeout)". Each write also holds a lock on the whole repository until it commits. GitHub states the result directly. "The slowest healthy replica sets the latency of every write". Each write "holds a repository-wide lock until it commits". So the slowest replica "also sets the throughput ceiling".
This became urgent as GitHub moved replicas into Azure. GitHub says the move's "timing and shape" were "driven by the agentic shift". It names two drivers: "rapidly growing traffic, a hard ceiling on the power capacity of our data centers". The Azure region is "about 1,000 miles from our data centers". As a result, each network round trip takes about 25 ms. A write needs several round trips.
The fix comes from Git's own rules. "Git's consistency requirements apply to each reference individually." A reference is a named pointer such as a branch or a tag. Two pushes to different branches cannot conflict, so they do not need to wait for each other. GitHub is "changing the scope of the quorum from the whole repository to individual references."
After the change, "only writes that contend on a shared reference must wait for each other". GitHub's virtual write replica also "coalesces transactions on disjoint references into a single set of network I/Os". A busy repository then pays for one round trip across many concurrent writes.
A worked example of a lock ceiling
This is a hypothetical example. GitHub does not publish a round-trip count per write, so assume four round trips of 25 ms.
- Compute the time for one write: 4 × 25 ms = 100 ms.
- With one write in flight per repository, the ceiling is 1,000 ms ÷ 100 ms = 10 writes per second.
- Now let agents push to four different branches at the same time.
- With one lock per reference, the four writes can run together. The theoretical ceiling becomes about 40 writes per second.
- Compare this with GitHub's measured result. In internal testing, writes to different references "consistently saw 3–4x higher throughput at lower latency."
GitHub describes the general rule. The change should "raise a repository's write throughput ceiling roughly in proportion to how many references are being written at once."
Limits of GitHub's change
- It is in progress. GitHub writes "we're changing." The 3-4x figure is from internal testing with realistic traffic. The busiest repositories have not all moved yet.
- Same-branch writes still wait. Writes to one reference still run one at a time, and "the quorum remains the bounding factor."
- New capacity is part of the story. GitHub says "Azure absorbed our growth." Today most repositories have two of their five replicas in Azure, and Azure serves about half of Git traffic. The narrower lock lets the busiest repositories use that new capacity.
How Cloud Gateways Handle Agent Traffic
monday.com and GitHub describe agent traffic that arrives at their own services. Cloud vendors now sell gateways that govern traffic from agents to models and tools. The products differ in purpose, but some controls apply to both directions.
AWS, Amazon Bedrock AgentCore gateway. AWS added per-user rate limits for AI traffic. Teams can "define OAuth or IAM-based rules for requests per minute, concurrent connections, and token throughput." The three limit types count different resources:
| Limit type | Unit | What it protects |
|---|---|---|
| Request limit | Requests per second or minute | Call volume. A 50 ms request and a 90-second streaming request each count as one unit |
| Token limit | Tokens per minute | Model capacity, for inference targets. The gateway estimates tokens first and reconciles after the call |
| Connection limit | Concurrent connections | How long each request holds an open connection |
The request-limit example shows why one unit is not enough for agents. A long streaming call and a short lookup use very different resources.
Microsoft, Azure API Management AI gateway. Microsoft describes the AI gateway in Azure API Management as a set of capabilities for managing AI backends. It covers authentication, load balancing, token quotas, MCP servers, and agent-to-agent (A2A) APIs. Microsoft notes that it "extends API Management's existing API gateway; it's not a separate offering."
AWS WAF. WAF is a web application firewall. Its AI traffic dashboards identify agent and bot traffic at the edge. You can then use those labels in gateway rules.
How the Approaches Compare
| monday.com | GitHub | AWS AgentCore gateway | Azure APIM AI gateway | |
|---|---|---|---|---|
| Problem | Per-call overhead at the API edge | Write lock wider than the consistency rule | Unbounded agent calls to models and tools | Governing AI backends and agent APIs |
| Main change | Go gateway rejects first; router only routes | Quorum per reference | Per-user request, token, and connection limits | AI features added to the existing API gateway |
| Reported result | 2.5x traffic, 92% less CPU, 80% less memory, 11x lower p99 | 3-4x write throughput in internal tests | No benchmark published | No benchmark published |
| Status | In production | In progress | Product feature | Product feature |
Which Response Fits Your Situation
| Your situation | Start here |
|---|---|
| Your edge CPU grows faster than your request count | Look for repeated parsing and process hops per request, as monday.com did |
| Rate-limited requests still use real CPU | Move rejection to the first layer that sees the request |
| Writes queue up while CPU stays low | Look for a lock that covers more data than one consistency unit |
| You call models and tools from many agents | Add per-caller limits on requests, tokens, and connections at a gateway |
| You run on a managed API gateway | Turn on its agent and token controls before you write custom code |
The recommendation for most teams is to fix the edge first. Edge overhead is usually easier to measure and change than storage locks. If your writes queue while CPU stays low, fix the lock first.
Audit Your Edge and Write Path
- Pick one real agent task and count its API calls.
- Trace one call from the network edge to storage.
- Count the times the request is parsed or serialized.
- Count the process boundaries the request crosses, such as sidecars, coprocessors, and plugins.
- Find where a rate-limited request is rejected. If it is rejected after parsing or routing, move the check earlier.
- List every lock and queue on the write path.
- For each lock, write down the unit your consistency rule needs, such as one record, one branch, or one account.
- If a lock covers more than that unit, narrow it.
- Test the change with read-only shadow traffic first. Do not copy writes.
- Measure added latency at p50 and p99, and CPU per request, before and after.
For concurrent agent writes to the same record, also read Two Agents, One Record. For a related queue problem in build systems, see The CI Bottleneck From AI Coding. For API design aimed at agent callers, see Background Coding Agents Need an Agent-First API Layer.
When This Is the Wrong Fix
- Your traffic is mostly heavy work. If most CPU goes to the business logic itself, removing edge overhead saves little. Scale the services, or make the work cheaper.
- Your writes hit the same record. A narrower lock does not help when agents write to one record. GitHub's same-reference writes still wait. Use idempotency keys and conflict checks instead.
- Your volume is small. A rebuild like monday.com's is a large project. At a few million calls a week, a managed gateway with per-caller limits is usually enough.
- You cannot test safely. If you cannot shadow read traffic, change one hop at a time and measure each step.
Make Your Systems Ready for Agent Callers
When agents start to call your APIs, the cost per call decides your infrastructure bill and how fast the agents finish their tasks. OpenNash puts forward-deployed engineers on your team and works with Fortune 2000 companies. A typical engagement:
- Measure agent fan-out and per-call cost on your real traffic.
- Find late rejections, repeated parsing, and wide locks on the request and write paths.
- Move policy checks to the first layer and set per-caller limits on requests, tokens, and connections.
- Roll out changes with shadow traffic, evals, and before-and-after metrics.
- Train your platform team to run and extend the system.
At the end, you own the gateway configuration, the code, the dashboards, and the runbooks.
Book a call with an OpenNash forward-deployed engineer or email us at hello@opennash.com. In the first session, we review your top AI use cases and the systems they call. We then show which changes give the largest cost reduction and the fastest ROI.