Agent pipelines that called an LLM at every step reportedly made up to 29.6% 11 fewer LLM calls once small decision models took over some of the choices. TLDR highlighted that result this month. Almost a third of those calls had no business going to a big model.
The same week, the big labs pushed in the opposite direction. Google shipped Gemini 4 Argon with up to 1 million 05 output tokens. Meta aimed Muse Wonder at long agentic work in coding, legal analysis and finance.
Both cost money to reach. Meta's Muse plans run $20 09 to $100 a month, and Google puts Argon behind its Pro and Ultra tiers. Every call an agent skips is cash and seconds you keep.
One admission up front. The 29.6% 08 comes from specific tested pipelines, and nobody has shown it holds for every agent. The direction holds across three separate tests this year, though, and the math is simple.
Paying Frontier Prices for Yes-or-No Questions
The branch tax is what an agent pays when a frontier model answers a question a few lines of code could answer. Did the tool return anything? Is the JSON valid? Should the loop stop or retry?
What a 74% routing cut does to a $1,000 Opus bill
Each answer is a branch, and a typical agent sends every branch to its most expensive model. So the tax repeats on every step. Gartner estimated in March 2026 that agentic models use 5 to 30 12 times more tokens per task than ordinary chatbot exchanges. More steps mean more branches. More branches mean more tax.
The fix starts with sorting. Every step in an agent run lands in one of three buckets:
- Deterministic operations, such as policy checks and schema enforcement. Plain code handles these with zero model calls.
- Routine control, such as validation and retries. A small classifier or small model can take these.
- Reasoning-heavy work, such as ambiguous planning and high-risk actions. This is where the frontier model earns its price.
The pattern has shown up three times in 2026. The pipelines TLDR highlighted cut calls 29.6% 08. An ICML workshop paper called AgentRouter cut cost 72% 01, and a LangChain benchmark cut cost 74% 02. Different teams, different setups, same finding: most branches belong in the bottom two buckets.
Inside a 12-Million-Parameter Traffic Cop
Think of AgentRouter as the mailroom clerk who reads the address on each envelope and drops it in the right tray. Nobody asks the CEO to sort the mail. The clerk is cheap and right most of the time.
Rudrendu Kumar Paul and Sourav Nandy presented AgentRouter at ICML 2026 workshops, including the LM4Plan session on July 11. The router is a 12-million-parameter 07 classifier that adds under 5 milliseconds per step. At routing time it reads five features and sends each step to one of four model tiers. The authors trained it on 50,000 annotated steps drawn from a mix of agent tasks.
They report 72% 03 lower cost than a frontier-only baseline, with under 3% 14 drop in end-to-end task completion. Their abstract claims frontier-only enterprise agents waste 60% 23 to 80% of inference budget on subtasks smaller models handle equally well. Older routers applied per step did worse on the same benchmarks: RouteLLM cut cost 31% 15 and FrugalGPT 44% 16. Those tools learned from single-turn queries, according to the authors. They miss how one step's quality affects the next.
LangChain ran a similar test, published August 11, 2026, with NVIDIA's Switchyard router. Routing between NVIDIA's Nemotron 3.5 Lightning and Anthropic's Claude Opus 4.8 cut total cost 74% 04 against Opus alone and kept 93% 17 of the original accuracy.
Say your agent runs up a $1,000 25 monthly Opus bill. A 74% cut leaves $260. That's $740 saved a month, or $8,880 a year. The price is 7% 26 of your accuracy, and only you know whether your workload can absorb that.
AgentRouter's own numbers carry the warning. Per-step routing accuracy hits 91% 18 on minimal-complexity steps and 85% on efficient-tier steps. On the harder mid-range and frontier tiers, it falls to 76% 20 to 82%. Routing works best where mistakes are cheap and worst where they hurt.
Misroutes fail quietly, too. A small model can return a perfectly valid label like "retry" while misreading the state of the run. The schema check confirms the format and misses the meaning entirely. Anthropic's own guidance warns that agents carry higher cost and compounding errors, and it favors predictable workflows for well-defined work.
At the top, capability still matters. Anthropic reports its multi-agent systems beat single agents by 90.2% on tasks needing several independent lines of investigation. AgentArch tested 18 21 configurations and found top models still struggled with complex enterprise workflows. We think the trade pays off for the bottom two buckets and stays risky for the top one.
Cheap routing wins where mistakes are cheap
A tiny classifier cut agent costs sharply.
AgentRouter, a 12-million-parameter 07 router adding under 5 milliseconds per step, cut cost 72% 01 against a frontier-only baseline. Task completion dropped by under 3% 14, and older per-step routers like RouteLLM and FrugalGPT managed only 31% 15 and 44% 16.
Routing accuracy sags on the hardest steps.
Per-step accuracy hits 91% 18 on minimal-complexity steps but falls to 76% to 82% on mid-range and frontier tiers. A small model can return a valid label like "retry" while misreading the run, and the schema check misses it.
The branch tax grows with agent adoption.
Gartner projects 40% of enterprise applications will feature task-specific agents by the end of 2026, up from under 5% in 2025. Nobody knows yet whether frontier prices will fall fast enough to shrink the tax on their own.
2031. Routing predates agents. RouteLLM kept 95% 22 of GPT-4's MT-Bench performance while sending only 26% of queries to GPT-4, but that was single-turn chat. In 2026 the idea moved inside the agent loop: AgentRouter in July, LangChain's benchmark in August, TLDR's 29.6% 08 in October.
The demand underneath is climbing steeply. Gartner projects 40% of enterprise applications will feature task-specific agents by the end of 2026, up from under 5% in 2025. Capgemini Research Institute figures put actual deployment of AI agents at partial or full scale at 14% 24 of organizations.
Run Fortune Business Insights' forecast of 40.5% annual growth for the agentic AI market forward and it compounds fast. 1.405 27 to the fifth power is about 5.5, so $9.14 billion becomes roughly $50 billion by 2031. If agentic models burn 5 to 30 12 times the tokens of chat, the branch tax grows with that whole pile.
The bet is lopsided. A router costs 12 million parameters and 5 milliseconds a step. The savings compound with every step an agent adds, and Gemini 4 Argon's 1 million 05 output tokens invite longer runs with more branches. Bigger models raise the ceiling on what an agent can do. Routing lowers what each step costs.
Nobody knows yet whether frontier prices will fall fast enough to shrink the tax on their own. Our read: teams who tag their steps now will hold a cost edge through 2031, whatever frontier prices do.
Tag Every Agent Call by Its Job
This build takes one afternoon and reruns every week. It works on any agent that logs its model calls, whether you built it in LangChain or plain Python.
Step one: log every model call for a week. Capture the prompt, the output, the model, the token count and the cost. A trajectory is the full chain of steps in one agent run, so tag each call with its run ID.
Step two: label each call with one of the three buckets. A spreadsheet works fine for this. Expect the routine pile to be large. The pipelines TLDR highlighted dropped 29.6% 08 of their LLM calls.
Step three: replace the deterministic calls with code. A JSON schema check, an empty-result check and a stop condition each take a few lines. Those calls drop to zero model cost.
Step four: route the routine calls to a small model with a fallback. A fallback is a rule that sends the step to the frontier model when the small model's answer fails a check. Escalate on task type, novelty, reversibility and money at stake. Confidence scores alone miss the cases where small models are confidently wrong.
Step five: measure the totals. Track cost per finished task, retry rate, escalation rate and severe errors. Fewer calls with more retries can cost you more, so the per-task number is the one that counts.
Expect breakage in week one. The router will misfile a step that looked routine and turned out hard. Find it in the logs, move that step type up a bucket, and run the week again. If you want a ready-made router to test against your own logs, NVIDIA's Switchyard is one place to start.
Sort every agent call into its bucket and cut the waste
- Log every model call for a week. Capture the prompt, output, model, token count and cost, and tag each call with its run ID.
- Label and replace the easy calls. Sort each call into deterministic, routine or reasoning-heavy, then swap deterministic calls for a JSON schema check, an empty-result check or a stop condition in a few lines of code.
- Route routine calls with a fallback. Send them to a small model that escalates on task type, novelty, reversibility and money at stake, then track cost per finished task, retry rate and escalation rate.
Tag your steps now and the savings compound with every branch.
Three separate 2026 tests found the same pattern: most agent branches belong in plain code or a small model. The cuts ranged from 29.6% 08 of calls to 74% 26 of cost, with accuracy losses each team must weigh. Routing is weakest on hard, high-stakes steps, so keep the frontier model there and escalate whenever a check fails. We think teams who audit their call logs this quarter will hold a cost edge as agent runs get longer.