State of LLM Routers in 2026
thou shalt not route to the wrong path
Every few months the AI industry seems to discover a new optimization. The focus moved from prompt engineering to retrieval, then to reasoning, then to agents, and more recently towards routing.
Infrastructure providers are adding routing rules. Agent companies are routing between frontier and smaller models. Research papers are proposing learned routers, cascades and verification loops. And open-source systems are beginning to expose routing as a first-class primitive. If you only look at the individual announcements, the trend feels fragmented, because each company appears to be solving its own local problem in its own corner of the stack.
Recently, OpenRouter introduced Fusion, a system that fans a request out to multiple frontier models before synthesizing a final answer, arguing that some tasks justify spending additional inference to reduce uncertainty.
Cognition released Devin Fusion, in which a frontier model collaborates with a lower-cost “sidekick” throughout a software engineering task rather than relying on a single model from beginning to end.
Harvey publicly explained why its legal platform is multi-model by design, because different models exhibit different strengths depending on the legal workflow being executed.
Vercel added routing rules to its AI Gateway, allowing organizations to express deployment policy, provider preferences and operational constraints without changing application code.
Splunk, framed the same move as basic cost discipline in reduce agent cost by model routing, matching each step of a workflow to the least expensive model that can handle it reliably.
And just this week, vLLM announced Semantic Router, describing it as the next step toward a “Mixture of Models” architecture that composes routing, policies, reasoning budgets and workflow signals into a unified inference layer.
These companies are solving different problems, serving different customers and building on different stacks, and yet I don’t think these are independent innovations so much as a convergence on the same architectural layer.
The industry still calls these systems routers, but that name increasingly undersells what they are becoming, because over the next few years the interesting competition in AI infrastructure may not be between foundation models at all, but between the systems that decide how intelligence is allocated across a computation.
The router, in that sense, is simply the first visible implementation of what is gradually becoming an intelligence control plane.
Every Platform Eventually Builds a Scheduler
Operating systems schedule CPU time because processors are shared resources, databases build query planners because there are many ways to answer the same question with dramatically different execution costs, and networks rely on routing protocols because packets can travel across many possible paths. In the same vein, Kubernetes decides where containers should execute, and GPU schedulers determine which training jobs receive scarce accelerator time. These systems emerged independently, but they all exist for the same reason: once a resource becomes sufficiently valuable, deciding where it should be spent becomes nearly as important as the resource itself.
For most of software history, computation was front-loaded, in the sense that building the product required engineering effort, while serving another user was relatively inexpensive. AI changes that equation because every interaction now purchases fresh computation. Reading another page of context, running another retrieval query, thinking for another second, invoking another tool or verifying another answer all incur incremental inference cost. Intelligence has become something that is consumed continuously rather than something that is embedded once inside an application. Ben Thompson made the same point from the economics side when he argued that in AI marginal costs are back in a big way. Unlike traditional software, inference is a real cost of goods sold that scales almost linearly with revenue, which is precisely why deciding where those tokens go has stopped being an afterthought.
This shift is easy to miss because we still talk about models as though they are the product, comparing GPT against Claude, Claude against Gemini, and Gemini against DeepSeek. Benchmarks reinforce this framing by asking which model achieves the highest score on a particular dataset, and product launches encourage it even further because every release introduces another model that is supposedly more capable than the last. That perspective was useful when an application largely consisted of a prompt and a completion, but it is becoming less useful now that production systems increasingly resemble workflows instead of conversations.
Consider a coding agent, where the final code generation step is only one small part of what actually happens. Before any code is written, the system searches repositories, reads documentation, identifies relevant files, constructs a plan, reasons about architecture, edits multiple locations, executes tests, diagnoses failures, repairs regressions and finally explains the result to the user. A document-processing system follows a similarly complex path. It classifies document types, separates digital text from scanned images, invokes OCR, detects tables, identifies charts, reconciles extracted values and validates structured outputs before returning anything useful. Legal assistants, spreadsheet agents and research systems all exhibit the same pattern.
They are no longer making one inference; they are orchestrating dozens of computational decisions whose costs, uncertainties and consequences differ substantially.
Once work is distributed across a trajectory rather than concentrated inside a single model invocation, the optimization problem changes, so that the obvious question is no longer which model should answer this prompt? but rather where should intelligence be spent during this workflow?
That distinction may seem semantic at first, yet it changes the way we think about routing entirely. A router that merely chooses between GPT and Claude is solving a relatively narrow classification problem. A different kind of system is solving something much broader: one that decides whether additional reasoning is worthwhile, whether retrieval should happen before reasoning, whether a specialist parser is preferable to a frontier model, whether verification should trigger another round of inference, whether cached computation can be reused, or whether uncertainty justifies consulting multiple independent models. That system is allocating intelligence under constraints of cost, latency and uncertainty.
This is why so many recent announcements look superficially different while feeling strangely related.
Customers Buy Completed Work, Not Tokens
That idea also explains why routing has suddenly become economically important, because if intelligence were effectively free there would be little reason to optimize its distribution, and applications could simply send everything to the strongest available model and ignore the cost. The fact that nobody is building products this way is revealing, since every serious AI company now worries about inference economics precisely because inference has become part of the product’s marginal cost. Yet even this framing often remains incomplete, because we continue measuring the wrong quantity.
Token prices dominate discussions of AI economics because they are visible, as every provider publishes prices per million input and output tokens, and the spread is enormous. Those numbers are useful, but they turn out to be a surprisingly poor measure of economic efficiency.
Imagine two coding agents. The first uses an expensive frontier model, produces the correct patch immediately and passes every test on the first attempt. The second routes the task through a cheaper model, retries several times, accumulates additional context, fails validation, escalates to a stronger model and finally succeeds after multiple repair attempts. The second workflow may have consumed significantly cheaper tokens while costing substantially more to produce the same outcome. The same phenomenon appears in document understanding, customer support and spreadsheet automation, because cheap inference often becomes expensive once failures trigger additional computation.
The deeper truth is that customers never purchase tokens; they purchase completed work, which suggests a different objective function in which intelligent systems should minimize cost per accepted task rather than cost per token.
This equation looks almost trivial, but it changes the optimization target entirely. Suddenly retrieval, verification, reasoning, routing and human escalations all become part of the same economic system, and model pricing becomes only one variable among many. It is also the exact conclusion the practitioners reach: a model can cost eighty percent less per call and still lose money if it triples the failure rate or sends more work to human reviewers.
Once viewed this way, routing starts to look less like a machine learning classifier and more like a resource allocator. That raises a deeper question. If intelligence is now a resource that can be allocated, how should we decide where the next unit of intelligence creates the most value? That single question turns out to connect everything into one coherent framework.
The Expected Value of Intelligence
One of the more interesting developments over the past eighteen months is that the industry has quietly started exposing how much intelligence a system should use, not just which intelligence it should use, as OpenAI introduced reasoning effort, Anthropic added configurable thinking modes, and Gemini exposed thinking budgets. These features are often described as product improvements, but architecturally they represent something more significant, because they acknowledge that inference is no longer a binary decision and that intelligence can instead be purchased in varying quantities. That immediately raises an economic question, namely, if another second of reasoning costs money, when is it worth paying for?
Research on test-time scaling has largely answered the first half of that question, showing that additional inference frequently improves capability without changing model parameters, and that models allowed to decompose problems, revisit assumptions or evaluate alternative solutions, consistently outperform identical models forced to answer immediately — an observation on which the recent wave of reasoning models is built almost entirely. The second half of the question has received much less attention, however, because it asks when a system should stop buying additional intelligence, and this is exactly the point where routing and reasoning become the same problem.
The implication is surprisingly powerful, because a routing system should not simply ask which model is most capable but should instead estimate the expected value of another unit of intelligence, which depends on several things simultaneously:
How likely additional computation is to change the answer.
What happens if the current answer is wrong — the cost of a mistake.
How expensive another round of inference is.
How much latency remains acceptable.
Whether the uncertainty comes from missing information, weak reasoning or an unreliable execution step.
What is striking is that none of these questions mention a specific model, so that the model has effectively become an implementation detail while the real optimization problem is one of allocating additional intelligence only where it produces meaningful gains. I find it useful to think about this as the Expected Value of Intelligence (EVI). Suppose a workflow has already produced an intermediate result and the system now faces a choice between returning the answer immediately or spending another unit of computation, whether that unit takes the form of deeper reasoning, another retrieval query, a stronger model, an additional verifier or even a human review. The expected value of that decision can be expressed conceptually as
where ΔP(Success) is the increase in the probability of successfully completing the task, V(Success) is the value of completing it correctly, and ΔC is the additional cost of purchasing more intelligence.
This equation is intentionally simple, and it is not meant to be a deployable routing algorithm so much as an explanation of why routing architectures that appear unrelated so often turn out to be solving the same optimization problem.
Take OpenRouter Fusion, which intentionally spends more computation than a traditional router by having several models work independently before a judge synthesizes the final answer, an approach that would appear irrational if routing were fundamentally about minimizing cost. Yet it makes perfect sense through the lens of expected value, because on difficult research tasks reducing uncertainty may produce more value than the additional inference costs, so that the optimization target shifts from minimizing compute → maximizing confidence.
The same framework explains Devin Fusion. Most routing systems make one decision before the first token is generated: they examine the prompt, classify it as easy or difficult, and select an execution path. Software engineering rarely behaves that way. A task that initially appears simple may reveal architectural complexity once the repository has been explored, just as a task that seems intimidating may collapse into a mechanical refactoring once the relevant files are identified. Devin Fusion therefore keeps both a frontier model and a smaller sidekick active throughout the trajectory, because the value of additional intelligence changes as execution unfolds. It continuously updates its estimate of expected value rather than committing to a single decision at the beginning.
This distinction becomes clearer once we stop thinking about prompts and start thinking about trajectories. A software engineering agent may make fifty model calls before returning a result. A research agent may retrieve documents, summarize them, identify contradictions, formulate new search queries and repeat the process several times. A customer-support agent may invoke databases, policy engines and external tools before composing a final response. The important observation is that each step has a different relationship between cost and value.
Planning usually consumes few tokens while determining the direction of the entire trajectory. Repository search consumes many more tokens but is comparatively easy to verify. Document extraction is computationally expensive but structurally constrained, and reflection may improve difficult reasoning while adding almost nothing to routine work. If intelligence is allocated uniformly across all of these stages, the system almost certainly wastes computation.
FrugalGPT, one of the earliest routing papers, approached the problem through cascades that attempted inexpensive routes first and escalated only when necessary, reporting dramatic cost reductions while maintaining competitive quality on its evaluated tasks. The headline numbers attracted attention, but the deeper contribution was architectural, because FrugalGPT implicitly treated intelligence as something that could be purchased incrementally rather than all at once.
RouteLLM took a different approach by learning routing policies directly from preference data and predicting whether the stronger model would provide enough additional value to justify its higher cost, and here again the important idea was not merely another classifier but another attempt to estimate the expected value of additional intelligence before purchasing it. Later work has made the theory explicit, since a unified treatment of routing and cascading shows the two to be points on the same spectrum of when-to-escalate decisions, while When to Reason extends the same logic to reasoning itself by learning when a model should think harder rather than merely which model should respond.
Viewed separately these papers describe different routing strategies, but viewed together they reveal a broader transition in which the field is gradually moving from asking which model should answer? to asking how much additional intelligence is worth buying right now? That distinction becomes especially important once we examine why generic routing systems eventually stop improving, a reason that turns out to have very little to do with models and everything to do with information.
The Limitation Isn’t the Router — It’s What the Router Knows
One of the more surprising findings from the last generation of routing research is not how well routers perform but where they stop improving, because early papers made it seem as though routing was simply another supervised learning problem in which you could collect enough prompts, observe which model performed better, train a classifier, and watch the system converge toward an efficient allocation policy. Reality has been less cooperative.
Benchmarks such as RouterBench and LLMRouterBench were created to evaluate this assumption under controlled conditions, collecting hundreds of thousands of model-task pairs across many models and datasets, and the results were simultaneously encouraging and humbling. There was clear evidence that routing works, since different models really do have complementary strengths and there are genuine opportunities to reduce cost while maintaining quality, and yet the strongest routing systems remained surprisingly far from an oracle that already knew in hindsight which route would have been optimal for every task, with more sophisticated classifiers often producing only incremental improvements over relatively simple approaches. A dedicated study of the routing plateau made exactly this its central finding, showing that generic prompt-level routing saturates well below its ceiling and that adding classifier capacity does not close the gap.
That observation is easy to misinterpret, because one explanation is simply that routing algorithms need to become better, whereas the more interesting explanation is that the remaining signal is no longer contained in the prompt at all. A prompt is an unusually compressed description of the work that follows, so that instructions like “review this contract,” “fix this bug,” “extract these invoices,” or “summarize this report” reveal remarkably little about the actual computation the system will eventually perform. The important information often appears only after execution has begun.
A contract may contain an unfamiliar clause requiring deeper legal reasoning. A bug may turn out to involve concurrency rather than syntax. An invoice may be handwritten instead of digitally generated, or a retrieval step may return contradictory evidence. In each case the workflow gradually uncovers the difficulty of the task as it progresses. This is exactly the gap that TwinRouterBench was built to measure. By studying routing decisions inside agent trajectories rather than at the prompt boundary, it shows that the information needed to route well is generated by execution rather than available before it.
This explains why production systems increasingly route using signals that never appear in the original prompt. Harvey does not simply inspect the user’s question before choosing a model. It understands whether the work involves due diligence, litigation, drafting or document review, because those distinctions emerge from the legal workflow itself, and that is precisely why it argues for a multi-model architecture rather than committing to a single provider.
A coding agent, in the same way, knows which files have already been explored, how many tests have failed, whether the repository is monolithic or modular, and which tools have already produced useful evidence.
A document-processing system knows whether OCR confidence is deteriorating, whether table boundaries have been detected successfully, and whether extracted values reconcile across pages.
None of these signals exists at the moment the prompt arrives. This changes the way we should think about routing entirely: a generic router sees language, while a production application sees state. That distinction matters because state carries information about the computation that has already occurred. Every tool invocation, retrieval query and intermediate result reduces uncertainty, and every failure exposes something the initial prompt could never reveal. A router operating inside the workflow therefore has access to a far richer description of reality than a router operating only at the API boundary.
The Minions work sharpens the point in a different way, because when its authors paired a small on-device model with a frontier cloud model on long-document tasks, they found that how the work was divided mattered more than which models were used: a naive multi-turn conversation between the two models retained only about 87% of frontier quality, while giving the frontier model the job of decomposing the problem into focused subtasks that the local model ran in parallel recovered nearly 98% of that quality at a fraction of the cost. Changing only the handoff, and not the models, produced a dramatically better quality-cost tradeoff. Small models are reliable when given a clean boundary, a compact input and a checkable answer, but unreliable when forced to preserve an ambiguous plan across a long chain of steps. The unit of optimization, in other words, is not the model but the structure of the computation.
This is also why recent industrial systems increasingly blur the line between routing and orchestration, since the router is no longer making a single decision before execution begins but participating in the execution itself. The vLLM Semantic Router work is particularly interesting for this reason. Rather than framing routing as a prediction problem that asks which model should answer?, it treats routing as a programmable layer that composes many independent signals into an execution policy: reasoning budgets, deployment constraints, privacy requirements, modalities, workflow state and semantic intent. Routing, in other words, stops being a classifier and starts behaving like a policy engine that continuously interprets the state of an evolving computation.
There is a historical parallel here, because early database systems also relied on relatively simple heuristics for query execution, and as databases grew larger and workloads more diverse, those heuristics gave way to cost-based query optimizers that estimate the cost of many possible execution plans before choosing one. Crucially, the optimizer does not optimize SQL syntax but the computation required to answer the query, and AI systems appear to be moving toward an analogous architecture in which prompts become declarations of intent and the intelligence control plane becomes responsible for constructing and adapting the execution plan.
Once routing is viewed this way, another component quietly moves to the center of the architecture. Every routing decision is ultimately a prediction about value — the prediction that another retrieval call will help, that another model is unnecessary, or that verification can safely terminate the workflow. Predictions are only useful when they can be corrected. A router that never learns from its mistakes gradually becomes conservative, expensive or brittle, so it needs a feedback mechanism capable of telling it whether spending additional intelligence actually improved the outcome. That feedback mechanism is the evaluator. The same practitioners who advocate routing are emphatic that it only works with tracing and evaluation underneath it, recording the cost, latency and outcome of every step while scoring whether each output actually meets the bar.
The more I look at modern routing systems, the more I am convinced that the evaluator, rather than the router, is where the real competitive advantage is accumulating. The router asks a question; the evaluator determines whether the answer was worth the cost. Once that evaluator exists, routing becomes an online optimization problem rather than a collection of heuristics, and it becomes possible to estimate not just whether a task succeeded but whether the additional intelligence that was purchased created enough value to justify itself. That is the point at which intelligence stops being a fixed capability of a model and becomes an economic resource that can be measured, allocated and continuously improved.