Building an LLM Router: Cost vs Quality at Scale

Every small business that has shipped an AI feature hits the same wall: the model that gives the best answers is the model that eats your margin. GPT-5's pricing does not care about your SaaS pricing. So in early 2026 I built a router that decides, per request, which of four providers should handle the work. This is the full design, including the parts I got wrong the first time.

We run this in production for three client products today: a support triage bot, a document summarizer, and a RAG-based internal knowledge search tool. Together they handle roughly 140,000 LLM requests per month. Before the router, a single-provider setup would have cost about $1,900 per month. The routed setup costs about $550 for comparable output quality, measured by a human-reviewed quality score on a fixed eval set each week.

The core idea is simple: not every request deserves the same model. A "what is my invoice total" question is not a "rewrite this legal clause" question. A router sits between your app and the model APIs, classifies the incoming request, picks a provider and model, enforces a latency budget, and falls back if anything fails.

The Provider Landscape as of October 2026

Four providers carry almost all of our traffic. Pricing moves constantly, so treat these as snapshots and check the current numbers before you copy anything:

Sources: OpenAI pricing, Anthropic pricing, Google AI pricing. We dropped the fourth slot we used to hold for a budget provider because Gemini 3 Flash's pricing made it pointless to maintain a second cheap tier.

The gap between the cheapest and most expensive tier is roughly 40x on input tokens. That gap is your budget. The router's job is to spend as many requests as possible in the cheap tiers without anyone noticing the difference.

Cost vs Quality Routing

The heart of the router is a classification step. Before any provider call, we score the request on three signals:

1. Task risk. Can a wrong answer cause real harm (medical, legal, financial advice) or is it reversible (summaries, drafts, search answers)?
2. Complexity proxy. Token count of the prompt, number of retrieved documents, and whether the prompt contains multi-step instructions like "compare and then recommend."
3. User tier. Paying customers on higher plans get the flagship model by default. Free-tier users get routed aggressively.

Each signal maps to a tier: cheap (Gemini 3 Flash or GPT-5 mini), standard (GPT-5), or premium (Claude Sonnet 4.6 or Gemini 3 Pro). About 78% of traffic lands in the cheap tier, 17% in standard, and 5% in premium.

We did not guess these thresholds. We ran a shadow-traffic experiment for two weeks: every incoming request went to both the cheap and the expensive model, we logged both outputs, and reviewers scored them blind. The result was surprising. For summarization, extraction, and FAQ-style retrieval questions, the cheap tier scored within 3 points of the flagship on a 100-point rubric. For multi-hop reasoning and anything involving numeric reasoning across long documents, the gap jumped to 19 points. That data is what set the routing boundaries, and it is why I tell people not to build a router from vendor benchmark tables alone. Your traffic is not MMLU. If you are building retrieval-heavy pipelines, our building a RAG system for business writeup covers the retrieval side that the router sits on top of.

Building It with the AI SDK

We use Vercel's AI SDK because its provider-agnostic model object makes routing almost trivial. Every model is just a string plus a provider. Here is the core of our router, trimmed for clarity:

const TIER_MODELS = {
  cheap:    ["google:gemini-3-flash", "openai:gpt-5-mini"],
  standard: ["openai:gpt-5"],
  premium:  ["anthropic:claude-sonnet-4.6", "google:gemini-3-pro"],
};

async function route(req) {
  const tier = classify(req); // returns cheap | standard | premium
  for (const model of TIER_MODELS[tier]) {
    try {
      return await generateText({ model, prompt: req.prompt });
    } catch (e) { log(e, model); } // fall through to next candidate
  }
  throw new Error("all providers failed");
}

That is the whole fallback chain. If a provider throws, rate-limits, or times out, we move to the next model in the tier's list and log the failure. In production this catches roughly 0.4% of requests per day, almost always rate limits on the cheap tier during traffic spikes. The logged failures feed directly into our hallucination and failure monitoring pipeline, because a fallback is also a moment where output style can drift.

One real lesson from the build: keep the classification step deterministic and cheap. Our first version called a small model to classify each request. That added 400ms of latency and about 8% to the bill before we noticed. Now classification is plain code: keyword checks, token counts, and user-plan lookups. It runs in under 2ms.

Latency Budgets

Users feel latency before they feel cost. We set explicit p95 budgets per request type: 2.5 seconds for chat, 6 seconds for summarization, 15 seconds for long-document analysis. The router enforces these with a hard abort, and the abort itself triggers the fallback chain.

Two findings from our latency dashboards. First, time-to-first-token varies more than total generation time, and streaming hides most of it. Every one of our product surfaces streams tokens, so our effective perceived latency budget is closer to 900ms, which is the threshold before users start noticing a pause. Second, the cheap tier is not always the fast tier. On prompts over 4,000 tokens, Gemini 3 Flash's 1M context cache hit rate beats the smaller-context models, and it lands inside budget more often than GPT-5 mini. Measure your actual traffic. Your intuition about which model is "fast" is probably based on short prompts.

Fallback Chains and Failure Modes

A fallback chain needs an end. Ours terminates in three ways:

1. Cross-tier fallback. If the cheap tier fails twice, we escalate to the standard tier and flag the request for cost review. This happens in about 0.1% of requests and costs us roughly $11 extra per month. Cheap insurance.
2. Graceful degradation. If everything fails, the support bot answers from cached FAQ content and tells the user a human will follow up. Never show a raw error to a customer.
3. Circuit breakers. If a single provider fails more than 20 times in 5 minutes, we pull it from rotation for 15 minutes. This saved us during the Gemini partial outage in August 2026, where fail-fast plus the breaker kept p95 latency flat while the provider was degraded.

We keep one provider-specific detail per model in the chain: structured output reliability. Some of our endpoints force JSON via the AI SDK's jsonMode. Claude Sonnet 4.6 and GPT-5 handle it near-perfectly. Gemini 3 Flash occasionally emits a preamble before the JSON. Our chain puts the JSON-sensitive endpoints on models where structured output is boring and reliable, because a parse retry costs more than the price delta you saved.

What Most Guides Get Wrong

Most "LLM routing" tutorials reduce the problem to picking a model per request and stop there. Three gaps keep showing up when teams copy those patterns:

They tune for benchmark scores instead of their own evals. Vendor leaderboards measure a different distribution than your traffic. Our routing rules would look wrong if we had built them from public benchmarks. The shadow-traffic experiment cost us about $300 and paid for itself in the first month.

They ignore output-style drift across tiers. A summary written by GPT-5 mini reads differently than one from Claude Sonnet. If your UI promises a consistent voice, mixing tiers naively makes the product feel broken even when the answers are right. We pinned per-endpoint style instructions and pinned the premium tier for customer-facing prose.

They route once at design time instead of continuously. Prompts change, users change, provider pricing changes monthly. We re-run the eval sweep every quarter and have moved tier boundaries three times in nine months. A router you never re-tune becomes a cost liability instead of a cost control.

When Not to Build a Router

Honest counterpoint: if you are under about 20,000 requests per month, a router is premature. At that volume the difference between cheapest and best is maybe $100 per month, and the engineering time plus the eval harness will cost more than that to build. Start single-provider, add a second provider behind a simple failover check, and only build the classification layer once routing would save you more than a day of engineering per month.

Also skip routing if your workload is uniform. If every request is the same shape, like a fixed extraction schema over invoices, you do not need a classifier. You need one good model, a tight prompt, and a fallback. Routing earns its complexity when traffic is heterogeneous.

A Decision Framework You Can Steal

Score your use case on four questions before writing any code:

1. Traffic volume: Under 20K requests per month, skip. Over 100K, a router is almost certainly worth it.
2. Quality variance tolerance: If your users notice a 10-point quality dip, you need stronger tier separation and more premium traffic than you expect.
3. Failure tolerance: If a wrong answer has legal or safety consequences, route those classes of requests to the flagship tier unconditionally. No classification. Hard rule.
4. Team eval capacity: If nobody owns the weekly eval review, the router will quietly rot. Routing is a process, not a component.

Our rough math for clients: routing pays off when the cheapest viable tier costs less than 40% of your current single-model bill and quality variance stays inside your tolerance band. Both conditions are measurable in a two-week experiment, and the same instrumentation discipline carries over if you later formalize how you measure AI ROI for the business.

Conclusion

Multi-model routing turned our AI feature costs from a scary line item into a managed budget. The pattern is not exotic: classify each request on risk and complexity, route it to the cheapest model that historically handles that class well, chain fallbacks per tier, enforce latency budgets with hard aborts, and re-tune the boundaries on a schedule. The AI SDK removes most of the plumbing pain, which means your real work is the eval harness and the discipline to keep measuring. Start with shadow traffic, not with routing rules. The data will tell you where the boundaries belong, and it will usually surprise you.