I Built a Model Router That Picks the Right LLM for Every Call — Here's the Python

Two months ago my codebase had exactly one LLM client, hardcoded to one model. First I ran everything on the cheapest model, because the bill looked great. Then a user showed me a reply that was confidently wrong, and I did what everyone does: I switched everything to GPT-4o. The bill went up 28x and the quality on the easy 90% of traffic didn't improve. The mistake wasn't picking the wrong model — it was picking one model.

The fix turned out to be a router: a few hundred lines of Python that classify each request, send it to the cheapest model that's good enough, validate the output, and fall back to a stronger model when validation fails. This is the step-by-step version, with the real cost numbers from my traffic.

The problem with one model

Every request to an LLM API is not the same task. In my pipeline they split roughly like this:

• Classification and extraction — tickets, intents, JSON fields. High volume, low complexity.
• Summarization — long documents down to a digest. Medium volume, needs coherence.
• Translation — Chinese↔English for a bilingual team. Needs strong language quality.
• Complex reasoning — escalations that need actual deduction. Low volume, high stakes.

If you send all four to the cheap model, the 5% hard ones come back mediocre and you ship bad output. If you send all four to the premium model, you pay $10.00 per 1M input tokens to classify a ticket. Routing is the middle path: a small classifier decides the task, a lookup table decides the model.

Step 1: Pick a model per task

Prices per 1M tokens, as of this writing:

My routing table came from running the same prompts across all four models (the evals are a story for another post). What I settled on:

Step 2: The router

Nothing clever here — a dict of routes and a function that calls the OpenAI-compatible SDK. All four models sit behind one endpoint, so switching models is just a string:

import openai

client = openai.OpenAI(
    api_key="sk-...",
    base_url="https://api.tokencnn.com/v1",  # one endpoint, every model
)

ROUTES = {
    "classify":   {"model": "deepseek-v4-flash", "max_tokens": 150},
    "extract":    {"model": "deepseek-v4-flash", "max_tokens": 300},
    "summarize":  {"model": "glm-5-130b",        "max_tokens": 800},
    "translate":  {"model": "qwen3-235b-a22b",   "max_tokens": 500},
    "reason":     {"model": "gpt-4o",            "max_tokens": 1500},
}

def call_model(model, messages, max_tokens):
    resp = client.chat.completions.create(
        model=model,
        messages=messages,
        max_tokens=max_tokens,
    )
    return resp.choices[0].message.content

def route(task, messages):
    spec = ROUTES[task]
    return call_model(spec["model"], messages, spec["max_tokens"])

Step 3: Classify first, route second

The classifier is just another cheap call with temperature=0 and a tight label vocabulary. The important bit: fall through to the expensive route when unsure — a misclassified hard request sent to a weak model is worse than an easy request sent to GPT-4o:

CLASSIFIER_LABELS = "classify, extract, summarize, translate, reason"

def pick_task(text):
    resp = client.chat.completions.create(
        model="deepseek-v4-flash",
        messages=[{"role": "user", "content":
            f"Label this request with exactly one of: {CLASSIFIER_LABELS}.\n\n{text}"}],
        max_tokens=10,
        temperature=0,
    )
    label = resp.choices[0].message.content.strip().lower()
    return label if label in ROUTES else "reason"  # fail safe, not fast

Step 4: Validate, and fall back

This is where routing earns its keep. Cheap models fail differently, not less often: GLM-5-130B occasionally pauses mid-stream, DeepSeek V4 Flash sometimes returns malformed JSON on gnarly schemas, and both can hallucinate a tool call that doesn't fit the schema. So I never trust the first pass — I validate, and retry with the next tier up on failure:

import json

def route_with_fallback(task, messages, validate):
    spec = ROUTES[task]
    for model in [spec["model"], "gpt-4o"]:  # escalate to premium
        text = call_model(model, messages, spec["max_tokens"])
        try:
            validate(text)          # schema check, exact-match, hidden test
            return model, text
        except (json.JSONDecodeError, ValueError):
            continue
    raise RuntimeError(f"validation failed on both models for task={task}")

The fallback fires on the 2–4% of requests that fail validation, so the GPT-4o calls stay rare — but the option is what makes the cheap model usable on the long tail. Retrying a sub-millicent call beats paying $30.00 per 1M output tokens on everything.

What routing actually costs

Here are the real numbers from 10,000 requests/day, ~1,680 input and ~190 output tokens per request on average. First, what you'd pay if every request went to a single model:

STEPS = 10_000        # requests per day
IN_PER_STEP = 1_680   # avg input tokens per request
OUT_PER_STEP = 190    # avg output tokens per request
FINAL_OUT = 0

def daily_cost(p_in, p_out):
    total_in = STEPS * IN_PER_STEP
    total_out = STEPS * OUT_PER_STEP + FINAL_OUT
    return (total_in * p_in + total_out * p_out) / 1_000_000

for name, p_in, p_out in [
    ("DeepSeek V4 Flash", 0.35, 1.10),
    ("Qwen3-235B-A22B",   1.60, 6.40),
    ("GLM-5-130B",        1.20, 4.80),
    ("GPT-4o",           10.00, 30.00),
]:
    print(f"{name:18} ${daily_cost(p_in, p_out):.2f} / day")

DeepSeek V4 Flash   $7.97 / day
Qwen3-235B-A22B     $39.04 / day
GLM-5-130B          $29.28 / day
GPT-4o              $225.00 / day

Now the same traffic through the router — same token volume, only the paying model changes, so every number below is reproducible from the constants above:

So the routed bill is 89% cheaper than running everything on GPT-4o ($225.00 → $25.13), while keeping GPT-4o quality on the 5% that needs it. Two honest flipsides: it's 3.2x the all-DeepSeek floor — routing isn't about being cheapest, it's the cheapest bill that meets your quality bar — and the 5% routed to GPT-4o is 45% of the total bill. That's the premium you're actually paying for, and it's exactly where you want it.

The honest part: when a router is overkill

• Uniform traffic doesn't need one. If every call is the same shape (a nightly batch job), one model and one price is simpler and the classifier just adds latency. I don't route batch jobs.
• The classifier adds a call and ~0.7s. That's the TTFT of DeepSeek V4 Flash on top of every request. Fine for background pipelines; noticeable for a user staring at a chat cursor. If latency matters more than 70 cents, hardcode the route.
• Prices and models move. The Chinese labs republish prices and silently upgrade models more often than you'd expect. The ROUTES dict is config, not a law — I re-check the mix monthly, and an eval harness catches regressions when a provider swaps a model underneath.
• Validation is the real work. The router is the easy 50 lines. Writing the schema checks and tests that make the fallback trustworthy took me a week. If you can't validate output, routing is just a more complicated way to pick a model — don't bother.

One API key instead of four dashboards

The annoying part wasn't the code — DeepSeek, Qwen, and GLM each have their own console, billing, and rate limits, so a router meant maintaining four accounts. I route everything through tokencnn.com: a single OpenAI-compatible endpoint where deepseek-v4-flash, qwen3-235b-a22b, and glm-5-130b sit behind one API key, so the router above works unchanged and switching a route is a one-line edit. Sign up with just an email — no China phone number, no WeChat — and the $1 free credit covers roughly a week of routed traffic:

curl https://api.tokencnn.com/v1/chat/completions \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v4-flash",
    "messages": [
      {"role": "system", "content": "Classify this ticket."},
      {"role": "user", "content": "Label this request: ..."}
    ]
  }'

Bottom line

The best model isn't one model — it's a policy. Classify each request, route to the cheapest model that's good enough, validate the output, and escalate only when validation fails. On my traffic that's an 89% cut vs all-premium without giving up GPT-4o where it matters, at the cost of ~0.7s of classifier latency and one honest constraint: you have to be able to validate output, or the fallback is just theater.

What does your routing table look like? I'd genuinely like to steal your task→model mapping — especially the task where you learned the hard way that the cheap model wasn't good enough.