Production Model Routing: Cut Cost Without Hiding Regressions

Model routing chooses the best model for each request instead of sending every request to one default. In the common cost-saving design, easy work goes to a smaller, faster model and difficult work goes to a stronger one. The promise is lower cost and latency without lower quality.

The dangerous part is the last phrase. Routing failures concentrate on the hard tail: the requests that most needed the strong model. Aggregate metrics can look healthy because the easy majority still succeeds. A production router must therefore be designed as a quality-control system, not merely a cost switch.

First gate capabilities, then optimize

Before estimating difficulty, remove models that cannot satisfy hard requirements:

• required tool or function-calling support
• text, image, audio, or other modality
• context-window and output-length needs
• structured-output or schema guarantees
• latency ceiling and streaming behavior
• data residency, retention, and provider policy
• approved regions, tenants, or compliance boundaries

Capability routing is deterministic. A vision request never belongs on a text-only model, however cheap it is. A regulated tenant may have only one eligible provider. Applying these gates first prevents the cost router from making impossible choices.

Confirm the traffic has an easy segment

Routing adds code, latency, observability, failure modes, and operational ownership. It pays only when a meaningful share of traffic can move to a cheaper tier.

Sample real requests and label them by:

• task or endpoint
• input and expected-output size
• required capabilities
• difficulty or number of reasoning steps
• risk if the answer is wrong
• quality on each candidate model
• cost and latency on each candidate model

If nearly every request needs the strongest tier, do not build a router. Optimize caching, prompts, batching, or provider pricing instead. Architecture is not a substitute for a favorable traffic distribution.

Use the cheapest signal that works

Routing signals form a cost ladder:

1. Explicit task route. Extraction, classification, summarization, coding, and support may already enter through different endpoints.
2. Deterministic heuristic. Input length, language, file type, tool requirement, tenant tier, or structured-output flag.
3. Small classifier. A fast model trained or prompted to predict difficulty or route class.
4. LLM router. A general model reads the request and selects a tier.

Every router decision happens before the useful model call, so routing overhead lands on every request. An additional LLM call can erase the latency and token savings it was supposed to create. Start with explicit product knowledge and heuristics; move upward only when offline evaluation proves the simpler signal misses too many valuable cases.

Write thresholds as configuration, not scattered conditionals:

routes:
  extract:
    default: fast
    escalate_if: schema_invalid
  support:
    default: fast
    strong_if:
      - input_tokens > 4000
      - account_risk == high
      - requires_tools == true

Thresholds need an owner, version, and reason. Otherwise they become folklore no one can tune safely.

Cascades protect the hard tail

A model cascade calls a cheaper tier first and escalates when its output fails a validation check. Validation can include:

• JSON-schema or type validation
• required fields or citations
• deterministic business rules
• refusal, uncertainty, or incomplete-answer detection
• a lightweight task-specific verifier
• a calibrated judge for genuinely semantic criteria

The cascade turns cheap-model failures into added latency rather than silent quality loss. It also makes the economics visible:

expected cost = router overhead
              + cheap model cost
              + escalation rate × strong model cost

The same reasoning applies to latency. Track both one-hop and escalated requests so a healthy median does not hide a poor tail.

High-stakes requests should not depend on a noisy difficulty estimate. Route irreversible actions, safety decisions, sensitive legal or financial output, and other expensive-to-be-wrong paths directly to the approved strong tier with human or deterministic controls as appropriate.

Distinguish escalation from provider fallback

An escalation changes model strength because the request or output is difficult. A fallback changes model or provider because the primary is unavailable, rate-limited, or too slow. They solve different failures and should be observable separately.

A robust route might be:

fast primary ──invalid/uncertain──> strong primary
     │                                  │
     └──outage/rate limit──> fast backup└──outage──> strong backup

Keep response validation after every branch. Provider compatibility does not guarantee equivalent formatting, safety, or quality.

Evaluate by route and slice

Build a frozen eval set containing every task type and difficulty band. For each candidate route, record:

• task quality or accuracy
• schema-valid and refusal rates
• cost per successful request
• p50 and p95 latency
• escalation and fallback rates
• router confusion: easy sent strong, hard sent cheap

Report the cheap route, strong route, escalated requests, and blended total separately. Slice by language, input length, tenant, modality, and risk class. The aggregate is an outcome; the slices explain whether it is safe.

An “easy” false negative wastes money by sending work to the strong tier. A “hard” false positive silently degrades quality by sending it cheap. Price those errors differently in the routing objective.

Shadow before controlling traffic

In shadow mode, the proposed router makes decisions and candidate models run on sampled traffic, but the existing production route still serves the user. Compare the hypothetical outcome against the known-good baseline.

Then canary the router on a small traffic share, starting with low-risk task classes. Increase gradually while watching per-route quality, cost, latency, error, escalation, and fallback dashboards. Keep an immediate configuration switch that returns all traffic to the known-good model.

Do not tune a threshold directly on the same cases used to report its success. Keep a held-out evaluation split and confirm production behavior on fresh traffic.

Operate the router as a product

Models, prices, traffic, and provider reliability change. Log the route decision, signal values, selected model, validation result, escalation reason, fallback reason, cost, and latency for each request. Sample live outputs for online evaluation and promote routing misses into the offline dataset.

Set budgets and alerts by route. A spike in escalation rate can mean the cheap model changed, traffic became harder, or validation tightened. A drop may be an optimization—or a broken verifier that stopped catching failures.

Use the Model Router Designer to turn a traffic sample into a routing specification, and the Provider Fallback Wrapper to add availability fallbacks without confusing them with quality escalation.