the rules
Routing strategies are how an LLM router turns a pool of models into a system that behaves the way you want. OrcaRouter is one platform built around this pattern; the strategies are the rules you configure.
Cheapest first
The default for tolerant work. Extraction, classification, summarising supplied text, formatting — tasks where quality differences barely matter. The rule sends these to the economical model and escalates only when a cheaper model fails or the task needs more. This is where the largest cost saving lives.
Fastest first
The rule for user-facing work. Time to first token decides whether a request feels fast, and the model with the best latency under current load is not always the obvious one. The router picks it, and the product feels faster.
Capability-aware
The rule for hard work. Code generation, multi-step reasoning, agent steps — tasks where the frontier model earns its price. The rule forces these to the strong model regardless of cost, because a wrong answer costs more than a few extra cents.
Failover order
The safety net. When the preferred provider is down or rate-limited, the router moves to the next healthy option automatically. The order is configuration, so you decide who the backup is.
Combining them
Most systems use all four: cheap-first for the volume, fastest-first for the user-facing slice, capability-aware for the hard tail, failover always. The router applies them per request, against live data, and records the outcome so you can tighten the rules over time.
Tuning the rules against data
The rules are a starting point, not a final answer. The router records every call, so after a week of real traffic you can see which assumptions were wrong: the task type you thought needed the strong model clears on the cheap one; the model you included in the pool never gets picked; the failover that fired twice signals a provider worth watching. Each finding becomes a rule change, and each change is a configuration update rather than a release. That is the loop that makes routing improve — the rules start as your best guess and become your measured answer.
Choosing rules without over-thinking them
The common mistake with routing strategies is over-configuring on day one. A new user reads about cheap-first, fastest-first, capability-aware and failover, and builds a twelve-rule matrix before seeing a single real request. The result is a configuration nobody understands and a debugging session every time a request routes unexpectedly. The right approach is the opposite: start with the simplest rules that capture your real constraints, and let real traffic teach you what to add.
A sensible starting set is three rules. Cheapest-first for the task types you know are tolerant — extraction, classification, formatting. Strong-model for the task types you know need capability — code, reasoning, agent steps. Failover to a healthy backup, always. That is enough to capture most of the value, and it is simple enough that every rule is explainable.
Then the router’s per-call record does the tuning. After a week, the data shows the assumptions that were wrong: the task type you thought needed the strong model clears on the cheap one; the model you included never gets picked; the failover that fired twice signals a provider worth watching. Each finding becomes a rule change, and each change is a configuration update rather than a release. The strategies start as your best guess and become your measured answer, and the loop is what makes routing improve rather than ossify.
The simple start that works
The right way to adopt routing rules is to start with three and grow from there. Cheapest-first for the task types you know are tolerant. Strong-model for the task types you know need capability. Failover to a healthy backup, always. That is enough to capture most of the value, and it is simple enough that every rule is explainable to the next engineer who inherits it. The per-call record then does the tuning: after a week, the data shows the task type that clears on a cheap model, the model that never gets picked, the provider that keeps failing over. Each finding becomes a rule change, and each change is a configuration update rather than a release. The strategies start as your best guess and become your measured answer.
See also: Why Regular Financial Checkups Help Businesses Stay Profitable
The rules as a living policy
A useful way to think of routing strategies is as a living policy: they start as your best guess and become your measured answer. The first set of rules encodes what you believe about your traffic — this task type is tolerant, that one needs capability, this provider is reliable. The router then records what actually happens, and the policy converges on reality. The task you thought needed the strong model clears on the cheap one; the model you added never earns a request; the provider you trusted fails over twice in a week. Each finding is a rule change, and the policy keeps improving without a release cycle. That is the difference between a static configuration and a policy that earns its keep.
The takeaway
Routing strategies are the rules that make a model pool useful: cheapest first for tolerant work, fastest first for user-facing work, capability-aware for hard work, and failover as the safety net. They compose per request, live in configuration, and improve with the data the router records. That is how a pool becomes a policy.
