answrsnow
AI Infrastructure · Essay

How Do You Route Requests Across Multiple LLM Providers in 2027?

By Kory White · Updated Nov 2026 · 9 min read
Azure API Management analytics screen showing language model token metrics and model usage, from Microsoft Learn documentation
Direct answer

Put one gateway between your application and the providers, then layer rules on it: retries and cooldowns first, fallbacks second, and cost, latency or task-based routing only once you can measure what each route is doing. The documented gateways all work this way. LiteLLM lists simple-shuffle, latency-based, least-busy and cost-based strategies, Kong lists seven balancing algorithms, and Cloudflare lets you branch on request headers, split traffic by percentage and fall back when a rate or budget limit is hit.

The evidence for routing by task is a research result, not a vendor promise: the RouteLLM paper (June 26, 2024) reports that routers trained on preference data cut costs "by over 2 times in certain cases" without hurting response quality. The hard parts are differences the gateway has to translate: rate-limit accounting, prompt caching, tool-call formats and where data is processed. No source read here publishes gateway latency overhead, so this page gives none.

Start with a gateway, not provider code

A gateway is a proxy that exposes one API to your application and holds the provider credentials, retry rules and logging in one place. Azure API Management's AI gateway describes the job as authenticating access, load balancing across AI endpoints, monitoring interactions and managing token quotas across applications. Portkey lists fallbacks, load balancing across multiple API keys "to counter rate limits," automatic retries, conditional routing and request timeouts. Cloudflare AI Gateway lists analytics, logging, caching, rate limiting and "Request retry and fallback," and says it is available on all plans.

Compatibility is the entry point. LiteLLM says it "accepts and translates the OpenAI Chat Completion params across all providers," and Azure's gateway can expose multiple backends through a single OpenAI-compatible endpoint in its unified model API, which is in preview. These are vendor descriptions of their own products; the pages read do not compare them head to head, so this page does not rank them.

Retries, cooldowns and fallbacks

Treat these as three different tools. Retries handle transient errors against the same target. LiteLLM's routing docs say that for rate-limit errors it uses exponential backoff, while "For generic errors, we retry immediately." Cooldowns take a failing deployment out of rotation: the default cooldown is 5 seconds, triggered immediately by a 429, by more than 50% failures in the current minute, or by non-retryable errors such as 401, 404 and 408.

Fallbacks send the request somewhere else. LiteLLM's reliability docs describe three kinds: content-policy fallbacks, context-window fallbacks and general fallbacks for the remaining errors, such as rate limits. Kong notes that "Client errors don't trigger failover" and that you opt in to other triggers such as HTTP 429 or 502 through failover criteria; from version 3.10 its fallback works across targets with any supported format, so you can mix providers. Azure's circuit breaker applies the Retry-After value a backend returns when it decides how long to stop sending traffic.

AWS is explicit that the platform does not do everything: on its cross-Region inference page, it advises that to meet an availability target you should add bounded retries and multi-Region or multi-model fallback in your application. Bound the retries; the sources read give no safe default count, so pick one and test it.

Load balancing across keys, regions and deployments

Spreading load solves rate limits that a single key cannot. LiteLLM sets rpm and tpm per deployment, labels simple-shuffle "Default and Recommended for Production," and has an order setting where lower values take priority before fallbacks run. It warns that usage-based routing is "not recommended for production due to performance impacts." Azure lists round-robin, weighted, priority-based and session-aware balancing, with priorities useful for provisioned-throughput endpoints. Kong adds consistent hashing for sticky sessions, least-connections, lowest-latency, lowest-usage and a semantic algorithm.

On a managed platform, the region choice is part of the product. Amazon Bedrock offers geographic inference profiles that process within boundaries such as US, EU and APAC and global profiles that route to any supported commercial Region, which AWS says gives "Approximately 10% savings." AWS says there is no additional routing cost and that CloudTrail records where each request was processed in the inferenceRegion field.

Routing by cost, latency or provider health

Marketplaces route for you. OpenRouter's default sends each request to one provider chosen at random, weighted so that "Providers close to the best share most of the traffic," and gives no traffic to a provider whose errors or 429s are far worse than the model's other providers. Setting sort to price, throughput or latency bypasses that default, and allow_fallbacks defaults to true. If you run the gateway yourself, LiteLLM offers latency-based routing, which picks the deployment with the lowest response time, least-busy and cost-based routing that picks the cheapest healthy deployment within its rpm and tpm limits.

Cheapest-first routing only works if every route returns acceptable output. None of the sources read publishes a quality comparison between gateway routing strategies, so test your own prompts before enabling cost routing.

Routing by task: semantic and model-choice routers

Task routing sends easy requests to a cheaper model and hard ones to a stronger one. The RouteLLM paper by Ong, Almahairi, Wu, Chiang, Wu, Gonzalez, Kadous and Stoica, first submitted June 26, 2024 and revised February 23, 2025, trains routers on human preference data to choose between a stronger and a weaker model at inference time. The abstract reports cost reductions of "over 2 times in certain cases" without compromising response quality; the paper's own benchmarks, not yours, produced that number.

Product versions exist inside gateways. Cloudflare's dynamic routing builds flows from conditional nodes that test the request body, headers or metadata, percentage nodes for A/B tests and gradual rollouts, and rate-limit and budget-limit nodes that switch to a fallback. Kong lists a semantic algorithm that routes "based on prompt-to-model similarity." Commercial router services also exist; none was read for this page, so no claims about them are made.

Rate limits do not mean the same thing everywhere

Providers meter differently, and a gateway that counts only requests will misjudge capacity. Anthropic's documentation measures requests per minute, input tokens per minute and output tokens per minute, uses a token bucket algorithm, and says that for most models only uncached input tokens count toward the input limit. Its example: with a 2,000,000 input-token limit and an 80 percent cache hit rate you could process 10,000,000 total input tokens per minute. A 429 may carry a retry-after header, and the spend-cap 429 does not, so a retry loop must tell them apart.

At the gateway layer, Azure's token limit policy sets tokens-per-minute or a quota over a period per consumer key, with an example of 500 tokens per minute per subscription key. LiteLLM returns an immediate 429 when a request exceeds a parallel-request cap, with "no wait queue." Decide who absorbs the burst: the gateway, your application, or a queue.

Prompt caching changes the economics of routing

Switching providers can throw away a warm cache. Anthropic's docs (vendor documentation) say the default cache lifetime is 5 minutes with a 1-hour option, that a 5-minute cache write costs 1.25x the base input price and a 1-hour write 2x, and that reads cost 0.1x with exceptions for some models. Minimum cacheable length varies by model from 512 to 4,096 tokens. OpenAI's guide (vendor documentation) says caching is on by default, gives a 1,024-token minimum for its newest models, a "30m" default lifetime with a "12h" option, and a prompt_cache_key that influences routing; for earlier models it says there is no extra write charge.

The practical rule, our reading of these pages: cache terms differ in lifetime, write cost and activation, so a router that alternates one conversation between providers pays write costs repeatedly. Prefer sticky routing for long shared prefixes. Kong's consistent hashing and Azure's session-aware balancing exist for that kind of stickiness, though neither page ties them to caching.

Streaming and tool-calling differences

Tool definitions are not interchangeable on the wire. Anthropic defines tools with an input_schema, signals a call with stop_reason "tool_use" and tool_use blocks, and expects a tool_result block back keyed by tool_use_id. OpenAI defines functions with type, name, description, parameters and strict, returns function_call items with a call_id and arguments as a JSON-encoded string, and takes results as function_call_output items. A gateway that exposes one format has to translate these, as Azure's unified model API says it does for format translation.

Translation has limits. LiteLLM says support is model dependent within a provider, for example that Bedrock Llama models do not list tools or tool_choice, and by default raises an exception for unsupported parameters instead of dropping them; drop_params changes that. For streaming, LiteLLM's reliability docs note that falling back from a stream that broke mid-stream strips encrypted reasoning the target cannot decrypt. The pages read say little else about mid-stream failover, so test partial-output handling yourself.

Data residency, privacy and observability

Routing is also a compliance decision. Bedrock's geographic profiles keep processing within the stated boundaries, while global profiles can use any supported commercial Region; AWS says data moves over the AWS network, encrypted in transit. OpenRouter defaults data_collection to "allow," which permits providers that store data non-transiently and may train on it, and offers "deny" and a zdr setting that routes only to endpoints with a Zero Data Retention policy. A fallback that crosses a provider boundary can cross a policy boundary, so restrict fallback lists to approved providers.

Observability should come from the gateway because it sees every route. Azure logs prompts and completions to Azure Monitor and emits token metrics per consumer to Application Insights; the screenshot at the top of this page is its analytics view for language model APIs. Cloudflare lists analytics of "the number of requests, tokens, and the cost," and logging of requests and errors. Log which provider actually answered each request; without that, fallbacks hide outages from your dashboards.

A practical sequence (our reading of the sources)

This sequence is analysis built on the cited documentation, not a sourced benchmark. No source read publishes gateway latency overhead, failure-rate statistics across providers or typical savings from production routing.

Questions people ask

What is an LLM gateway?

A proxy between your application and model providers. Azure describes its AI gateway as handling authentication, load balancing across AI endpoints, monitoring and token quota management; LiteLLM, Portkey, Cloudflare and Kong document similar features.

What is the difference between a retry and a fallback?

A retry repeats the request against the same target, such as LiteLLM's exponential backoff for rate-limit errors. A fallback sends it to a different deployment or model, such as LiteLLM's context-window and content-policy fallbacks.

Can routing really cut LLM costs?

The RouteLLM paper, submitted June 26, 2024, reports cost reductions of over 2 times in certain cases without compromising response quality, using routers trained on preference data. That is a research result on its benchmarks, not a guarantee for your workload.

Does switching providers affect prompt caching?

Yes, cache terms differ by provider. Anthropic documents a 5-minute default lifetime with a 1-hour option; OpenAI documents a 30m default with a 12h option for its newest models. A cache built with one provider does not carry over to another.

How do I keep data in one region when routing?

On Amazon Bedrock, use a geographic inference profile, which processes within boundaries such as US, EU or APAC. On OpenRouter, the zdr setting limits routing to Zero Data Retention endpoints.

Sources

Facts were read from the linked pages on November 2, 2026. Vendor claims are labeled as vendor-stated and nothing is estimated; where a source did not publish a number, the page says so. The page image is a screenshot from Microsoft Learn documentation. If a page here is out of date, tell us and we will correct it from a source.

Share this page
LinkedIn: share linkLinkedIn: share post