How Do You A/B Test Different LLMs in Production in 2027?

Offline first, then online

Offline evaluation runs a pre-defined dataset through your application before release. Langfuse describes it as evaluating "your existing application on a pre-defined dataset" to confirm changes are ready to ship, while online evaluation scores "live incoming traces to measure quality on production data and track trends over time." You need both, because the offline set is a guess about real traffic and the online test is the check on that guess.

OpenAI's evals guide calls evals "an essential component to building reliable applications," especially "when upgrading or trying new models." Anthropic's guide to building tests says to design evals that mirror your real-world task distribution and to prioritize volume over quality in grading. A challenger that loses the offline run does not earn live traffic; one that wins only earns a small share.

Shadow, canary and split traffic

There are three ways to put a challenger in front of real requests, in rising order of risk:

• Shadow. The candidate runs on the same inputs but users never see it. Statsig's online evals describe how to "shadow-run 'candidate' versions without exposing users to them," with an LLM judging the outputs of both versions. You learn about quality, latency and cost with no user risk, but not about user behavior.
• Canary. A small share of live requests goes to the new model. Portkey's canary testing guide shows a Llama 2 test on 5% of traffic, with weights of 0.95 for the existing production target and 0.05 for the challenger.
• Split. A larger share goes to each model so that both arms collect enough data to compare. agentgateway's traffic split docs give an example that routes 80% to a cheaper model and 20% to a more capable one, and list A/B testing as a use.

A common order is shadow, then canary, then split. This is a pattern, not a standard, and no page read here prescribes the percentages. Choose a small first step you can afford to be wrong about.

Gateways and routers that can split traffic

A gateway sits between your application and the model providers, so the split needs little or no change in your application code. Portkey says its gateway "handles all the request transforms to make sure you don't have to change your code."

• Portkey. Each target gets a weight, and Portkey states that "Weights represent relative share of traffic" and normalizes them to sum to 100%. Its example weights of 5, 3 and 1 become 55%, 33% and 11%. A weight of 0 stops traffic to a target without removing it, which makes rollback a config change. It also documents sticky load balancing for "Ensuring consistent model behavior for A/B testing," but describes no dedicated A/B testing feature.
• agentgateway. Separate backends for each model and an HTTPRoute with weighted backend references distribute traffic, per its traffic split page.
• LiteLLM. The LiteLLM router spreads requests across deployments that share a model name and lists strategies such as simple-shuffle, least-busy, latency-based and cost-based routing. That page does not describe weighted or percentage-based splitting, so confirm in LiteLLM's current docs how to run a fixed-share test before relying on it.

Cloud providers also publish gateway and routing features; check your provider's own documentation, since none is quoted here.

Feature-flag and experimentation platforms

A flag platform can assign users to arms, keep each user on one arm and tie the result to a business metric.

• LaunchDarkly AI Configs. LaunchDarkly says experimentation lets you measure "how config variations affect end-user behavior using the metrics you define." A config can hold up to fifty variations, each pairing a model configuration with messages. Its Monitoring tab shows token usage and LLM call duration but does not measure end-user behavior, and you cannot run a guarded rollout and an experiment on the same flag at once.
• Statsig. Its online evals grade outputs with a score between 0 and 1.
• Langfuse. Langfuse covers datasets, experiments that compare changes, and LLM-as-a-judge scoring of live traces. It is an evaluation layer, not a traffic splitter.

A flag gives you per-user stickiness and metric tracking; a gateway gives you provider-level routing and failover. Many teams may use the two together, though no source read here says so.

What to measure

Compare the same four things on both arms, using the same traces:

• Task success. Whatever your product defines as a good outcome: a resolved ticket, a correct extraction, an accepted suggestion. Score it with code checks, an LLM judge or human review.
• User feedback. Thumbs, edits, retries and abandonment. LaunchDarkly's examples of signals are page views and clicks.
• Latency at p50 and p95. p50 is the median request and p95 is the 95th percentile, so it captures the slow tail. Anthropic's models overview only gives relative latency labels and says "Actual latency depends on prompt length, output length, and thinking effort," so measure your own.
• Cost per request. Token count times the price, plus retries. Multiply by your own average input and output tokens rather than comparing list prices alone.

Provider list prices, read on October 20, 2026 from Anthropic's models overview, per million tokens:

• Claude Fable 5.1$10 input, $50 output
• Claude Opus 5.5$4 input, $20 output
• Claude Sonnet 5.5$2 input, $10 output
• Claude Haiku 5.5from $0.10 input, from $0.50 output

The same page says Batch API requests are 50% off. Other providers' prices are not listed here because their pages were not readable for this guide; read them directly on the day you run the test. For cost levers beyond model choice, see how to reduce GPU costs when serving large language models and what causes high LLM inference latency.

Pin model versions in the test. Anthropic notes that "Every Claude model ID is a pinned snapshot," so an arm labeled with a specific ID stays comparable over the test. Check retirement dates too: the same page lists Sonnet 5.5 as not retiring sooner than September 28, 2027.

Statistics: sample size and when to stop

Decide the metric, the smallest difference worth detecting and the sample size before launch. Evan Miller's sample size calculator defines the minimum detectable effect as "the smallest effect that will be detected (1-β)% of the time." Its worked example, a baseline conversion rate of 10.2% and a detectable range up to 13.2%, needs 2,545 per variation. That is a conversion-rate example, not an LLM benchmark; your baseline and effect size change the answer.

Do not stop when the dashboard first turns significant. Miller's How Not To Run an A/B Test says to "Decide on a sample size in advance and wait until the experiment is over," because stopping as soon as there is a significant result makes reported significance levels unreliable. Keep users sticky to one arm so the same person does not see both.

Pitfalls

• Non-determinism. The same prompt can return different outputs, which is why one good answer proves little. OpenAI's guide says to run an eval against a dataset and review results across many inputs. Use enough samples that noise averages out.
• Prompt changes confounding the test. If the challenger also gets a new prompt, you cannot tell which change moved the metric. Change one thing per experiment, or test model and prompt as a named bundle.
• Judge bias. Zheng et al. examined "position, verbosity, and self-enhancement biases" in LLM judges, and found strong judges reached "over 80% agreement" with human preferences. Anthropic's guide suggests using a different model to grade than the one that produced the output. Spot-check judge scores against human review.
• Metric mismatch. Offline scores may not predict user outcomes. Keep a small human-reviewed sample from the live arm.
• Mixed signals. A cheaper model can win on cost and lose on task success. Set the primary metric and the guardrails (latency and cost ceilings) up front.

Ramp or roll back

Write the rule before the test: for example, ramp if the challenger meets the success metric without breaching your latency and cost ceilings, and roll back if it breaches a ceiling. The thresholds are yours; none is published as a standard. With weighted routing the rollback is a config change, and Portkey notes a weight of 0 stops traffic to a target without removing it from the config.

Afterward keep monitoring: models change and traffic shifts. See the live LLM monitoring guide and, if repeated prompts are common, what a semantic cache can cut.

Questions people ask

Sources

• Langfuse: Evaluation overview · Offline vs online evals, experiments, LLM-as-a-judge
• OpenAI: Evals guide · Why evals matter when trying new models
• Anthropic: Develop tests · Eval design and grading
• Portkey: Load balancing · Weighted traffic split, sticky routing
• Portkey: Canary testing · Small-share model test
• LiteLLM: Router load balancing · Routing strategies
• agentgateway: Traffic split · Weighted split between models
• LaunchDarkly: AI Configs experimentation · Experiments on model variations
• Statsig: Online evals · Shadow-run candidates, LLM judge
• Anthropic: Models overview · Prices, pinned IDs, retirement dates
• Evan Miller: Sample size calculator · Minimum detectable effect, example sample size
• Evan Miller: How Not To Run an A/B Test · Fix the sample size in advance
• Zheng et al.: Judging LLM-as-a-Judge · Judge biases and human agreement

Facts were read from the linked pages on October 20, 2026. Prices and product features change; confirm them on the vendors' pages. Thresholds, traffic shares and sample sizes here are examples or your own to set, not standards.

Top photo: Carl Lender / CC BY 2.0 via Wikimedia Commons