answrsnow
AI Infrastructure · Explainer

How Do You Load-Test an LLM Inference Service in 2027?

By Kory White · Updated Oct 2026 · 9 min read
An NVIDIA H200 GPU server board
Direct answer

Load-test an LLM inference service by driving it with realistic prompts at controlled concurrency and recording six numbers: time to first token (TTFT), inter-token latency (also called time per output token), end-to-end request latency, total output tokens per second, requests per second and the error rate. The first four are defined in NVIDIA's NIM benchmarking guide, and the guide warns that tools differ in how they compute them, so compare results only when definitions align.

The method is a sweep. Start at one concurrent request, raise concurrency in steps until the engine passes its maximum batch size, and plot total throughput against latency at each step. NVIDIA's guide says that if you have a latency budget, the matching point on that curve shows the highest throughput you can get within it, and the point where latency climbs with little throughput gain is where to stop.

Use a purpose-built tool such as AIPerf or vLLM's benchmark CLI for token-level timing, and a general load tool such as Grafana k6 or Locust when you need to test the whole stack in front of the model. Warm the service up first, keep the prompts realistic, and do not repeat identical prompts against a cache. The sections below cover each step and cite the docs behind it. This page quotes no benchmark results, because yours are the only ones that count.

The metrics that matter, defined

Definitions vary between tools, so fix them before you compare anything. These come from the NVIDIA NIM metrics page:

Report percentiles, not just averages. The vLLM benchmark prints mean, median and P99 for TTFT, time per output token and inter-token latency, and your service-level objectives should be written against the tail.

Choose realistic prompts, output lengths and concurrency

NVIDIA's parameters guide says your use case sets the input and output sequence lengths: longer inputs raise prefill memory use and TTFT, and longer outputs raise generation memory use and inter-token latency. It gives typical shapes:

If you have production traffic, the same guide says you can use real prompts as inputs. That beats synthetic text, because your real mix of lengths and shared prefixes drives both batching and cache behavior.

For load control, the guide defines concurrency as the number of concurrent clients, each with one active request, and request rate as the rate at which new requests are sent. It prefers concurrency for most benchmarks, because with a fixed request rate, outstanding requests can grow without bound when arrivals exceed throughput. It advises sweeping from one request to a value slightly above the engine's maximum batch size. Past that point requests queue, so TTFT rises while throughput levels off. Use request rate when you want to mimic arrivals from many independent users; vLLM's benchmark can send them as a Poisson process, with a burstiness setting to make traffic spikier.

For synthetic runs, NVIDIA also recommends setting `ignore_eos` to true so every request generates up to its token limit, and keeping the sampling method the same across a benchmark so runs stay comparable.

The tools: AIPerf, vLLM, LLMPerf, k6 and Locust

As a rule, use an LLM-aware tool when the question is how the model server behaves, and a general tool when the question is how the whole service behaves. NVIDIA suggests running AIPerf on the same host as the server to avoid network latency, unless you intentionally want network effects in the measurement.

The official examples show the shape of a run. NVIDIA's quickstart calls `aiperf profile` with a streaming chat endpoint, a mean input length, an output length, a concurrency and a request count. vLLM's page starts with `vllm serve <model>` and then `vllm bench serve` with a backend, endpoint, dataset and prompt count. Copy the exact flags from those pages, because they change between releases.

A step-by-step test plan: smoke, ramp, soak, spike

Grafana's k6 test-types guide gives a useful vocabulary: smoke tests check that the script works under minimal load, average-load tests check expected normal conditions, stress tests push beyond average, soak tests check reliability over extended periods, spike tests check survival under sudden, short, massive increases, and breakpoint tests raise load gradually to find capacity limits. Mapped to an LLM service:

Interpret the results and set SLOs

NVIDIA's quickstart explains the central chart: TTFT on one axis, total system throughput on the other, one point per concurrency. Two readings follow from it. With a latency budget, take the maximum acceptable TTFT, and the matching point gives the highest throughput you can reach inside it. With a target concurrency, find that point and read off both latency and throughput. The chart also exposes concurrencies where latency grows quickly with little or no throughput gain; stay below them. Similar charts can use inter-token latency, end-to-end latency or per-user tokens per second as the latency axis.

Turn the curve into service-level objectives by writing them against the metrics users feel. TTFT governs how fast a streamed answer starts, inter-token latency governs how smoothly it reads, and end-to-end latency matters for non-streaming and agent calls. State each as a percentile, such as P99, at a named concurrency and prompt shape. Because queueing begins once concurrency exceeds batch capacity, set the autoscaling trigger below that point rather than at it.

Cost per million tokens, as a method

Cost per million tokens is arithmetic on your own measurements, not a number to copy from a vendor. The method:

The result is only as good as the test: a cache-inflated or unwarmed run will understate cost. Recompute whenever the model, hardware, engine version or prompt mix changes.

Common pitfalls

Questions people ask

Which metrics should I report from an LLM load test?

Report time to first token, inter-token latency (time per output token), end-to-end latency, total output tokens per second, requests per second and error rate, with percentiles such as P99 rather than only averages. The first five are defined in NVIDIA's metrics guide and printed by vLLM's benchmark.

Is GenAI-Perf still the tool to use?

No for new work. The GenAI-Perf README says it is being phased out and tells users to use AIPerf for new benchmarking needs.

Should I control concurrency or request rate?

NVIDIA's parameters guide prefers concurrency for most benchmarks, because with request rate the number of outstanding requests can grow without bound when arrivals exceed throughput. Use request rate when you specifically want to model independent arrivals.

Can I load-test an LLM service with k6 or Locust?

Yes, for the stack around the model. NVIDIA notes that k6 can provide TTFT through timing events in the HTTP request, and Locust lets you script tests in Python. For per-token timing and sequence-length control, an LLM-aware tool is easier.

How long should a soak test run?

The k6 guide describes soak tests as assessing reliability and performance over extended periods and gives no single duration. Choose one long enough to cover your real traffic cycle and any slow degradation you want to catch.

How do I calculate cost per million tokens?

Divide the hourly price of the serving hardware by the output tokens per hour sustained at a concurrency that meets your latency objectives, then multiply by one million. Use warmed-up, cache-aware measurements from your own sweep.

Sources

Tools and flags change quickly. This page was written from the linked official documentation on October 18, 2026; check the current pages before you script a test, and trust your own measurements over anyone's published numbers.