What Causes High LLM Inference Latency, and Is FP8 the Best Fix in 2027?

By Kory White · Updated Oct 2026 · 8 min read

Direct answer

High LLM inference latency usually comes from three things: decode is memory-bound, requests wait in a queue before they run, and the KV cache fills GPU memory so the server cannot batch efficiently. Generating each output token means moving the model's weights and cached keys and values through GPU memory, and NVIDIA's inference guide says that transfer speed dominates the latency.

The fixes follow the cause. Smaller or lower-precision weights (FP8 is the one most teams try first) cut the bytes moved per token. Continuous (in-flight) batching and chunked prefill cut queueing. Paged KV cache management frees memory. Speculative decoding and tensor parallelism cut the time per token.

FP8 is not automatically the best fix. vLLM's own documentation reports up to a 1.6x throughput gain and says latency improvements are limited in its online dynamic mode, and it only runs on newer NVIDIA GPUs. Measure your own time to first token and inter-token latency before and after. Every benchmark figure below is quoted with its source, hardware and model, and most are the vendor's own.

First, which latency are you measuring?

"Latency" means different things, and the fix depends on which one is high. NVIDIA's benchmarking guide (April 2, 2025) defines the standard terms.

Time to first token (TTFT)The time it takes to process the prompt and generate the first token. NVIDIA says it generally includes request queuing time, prefill time and network latency.
Inter-token latency (ITL)The average time between the generation of consecutive tokens in a sequence. This is how fast the answer streams.
End-to-end latencyThe time from submitting a query to receiving the full response.

A slow first word points at queueing, a long prompt or a cold start. A slow stream points at decode, memory bandwidth and the KV cache. Streaming does not make the model faster, but Hugging Face's TGI docs show why it helps: at 100 tokens per second, a 1000-token reply takes 10 seconds to finish, but a streaming user sees half of it after five seconds while end-to-end latency stays the same.

Prefill versus decode: why the first token and the rest behave differently

An LLM request has two phases, and NVIDIA's inference optimization guide (November 17, 2023) describes both. In prefill the model processes the input tokens to compute the keys and values in one highly parallel pass, a matrix-matrix operation that effectively saturates GPU utilization. In decode it generates output tokens autoregressively, one at a time, which is matrix-vector work that underuses GPU compute.

That split explains most symptoms. Prefill time grows with prompt length and drives TTFT. Decode time grows with output length and drives ITL and total time. A long system prompt or a retrieved document hurts the first; a long answer hurts the second.

Decode is bound by memory bandwidth, not compute

NVIDIA's guide states it directly: the speed at which weights, keys, values and activations are transferred to the GPU from memory dominates the latency, so decode is a memory-bound operation. Adding raw compute does not fix a memory-bound step.

This is why memory bandwidth is the spec to read on a GPU. NVIDIA lists the H200 with 141 GB of HBM3e memory and 4.8 TB/s of bandwidth. For the same reason, NVIDIA's preliminary LLM claims for the H200 are shown against the H100: 1.9X faster on Llama2 70B (batch size 32 for H200 versus 8 for H100, input length 2K, output length 128) and 1.6X on GPT-3 175B (eight GPUs each, batch 128 versus 64). Those are NVIDIA's own figures, marked preliminary, and the batch sizes differ between the two systems, so they are not a like-for-like latency comparison.

Attention kernels matter here too. The FlashAttention paper (Dao et al., May 27, 2022) makes attention IO-aware, using tiling to reduce reads and writes between GPU high bandwidth memory and on-chip SRAM. Its headline results are training-time (a 15% end-to-end speedup on BERT-large at sequence length 512 against the MLPerf 1.1 record, and 3x on GPT-2 at sequence length 1K), so read them as evidence that memory traffic is the bottleneck rather than as an inference latency number.

The KV cache: memory that decides how much you can batch

To avoid recomputing keys and values for every earlier token, servers cache them in GPU memory. NVIDIA's guide notes the cache grows linearly with batch size and sequence length, and gives a sense of scale: a 7B-parameter model in FP16 or BF16 needs roughly 14 GB for weights, and Llama 2 7B at batch size 1 with a 4,096-token sequence has a KV cache of about 2 GB.

When the cache runs out, latency spikes. vLLM's tuning docs say that if KV cache space is insufficient, requests can be preempted and later recomputed, which can hurt end-to-end latency. Its suggested fixes are to raise gpu_memory_utilization for more cache memory, or lower max_num_seqs or max_num_batched_tokens.

The vLLM paper (Kwon et al., September 12, 2023) targets the waste: KV cache memory can be significantly wasted by fragmentation and redundant duplication, and its PagedAttention approach manages it in blocks. The authors report 2-4x higher throughput than FasterTransformer and Orca at the same level of latency. That is a throughput result from the paper's own benchmark setup on 2023-era systems, so expect different numbers on current engines.

Batching and queueing: the latency you cannot see in the model

Batching raises GPU utilization and throughput, but done naively it adds waiting. NVIDIA's guide explains that with static batching all requests in the batch must wait until the longest one finishes. In-flight (continuous) batching fixes this: the runtime immediately evicts finished sequences and admits new ones.

NVIDIA's own TensorRT-LLM announcement (September 9, 2023) says in-flight batching and kernel optimizations minimally double throughput on a benchmark of real-world requests on H100 GPUs. That is a vendor throughput claim, not a measured TTFT.

Prefill can also stall decode. vLLM's docs describe chunked prefill, where the scheduler prioritizes decode requests, improving inter-token latency. The knob is max_num_batched_tokens: smaller values favor ITL and higher values favor TTFT. If your TTFT is high at peak traffic, check queue depth and concurrency limits before changing the model. TGI, for example, returns an overloaded error when too many requests arrive, with --max_concurrent_requests setting the limit.

Quantization and FP8: smaller numbers, fewer bytes per token

Quantization reduces the precision of weights and activations. NVIDIA's guide notes that most deep learning models can be represented with eight or even fewer bits per value. Since decode is memory-bound, fewer bytes per weight means less data to move per token.

Hardware. vLLM's FP8 documentation says FP8 computation is supported on NVIDIA GPUs with compute capability 8.9 or higher (Ada Lovelace, Hopper, Blackwell). Turing and Ampere get only weight-only W8A16 support.
Claimed gains. The same page says FP8 allows a 2x reduction in model memory requirements and up to a 1.6x improvement in throughput. It gives no explicit latency number, and says that in online dynamic quantization mode latency improvements are limited.
NVIDIA's numbers. On H100 with TensorRT-LLM, NVIDIA reports its FP8 results as 4x faster than A100 on GPT-J-6B (8x with TensorRT-LLM and in-flight batching) and 4.6x faster on Llama 2 70B, both on CNN/DailyMail summarization. These compare different GPU generations and are vendor throughput figures, so they do not isolate what FP8 alone does for latency.
Accuracy. NVIDIA says FP8 retains higher accuracy than INT8 or INT4 but gives no supporting numbers in that post. Run your own evaluation on your own prompts before shipping.

So is FP8 the best fix? It is often the lowest-effort one on supported hardware, because it needs no model code changes and frees memory for a larger KV cache. But it will not help a queueing problem, a cold start or a prompt that is far too long. Fix the cause you measured.

Speculative decoding: fewer slow steps per answer

Speculative decoding uses a cheaper process to draft a continuation, then verifies it with the main model in parallel, as NVIDIA's guide describes. The original paper by Leviathan, Kalman and Matias (submitted November 30, 2022) shows a 2X-3X acceleration on T5-XXL against the standard T5X implementation, with identical outputs.

In practice it suits specific workloads. vLLM's docs say it is for reducing inter-token latency under medium-to-low queries per second, in memory-bound workloads, and that real gains depend on model family, traffic pattern, hardware and sampling settings. Model-based methods such as EAGLE, MTP and draft models give the best latency reduction, while n-gram and suffix decoding give modest speedups. At high load the extra draft work can compete with batching, so test at your real traffic level.

Parallelism, caching and cold starts

Tensor parallelism shards individual layers into smaller independent blocks that run on different devices, as NVIDIA's guide describes. Splitting a model across GPUs lets each read fewer weights per step, but adds communication between GPUs, so it helps most when a model does not fit on one device or decode is bandwidth-starved.

Prefix caching attacks TTFT for repeated prompts. In Hugging Face's TGI v3 benchmark, 20 long-prompt requests on Llama 3.1 70B with 8xH100 took 2s on TGI versus 27.5s on vLLM with prefix caching on the repeat run. On the first run, when nothing is cached, it was 12.1s versus 28.3s. This is the TGI team's own benchmark, and they note that behind multiple replicas the cache is absent unless a user's requests stick to the same replica.

Network and cold starts do not show up in model benchmarks at all, but NVIDIA notes TTFT includes network latency. If the first request after idle is slow, check whether the model is loaded and warm, and measure from the client, not only inside the server.

A practical order of operations

Measure first. Record TTFT, ITL and end-to-end latency separately, at your real concurrency, using a load test.
If TTFT is high: check queueing and concurrency, prompt length, prefix caching and network or cold-start effects.
If ITL is high: check KV cache pressure and preemption, chunked prefill settings, then lower precision such as FP8 on supported GPUs, then speculative decoding if traffic is light to medium.
If the model does not fit well: use a smaller model, quantization or tensor parallelism.
Re-measure after each change. Vendor figures here come from specific models, hardware and batch sizes that may not match yours.

For how to run that measurement, see our guide to load-testing an LLM inference service.

Questions people ask

Why is LLM inference slow?

Generating each token means moving the model's weights and the KV cache through GPU memory, and NVIDIA's inference guide says that transfer speed dominates decode latency. Queueing and long prompts add to time to first token. Source: NVIDIA.

What is the difference between time to first token and inter-token latency?

Time to first token is the time to process the prompt and generate the first token, and generally includes queuing, prefill and network latency. Inter-token latency is the average time between consecutive tokens. Source: NVIDIA.

Does FP8 reduce latency?

It can, by cutting memory per weight, and vLLM's docs claim a 2x reduction in model memory and up to 1.6x throughput. The same page gives no explicit latency number and says latency gains are limited in online dynamic mode. It needs a GPU with compute capability 8.9 or higher. Source: vLLM docs.

Is speculative decoding worth it?

For light to medium traffic on memory-bound workloads, vLLM says it reduces inter-token latency, though real gains depend on model, traffic, hardware and sampling settings. The original paper reported 2X-3X on T5-XXL. Sources: vLLM, Leviathan et al..

What does continuous batching do for latency?

It evicts finished sequences immediately instead of making every request wait for the longest one in the batch, which reduces waiting and raises GPU utilization. Source: NVIDIA.

Sources

NVIDIA: Mastering LLM Techniques, Inference Optimization · Prefill, decode, KV cache, batching, quantization, parallelism
NVIDIA: LLM Inference Benchmarking, Fundamental Concepts · TTFT, ITL and end-to-end latency definitions
NVIDIA: TensorRT-LLM on H100 · FP8 and in-flight batching vendor figures
NVIDIA H200 product page · Memory capacity, bandwidth, preliminary inference claims
vLLM paper (Kwon et al.), arXiv 2309.06180 · PagedAttention and KV cache memory waste
Leviathan et al., speculative decoding, arXiv 2211.17192 · 2X-3X on T5-XXL
Dao et al., FlashAttention, arXiv 2205.14135 · IO-aware attention
vLLM docs: Optimization and Tuning · Chunked prefill, preemption
vLLM docs: Speculative Decoding · When it helps and caveats
vLLM docs: FP8 W8A8 · Hardware support and claimed gains
Hugging Face TGI: v3 overview · Prefix caching benchmark and caveats
Hugging Face TGI: Streaming · Streaming and perceived latency

Figures were read from the linked pages on October 18, 2026. Benchmarks are the vendors' or authors' own, on the hardware, models and dates stated, and will not match your workload. Engine defaults and supported features change quickly, so check current documentation before you tune.