What Is GPU Memory Fragmentation and How Do You Avoid It in 2027?

By Kory White · Updated Oct 2026 · 8 min read

Direct answer

GPU memory fragmentation is when free GPU memory is split into many small, non-contiguous chunks, so a large allocation fails with an out-of-memory error even though the total free memory looks big enough. The memory exists, but not in one piece.

You avoid it by combining a few habits: configure the caching allocator well (in PyTorch, expandable_segments, plus max_split_size_mb and garbage_collection_threshold as tuning options), keep tensor shapes consistent or padded so freed blocks get reused, preallocate memory where your framework supports it, use a paged KV cache such as vLLM PagedAttention for LLM serving, and restart long-running processes that have drifted. Measure first: a memory snapshot shows whether fragmentation is really your problem.

Most of the specific figures below come from the PyTorch and JAX documentation and the vLLM paper. Where a number is not published, this page says so.

What fragmentation means on a GPU

A GPU allocator hands out blocks of device memory and takes them back as tensors are freed. If many differently sized blocks are allocated and freed in an irregular order, the free space ends up as holes between live blocks. Total free memory can be large while the largest single hole is small, and an allocation that needs one contiguous block fails.

Two kinds are worth separating. External fragmentation is the holes between blocks described above. Internal fragmentation is wasted space inside a block that was reserved bigger than what is actually used, which is the main problem in LLM key-value (KV) caches, covered below.

Fragmentation is only one reason for an out-of-memory error. Memory held by live tensors, memory outside the framework allocator and plain over-sized batches all produce the same message, so diagnose before you tune.

Rows of server racks in a data center — Photo: Carl Lender / CC BY 2.0 via Wikimedia Commons

How the PyTorch caching allocator works

The PyTorch CUDA memory management notes say it uses a caching memory allocator to speed up allocations. Freed memory is kept and reused without device synchronization instead of going back to the driver. One side effect: the unused memory the allocator is holding still shows as used in nvidia-smi, so that tool alone cannot tell you whether you are short of memory or just holding a cache.

The docs separate two counters. memory_allocated() tracks memory used by tensors, while memory_reserved() tracks the total the allocator is holding. A large gap between them means a lot of cached, not-in-use memory, which is where fragmentation hides. empty_cache() hands unused cached memory back, but it does not free memory held by live tensors.

PyTorch allocator settings that matter

Allocator behavior is set with an environment variable. The docs give the form PYTORCH_ALLOC_CONF=option:value,... and keep PYTORCH_CUDA_ALLOC_CONF as a backward-compatible alias. The options most relevant to fragmentation:

expandable_segments. Experimental, default False. When True, the allocator grows one segment per stream instead of creating a new segment per allocation, which the docs say reduces unusable slivers when batch sizes change. This is the first option to try.
max_split_size_mb. Stops the native allocator from splitting cached blocks larger than this size. The default is unlimited. The docs say it should be used as a last resort, and that its performance cost ranges from zero to substantial. No recommended value is published, so you have to find one by testing.
garbage_collection_threshold. Starts reclaiming unused cached memory once usage passes a fraction of total GPU memory. The default is 1.0 and the docs example is 0.8; the value must be above 0.0 and below 1.0.
roundup_power2_divisions. Rounds allocation sizes to power-of-2 divisions so freed blocks are more likely to fit later requests. It accepts one value for all sizes or ranges such as [256:1,512:2,1024:4,>:8].
backend. native is the default; cudaMallocAsync is the alternative and requires CUDA 11.4 or newer.

Example, as an environment variable set before the process starts: PYTORCH_ALLOC_CONF=expandable_segments:True. Change one setting at a time and compare peak reserved memory and speed before and after, because the docs do not promise a result for any workload.

How to diagnose it

Start with the numbers. The docs describe memory_stats() for detailed data, memory_summary() as a readable summary that helps with tuning, and memory_snapshot() for the full allocator state. If reserved memory is far above allocated memory when an allocation fails, fragmentation or an oversized cache is likely.

For the real picture, use the PyTorch memory snapshot tools. Call torch.cuda.memory._record_memory_history() before the code you want to watch, then torch.cuda.memory._dump_snapshot("my_snapshot.pickle"), and drag the file onto pytorch.org/memory_viz. The docs say the viewer does not upload the data. It shows an active memory timeline and an allocator state history in which each segment is displayed split into allocated and free blocks, and out-of-memory errors appear as OOM events. The docs note that checking the state at an OOM can explain why an allocation failed even though reserved memory still exists, which is exactly the fragmentation case.

For long runs, set max_entries when recording. It defaults to sys.maxsize, each entry takes several KB, and the docs say snapshot files can reach the GB range. Memory allocated outside the PyTorch allocator, such as by NCCL, is not visible in the profiler; compare torch.cuda.device_memory_used() with the PyTorch numbers to spot it.

Fragmentation in LLM KV caches and how PagedAttention fixes it

In LLM serving the biggest consumer of fragmented memory is the KV cache. The vLLM paper, Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., submitted September 12, 2023, SOSP 2023), says each request KV cache is large and changes size during generation, and that poor memory management wastes much of it through fragmentation and duplication. Systems that reserve one contiguous slab per request sized for the maximum length waste space inside the slab (internal fragmentation) and leave holes between slabs (external fragmentation).

Waste in existing systems, paperOnly 20.4% to 38.2% of KV cache memory held the actual token states in the existing systems the paper measured
Waste in existing systems, vLLM blog60% to 80% of KV cache memory wasted through fragmentation and over-reservation
PagedAttention, paperNear-zero waste in KV cache memory; 2-4x higher throughput than FasterTransformer and Orca at the same latency
PagedAttention, vLLM blogWaste only in the last block of a sequence, under 4% in practice; up to 55% less memory for shared-prompt methods such as parallel sampling and beam search
Scale of the problemFor a 13B OPT model, 800 KB of KV cache per token, so as much as 1.6 GB for a request at the 2048-token maximum

PagedAttention borrows from operating-system paging. It splits the KV cache of each sequence into fixed-size blocks that do not have to sit next to each other, and allocates them as tokens are generated. The default block size in vLLM is 16 tokens. Because blocks are small and uniform, any free block fits any request, so external fragmentation largely disappears and internal waste is limited to the last, partly filled block.

These figures are the authors own, measured on the systems, models, GPUs and workloads in their paper and blog, and the throughput gain is larger for longer sequences, larger models and more complex decoding. Treat them as published results under those conditions, not a guarantee for your deployment.

The paper also notes that on an NVIDIA A100 with 40GB, running a 13B model, close to 30% of memory holds the KV cache while model weights take about 65%, which is why KV cache efficiency decides how large a batch you can run. For the cost and capacity side of this, see how to reduce GPU costs when serving large language models and how to scale LLM inference for thousands of concurrent users.

JAX and XLA: preallocation instead of fragmentation

JAX takes a different route. Its GPU memory allocation docs say JAX preallocates 75% of total GPU memory when the first operation runs (XLA_PYTHON_CLIENT_MEM_FRACTION controls the share), and that preallocating minimizes allocation overhead and memory fragmentation but can sometimes cause out-of-memory errors.

XLA_PYTHON_CLIENT_PREALLOCATE=false disables preallocation so JAX allocates as needed. The docs warn that a program using most of the available memory may then run out of memory.
XLA_PYTHON_CLIENT_ALLOCATOR=platform is described as very slow and not recommended for general use, but useful for a minimal memory footprint or for debugging out-of-memory failures.

The trade-off is the same as elsewhere: reserving one big region up front avoids fragmentation, at the price of leaving less for other processes sharing the GPU.

CUDA virtual memory management: the low-level fix

Underneath the frameworks, NVIDIA added low-level virtual memory management APIs in CUDA 10.2. cuMemAddressReserve reserves a virtual address range, cuMemCreate creates physical memory, cuMemMap maps physical memory into the reserved range, and cuMemSetAccess sets access rights. The stated aim from NVIDIA is to grow an allocation as you need more memory while keeping a contiguous address range, without the cudaMemcpy that growing a cudaMalloc buffer requires.

This is the same idea as expandable segments in PyTorch and the non-contiguous blocks of PagedAttention: separate the address the program sees from the physical memory behind it. The NVIDIA article does not itself discuss fragmentation, so the link is an inference from the design, not a claim from that page.

A practical checklist

Measure first. Compare memory_allocated() with memory_reserved() and take a memory snapshot at the failure.
Try expandable_segments:True. Then, if needed, garbage_collection_threshold, and max_split_size_mb as a last resort.
Keep shapes consistent. Padding or bucketing variable-length inputs to a few sizes gives the allocator blocks it can reuse. This is common engineering practice rather than a documented PyTorch setting, so verify the effect in a snapshot.
Preallocate where supported. JAX does so by default; in PyTorch you can run one worst-case batch first so the cache is sized for the largest request.
Use a paged KV cache for LLM serving. vLLM is built on PagedAttention.
Restart long-running processes. If reserved memory keeps growing and snapshots show many small free blocks, a periodic restart resets the allocator. No published schedule exists; pick one from your own monitoring.

Questions people ask

What is GPU memory fragmentation?

It is free GPU memory broken into non-contiguous chunks. A large allocation can fail with an out-of-memory error even though the total free memory looks sufficient, because no single free block is big enough.

Does nvidia-smi show fragmentation?

No. The PyTorch docs say unused memory held by its caching allocator still shows as used in nvidia-smi. Compare memory_allocated() with memory_reserved(), and use a memory snapshot to see how segments are split into free and allocated blocks.

Should I use max_split_size_mb?

Only after other options. The PyTorch docs say it should be used as a last resort and that its performance cost ranges from zero to substantial. The docs publish no recommended value, so test on your workload.

Does PagedAttention remove fragmentation completely?

Not completely. The vLLM paper reports near-zero KV cache waste, and the vLLM blog says waste occurs only in the last block of a sequence, under 4% in practice. Those are the authors results under their stated conditions.

Is expandable_segments safe to turn on?

The PyTorch docs mark it experimental and default it to False. It is designed to reduce unusable slivers when batch sizes change. Test it on your workload and check speed and peak memory before rolling it out.

Sources

PyTorch: CUDA semantics, memory management · caching allocator, allocator settings
PyTorch: Understanding CUDA memory usage · memory snapshots and the viewer
Kwon et al.: Efficient Memory Management for LLM Serving with PagedAttention · KV cache waste and PagedAttention
vLLM blog: Easy, fast, and cheap LLM serving with PagedAttention · waste figures and block sharing
JAX: GPU memory allocation · preallocation settings
NVIDIA: Introducing Low-level GPU Virtual Memory Management · CUDA virtual memory APIs

Figures were read from the linked pages on October 20, 2026. Options and defaults change between PyTorch, JAX and CUDA releases, and the benchmarks are the authors own, so check the current docs and test on your own workload before changing production settings.