answrsnow
AI Infrastructure · Explainer

How Do You Optimize Cold-Start Latency for Serverless AI Inference in 2027?

By Kory White · Updated Oct 2026 · 8 min read
Rows of server racks in a data center
Direct answer

You cut cold-start latency by shrinking or skipping each stage of startup, then paying to keep a few instances warm for the traffic that cannot wait. A cold start is the time a new replica needs before it can serve a request, and for an AI model it is made of pulling the container image, loading model weights, initializing the runtime and CUDA, and compiling or warming up. The fixes follow the stages: smaller images, weights delivered from fast storage or streamed, snapshots that restore an already-initialized process, quantized weights, and a minimum number of warm instances.

The tradeoff is idle cost. A warm GPU is billed whether or not it serves traffic, so the real decision is how many warm instances your latency target justifies.

No vendor source used here publishes a typical cold-start time for a given model, so none is given. The numbers below are each vendor's own, on its own platform, read from its documentation.

Where the time goes

Baseten's documentation splits a cold start into four phases: acquiring resources (creating the replica and scheduling it onto a node), pulling the container image, pulling the model weights, and loading the model. The last phase is where AI-specific work lives: for engines such as vLLM and SGLang, Baseten says it includes loading weights into GPU memory, capturing CUDA graphs and compiling kernels.

AWS describes the same dependency for SageMaker Serverless Inference: cold start time depends on your model size, how long it takes to download your model, and the start-up time of your container. Modal makes the point from the other side: some work only needs to happen on the first invocation, and containers are not considered warm until their initialization methods finish.

None of these pages publishes a typical duration for each stage, because it depends on model size, storage and hardware. Measure your own stages before optimizing; Baseten, for one, reports startup time percentiles per deployment.

Make the image smaller and keep weights out of it

Baseten's first recommendation is smaller images: remove unused entries from requirements and system packages, and deliver weights through its Baseten Delivery Network rather than baking them into the image. It also notes that replicas on nodes that already have your image skip the image-pull phase.

Google's Cloud Run guidance for GPU services says container images are best suited for smaller models less than 10 GB, and that an image containing a large model takes longer to import. Its main recommendation is to store models in Cloud Storage, and it lists Cloud Storage FUSE mount options to reduce model read times. It calls internet downloads typically poor and unpredictable.

Modal's guide goes the other way for some setups: download the model ahead of time, into the image or a Volume, so it is fetched once. Modal says this can reduce boot times from minutes to seconds for models in the tens of gigabytes. The two views fit together: do the download once, ahead of time, and keep it out of the request path, whether that is in an image layer, a platform weight cache or a volume.

NVIDIA L4 data center GPU

Load weights faster

Once the image is up, the model still has to reach GPU memory. vLLM supports the Run:ai Model Streamer, a library that reads tensors concurrently while streaming them to GPU memory. Its docs say distributed streaming can significantly improve loading times from object storage or high-throughput network file shares, and list AWS S3, Google Cloud Storage and Azure Blob Storage as supported sources. The page gives no speedup figure, so test on your own storage.

Baseten's list adds practical tips: skip files you do not need with include and exclude patterns, pin a model revision, prefer safetensors files, and defer work that is not needed before the first request. Local NVMe and node-level caches work on the same principle, serving weights from storage close to the GPU instead of re-downloading them. Baseten says a later cold start no longer depends on the original Hugging Face, S3 or GCS source once weights are mirrored to its storage.

Shrink the weights with quantization

Smaller weights mean less to pull and less to move into GPU memory. Baseten lists FP8 or FP4 quantized weights among its ways to speed up weight loading. Treat that as one lever with a quality cost: run your own accuracy evaluation before and after, because the effect differs by model and task. For the serving-side view of quantization, see our guide to reducing GPU costs serving large language models.

Snapshot and restore instead of initializing

A snapshot saves the state of an initialized process so new instances resume from it instead of repeating startup. Modal's memory snapshots save a container's CPU memory state; Modal says they can dramatically reduce cold start latency, and that initialization-heavy functions often start 3-10x faster. It cites the example that loading torch takes over 20,000 file operations, which a snapshot can skip.

Modal's GPU memory snapshots extend this to GPU state, but the page labels them an alpha feature with real limits: they are generally incompatible with multi-GPU code, they do not speed up loading weights from storage and may add overhead there, and they can interact poorly with torch.compile. Check those caveats against your stack before relying on them.

AWS Lambda SnapStart is the same idea for functions: Lambda takes a Firecracker microVM snapshot of the initialized environment when you publish a version, and AWS says startup can be as low as sub-second. It supports Java 11 and later, Python 3.12 and later and .NET 8 and later, and does not support ephemeral storage greater than 512 MB, so it will not hold a large model. It is a pattern to borrow, not a GPU inference option.

A related idea inside the server: vLLM's sleep mode temporarily releases most GPU memory, including weights and KV cache, so the engine can wake up and resume inference without a full model reload. The docs say it can release up to 90%+ of GPU memory for other tasks but give no wake-up time.

Compile and warm up before the first request

Compilation and warm-up are part of the loading phase on Baseten, which says that for vLLM and SGLang it includes capturing CUDA graphs and compiling kernels. Its mitigation is torch compile caching, so new replicas reuse compiled artifacts. Whatever platform you use, the rule is the same: do this work once at build or snapshot time, and make sure the instance reports ready only after it finishes. Cloud Run's guidance says to allow sufficient time for the container, including models, to fully initialize and load.

Keep a few instances warm, and know what it costs

The surest fix is not to cold start. RunPod's endpoint settings default to 0 active workers, and setting it to 1 or more eliminates cold starts, but active workers incur charges continuously, including when they have no jobs. RunPod also enables FlashBoot by default on new endpoints, which retains worker state after spin-down for faster revival than a fresh boot and works best on endpoints with consistent traffic. Its idle timeout, default 5 seconds, keeps a worker running after its last job so a new request starts with no cold start.

Cloud Run is explicit about the bill: with GPUs, minimum instances are charged at the full rate even when idle. The same page says instances with an L4 or RTX PRO 6000 Blackwell GPU start in approximately 5 seconds, which is the platform's own figure for GPU availability, not for loading your model.

Modal exposes the same dials: min_containers keeps containers warm, and scaledown_window sets how long idle containers stay alive, with a default of 60 seconds and a range from 2 seconds to 20 minutes. Baseten's min_replica set to 1 or higher keeps a replica ready for the first request, and it suggests raising it ahead of known traffic spikes, using your p99 startup time as the lead time.

For AWS, SageMaker Serverless Inference scales to zero when there are no requests and suits workloads that can tolerate cold starts. Provisioned concurrency keeps the endpoint warm and responding within milliseconds, and you pay for it separately. Two limits matter for AI work: serverless endpoints top out at 6 GB of memory and a 10 GB container image, and GPUs are on the feature exclusion list, so large GPU models need a different option.

The tradeoff in one line: every warm GPU is a standing bill. Keep the floor at the smallest count that meets your latency target, use scheduled or pre-spike increases for known bursts, and let the rest scale from zero.

Route requests around the cold ones

Platforms handle routing differently, and the documentation says little about it. What the sources do show is the behavior: on a scaled-to-zero Baseten deployment, a synchronous request that triggers a cold start waits for the replica to load, and Modal's latency comes from inputs waiting in a queue plus first-invocation work. Practical routing, such as sending latency-critical traffic to the always-warm pool and batch or tolerant traffic to scale-from-zero capacity, is a design choice you make in your own gateway; none of the vendor pages used here prescribes a specific routing method or publishes numbers for it.

Questions people ask

What is a cold start in serverless AI inference?

It is the time a new replica needs before it can accept traffic. Baseten splits it into acquiring resources, pulling the image, pulling weights and loading the model, and SageMaker's docs say it depends on model size, download time and container start-up time.

What is the fastest way to remove cold starts?

Keep instances warm. RunPod says setting active workers to 1 or more eliminates cold starts, and Baseten says min_replica of 1 or higher keeps a replica ready. The cost is that warm GPUs are billed while idle.

Do snapshots work for GPU models?

Modal offers GPU memory snapshots but labels them an alpha feature, generally incompatible with multi-GPU code, and says they do not speed up loading weights from storage. AWS Lambda SnapStart supports Java, Python and .NET runtimes and not ephemeral storage above 512 MB.

Does SageMaker Serverless Inference support GPUs?

The documentation lists GPUs among the features not supported for Serverless Inference, with a 6 GB memory maximum and a 10 GB container image limit.

Should model weights go in the container image?

It depends on size. Google says images suit models under 10 GB and recommends Cloud Storage for larger ones; Baseten recommends delivering weights through its cache instead of baking them in; Modal recommends downloading ahead of time into the image or a Volume.

How much does quantization help cold starts?

None of the pages used here gives a figure. Baseten lists FP8 or FP4 quantized weights as a way to speed weight loading, and quality should be tested on your own tasks.

Sources

Figures were read from the linked vendor documentation on October 21, 2026; the Cloud Run pages show a last-updated date of 2026-10-20 and the other pages show none. Features, limits and prices change, and several are vendor-specific, so confirm on the current page and test on your own model.

Top photo: Carl Lender / CC BY 2.0 via Wikimedia Commons