Cutting inference cold starts: container images and weights

Where cold-start time actually goes

A cold start is everything between "we need another replica" and "that replica answers its first request at normal speed." Teams usually treat it as one number, which makes it impossible to fix. It is really six phases with different causes and different remedies:

Phase 1 is largely the platform's job. On Reviosa GPU instances the published median cold boot, request to SSH prompt, is 74 seconds, and serverless endpoints on Reviosa AI use warm pools and pinned weight caches to publish a 1.9-second zero-to-first-token cold start. Phases 2 to 6 are yours, and they routinely add several minutes on top.

Measure every phase first

You cannot optimize what you have not split apart. Wrap the startup path with timestamps:

ms() { date +%s%3N; }
t0=$(ms); docker pull "$IMAGE"
t1=$(ms); python fetch_weights.py --dest /mnt/weights
t2=$(ms); docker run -d --gpus all --name llm -p 8000:8000 -v /mnt/weights:/weights "$IMAGE"
until curl -sf localhost:8000/health > /dev/null; do sleep 0.5; done
t3=$(ms); python warmup.py --url http://localhost:8000
t4=$(ms)
echo "pull $(( t1 - t0 )) ms | fetch $(( t2 - t1 )) ms | start+load+init $(( t3 - t2 )) ms | warmup $(( t4 - t3 )) ms"

Then split the middle bucket using the engine's own logs. vLLM reports model loading time, torch.compile time and CUDA graph capture time at startup. Ship those numbers to your metrics system on every start, not just once, because they regress silently when someone bumps a base image or a model revision.

1. Slim the container image

Inference images bloat easily to 15 GB or more, and every gigabyte has to be downloaded, decompressed and unpacked before the container starts.

• Use a runtime base, not a devel base. CUDA devel images ship compilers and headers that inference never uses, and they are several GB larger.
• Build in stages. Compile or install in a builder stage and copy only the installed packages into the final stage.
• Skip caches. pip install --no-cache-dir, and clean apt lists in the same layer that creates them.
• Order layers by change frequency. System packages first, Python dependencies next, application code last, so a code change invalidates one small layer.
• Do not bake weights into the image. Multi-gigabyte layers pull and unpack slowly, and coupling weights to code means every code deploy reships the model.

FROM python:3.12-slim AS build
ARG VLLM_VERSION
RUN pip install --no-cache-dir --prefix=/install "vllm==$VLLM_VERSION"

FROM python:3.12-slim
COPY --from=build /install /usr/local
COPY server/ /app/server/
ENTRYPOINT ["python", "-m", "server.main"]

PyTorch wheels bundle the CUDA user-space libraries, so a slim Python base works with the host's NVIDIA driver. You will still have several GB of framework; the goal is to remove everything else.

2. Lazy-pull image layers

A container rarely reads most of its image at startup. The Slacker study (USENIX FAST 2016) found that pulling accounted for 76% of container start time while only 6.4% of the pulled data was read. Lazy-pulling snapshotters such as stargz-snapshotter (eStargz), SOCI and Nydus start the container immediately and fetch file contents on demand.

Two caveats for GPU images. First, importing PyTorch and initializing CUDA touches large shared libraries early, so the gains are smaller than for typical web services. Second, lazy pulling needs containerd configuration and converted images. If you run your own runtime on GPU instances it is worth testing; measure phase 2 before and after rather than assuming.

On long-lived instances the simplest win is the local layer cache: keep the base and dependency layers stable, and a redeploy pulls only the small application layer.

3. Pre-stage weights on NVMe volumes

Weight fetch is usually the biggest single phase, and it is pure bandwidth arithmetic. The estimates below use typical sequential rates; they are illustrative, not measurements of any particular network:

The fix is to never download from the internet on the critical path. Keep a golden copy of each pinned model revision in an in-region bucket, and stage it onto NVMe before the server starts. Every Reviosa GPU SKU includes local NVMe, persistent block volumes attach to any instance in seconds, and traffic between instances and storage on the private network is unmetered; see Storage. A volume that already holds the weights turns phase 3 into zero.

4. Load with safetensors and mmap

Pickle-based .bin checkpoints are deserialized into CPU memory and then copied, which is slow and doubles peak host memory. safetensors files are a small JSON header followed by raw tensor bytes, so they can be memory-mapped: loads are close to zero-copy, and a restart on the same machine reuses the page cache.

import time
import torch
from safetensors import safe_open

start = time.perf_counter()
tensors = {}
with safe_open("/weights/model-00001-of-00004.safetensors", framework="pt", device="cuda:0") as f:
    for name in f.keys():
        tensors[name] = f.get_tensor(name)
torch.cuda.synchronize()
print(f"{len(tensors)} tensors in {time.perf_counter() - start:.1f} s")

Further gains, in rough order of effort:

• Load pre-quantized weights. An FP8 checkpoint is half the bytes of BF16 and skips quantizing at startup.
• Parallel and streaming loaders. vLLM supports alternative load formats, including the Run:ai Model Streamer and fastsafetensors, which read tensors concurrently from disk or object storage.
• Pre-sharded checkpoints. With tensor parallelism, save one shard per rank so each GPU reads only its slice instead of the whole model.

5. CUDA graph capture and warmup

CUDA graphs remove per-step kernel launch overhead during decode, which is a real throughput win, but capturing them for many batch sizes costs startup time. Combined with torch.compile, engine initialization can take tens of seconds.

• Persist compile caches. vLLM caches compilation artifacts on disk; point its cache directory (VLLM_CACHE_ROOT) at a persistent volume so restarts reuse them.
• Capture fewer sizes. Limit the set of batch sizes captured to the ones your traffic actually reaches. The flag has changed across vLLM versions, so check vllm serve --help for yours.
• Use eager mode where startup beats speed. --enforce-eager skips graph capture entirely. It suits dev boxes and canaries, not high-traffic replicas.
• Sleep instead of restarting. Recent vLLM versions offer a sleep mode that offloads weights and frees the KV cache, then wakes in seconds, useful when several models share a GPU.

Finally, warm up before taking traffic. The first requests trigger lazy work such as library handle creation, allocator growth and kernel autotuning. Send a handful of synthetic requests covering short and long prompts, and only then pass the readiness check your load balancer uses. A health endpoint that returns 200 is not the same as a replica that serves at full speed.

6. Keep minimum replicas

The fastest cold start is the one you do not have. Keeping one warm replica costs its hourly rate from /pricing; compare that with what a slow first response costs you in SLO breaches or abandoned sessions. Per-second billing makes schedules practical: minimum one replica during business hours, zero overnight, scaled ahead of known traffic ramps. Scale on queue depth, not CPU, because the GPU queue is where users wait. Our vLLM production guide covers the metrics to watch.

A cold-start budget

Putting it together for an 8B model, here is an illustrative before-and-after budget built from the arithmetic above. It is a planning example, not a measurement:

Your numbers will differ. The method will not: measure each phase, attack the largest one, and repeat.

FAQ

What causes cold starts in LLM inference?

Image pull, weight download, weight loading into GPU memory, engine initialization such as CUDA graph capture and compilation, and first-request warmup. Weight download and image pull usually dominate, so measure each phase separately before optimizing.

Should I bake model weights into the container image?

Usually not. Multi-gigabyte layers pull and unpack slowly, and every code change reships the model. Keep weights on a persistent NVMe volume or in in-region object storage, pinned by revision, and mount or stage them at start.

Does --enforce-eager make vLLM start faster?

Yes. It skips CUDA graph capture, which shortens startup, but it lowers decode throughput. Use it for dev and canary replicas, and keep graphs for high-traffic replicas with a persistent compile cache.

How many minimum replicas should I keep for inference?

Keep at least one warm replica for user-facing endpoints during hours when traffic is expected, and scale to zero for internal or batch workloads. Compare the hourly cost of an idle replica with the cost of a slow first response.