Checkpointing at scale: stop losing hours to failed nodes

Failures are a rate, not an event

On a single GPU, a hardware failure is an unlucky afternoon. On hundreds of GPUs, it is a steady background rate you plan around. Meta's Llama 3 paper reported 419 unexpected job interruptions over a 54-day period of pretraining on a cluster of roughly 16,000 H100s, about one every three hours, with GPU and HBM faults the largest category. You do not need to run anywhere near that scale for the arithmetic to matter: failure rate grows linearly with node count, while the cost of each failure grows with how much work you lose.

The goal of a checkpointing strategy is to minimize total wasted time, which has three parts: time spent writing checkpoints, work lost since the last checkpoint when a failure hits, and time to restart. This post covers each, with the math to pick an interval and the storage layout to make writes cheap.

How big is a checkpoint?

For mixed-precision training with Adam, a full training checkpoint typically holds:

So an 8B model checkpoint is about 112 GB and a 70B model is about 980 GB, before data-loader state, RNG state, and scheduler state, which are small. Some setups skip the separate BF16 copy and store only FP32 master weights, bringing the total to about 12 bytes per parameter. Either way, checkpoint size scales with parameters, not GPUs, so it is fixed for a model while the write bandwidth available to you scales with the number of nodes writing in parallel.

The interval math: Young/Daly

The classic result, from Young (1974) and refined by Daly (2006), gives the checkpoint interval that minimizes expected waste:

tau_opt ≈ sqrt(2 × C × M)

C = time training is blocked per checkpoint
M = mean time between failures for the whole job

Job MTBF is node MTBF divided by node count: a job on 64 nodes fails 64 times as often as one node. With an interval tau, the expected fraction of time wasted is roughly:

waste ≈ C / tau + tau / (2 × M) + R / M

R = restart cost: detect, reschedule, reload, warm up

An illustrative example, with assumed numbers rather than measurements. Suppose your 64-node job sees one interruption every 12 hours (M = 43,200 s) and restart takes 15 minutes (R = 900 s).

Two lessons fall out. First, reducing blocking time is worth more than tuning the interval, because it lets you checkpoint more often at lower cost. Second, once checkpointing is cheap, restart cost dominates, so the time from failure to resumed training deserves as much engineering attention as the save path. At cluster scale, the difference between 7.4% and 2.2% waste is several node-days per month.

Recompute tau whenever you resize the job, since M changes with node count.

Make saves asynchronous

A synchronous save blocks every rank until the last byte is on storage. An asynchronous save splits the work: block briefly to copy state from GPU memory to pinned host memory, then write to storage in a background thread while training continues. The blocking time C drops from "time to write to storage" to "time to copy to host RAM", which is usually seconds.

PyTorch Distributed Checkpoint (DCP) supports this directly:

import torch.distributed.checkpoint as dcp
from torch.distributed.checkpoint.state_dict import get_state_dict

pending = None

def save(step, model, optimizer, extra_state):
    global pending
    if pending is not None:
        pending.result()          # never queue two saves at once
    model_sd, optim_sd = get_state_dict(model, optimizer)
    state = {"model": model_sd, "optim": optim_sd, "extra": extra_state}
    pending = dcp.async_save(state, checkpoint_id=f"/mnt/ckpt/step_{step:08d}")

Two cautions. Host memory must hold a staged copy of each rank's shard, so check that your per-node RAM can fit the node's share of the checkpoint alongside the data loader. And wait for the previous save to finish before starting the next; if saves start overlapping, your interval is shorter than your write time and you need faster storage or a longer interval.

Shard, don't gather

The worst checkpoint pattern at scale is gathering the full state dict to rank 0 and writing one file. It serializes the write through one process, one NIC, and one disk, and it can run rank 0 out of memory for large models.

Sharded checkpointing has every rank write its own shard in parallel, plus a small metadata file that describes how shards map to the global tensors. DCP does this by default with FSDP and tensor-parallel state. Write bandwidth then scales with node count: if each node can sustain several GB/s to its local disk, a 980 GB checkpoint spread over 8 nodes is about 122 GB per node and finishes in tens of seconds rather than many minutes.

Sharded checkpoints are also reshardable. DCP can load a checkpoint saved at one world size into a job with a different world size, which matters when a failure forces you to resume on fewer nodes, or when you scale up mid-run.

Where to put checkpoints: a two-tier layout

Speed and durability pull in different directions, so use two tiers.

Tier 1: fast, close to the GPUs. Write async checkpoints to local NVMe or to an NVMe block volume attached to the instance. Our 8×H100 nodes ship with 22 TB of local NVMe, and block volumes are triple-replicated NVMe that survive instance termination and can be reattached to a different instance. Keep the last two or three checkpoints here for fast restarts.

Tier 2: durable, decoupled from the cluster. Copy every Nth checkpoint, or every checkpoint for critical runs, to S3-compatible object storage in the background. Object storage is designed for 11 nines of durability and is independent of any node's health. It is also where checkpoints go to be shared: evaluation jobs, fine-tuning runs, and serving deployments read from there.

Upload from tier 1 to tier 2 with a separate process at low priority so it does not compete with training I/O, and verify each upload with checksums before deleting the local copy. Write a small "complete" marker only after every shard and the metadata are uploaded, and have the resume logic ignore any checkpoint without one. Partial checkpoints are worse than no checkpoint, because they fail at the worst possible time.

Fast restarts

Once saves are cheap, restart cost R dominates. Ways to shrink it:

• Automatic failure detection. Use torchrun with --max-restarts and a rendezvous backend so a failed rank triggers a coordinated restart, and set NCCL timeouts that detect hangs in minutes, not hours.
• Hot spares. For long runs, keep one spare node per few dozen in the job, already provisioned with the image and volumes mounted. With per-second billing, a spare costs exactly its runtime, and it is usually cheaper than the idle cluster waiting for a replacement.
• Fast image boot. Our median cold boot is 74 seconds from request to SSH; a replacement node joins far faster than the time it takes a person to notice the failure.
• Resume from tier 1. Reading from local NVMe or an attached block volume is much faster than pulling hundreds of gigabytes from object storage.

Run resume drills

A checkpoint you have never restored is a hope, not a backup. Schedule drills:

1. Kill a rank mid-run at a random step, monthly or before any long run, and time detection to resumed training.
2. Verify continuity. After resume, the loss at step N+1 should match an uninterrupted reference run closely. Large deviations usually mean the data-loader position, RNG state, or learning-rate scheduler was not saved.
3. Restore at a different world size to confirm resharding works before you need it.
4. Restore from tier 2 only, as if every node were gone, and measure how long it takes.
5. Write down R and feed it back into the interval math.

For the network side of multi-node jobs, see our InfiniBand and NCCL tuning guide. Storage rates for volumes, object storage, and snapshots are on pricing.

FAQ

How often should I checkpoint a large training run?

Use the Young/Daly approximation: the square root of 2 × blocking time × job MTBF. With async saves that block for seconds, the optimal interval is often 10-20 minutes for multi-node jobs.

What is asynchronous checkpointing?

Training blocks only long enough to copy state to host memory, then a background thread writes it to storage while training continues. PyTorch DCP provides this through async_save.

Should checkpoints go to NVMe or object storage?

Both. Write to local NVMe or a block volume for speed and fast restarts, then copy to object storage in the background for durability and sharing.

How big is a training checkpoint?

About 12-14 bytes per parameter with Adam and mixed precision, so roughly 112 GB for an 8B model and about 980 GB for a 70B model.