H200 vs H100: real-world LLM training benchmarks

What actually changed between H100 and H200

The H200 is not a new GPU in the way the H100 was. It is the same GH100 silicon, the same SXM board power envelope, and the same ~989 TFLOPS of dense BF16 tensor throughput. What changed is the memory system: 141 GB of HBM3e at 4.8 TB/s, up from 80 GB of HBM3 at 3.35 TB/s. That is 76% more capacity and 43% more bandwidth, with zero additional compute.

This framing tells you exactly what to expect. Workloads that are compute-bound should be nearly identical. Workloads that are memory-bound — or that are forced into expensive workarounds by the 80 GB ceiling — should improve, sometimes dramatically. We wanted to know where real training jobs land on that spectrum, so we measured.

Benchmark setup

All runs used a single node with 8 GPUs connected over NVLink, PyTorch 2.7 with FSDP2, BF16 mixed precision, FlashAttention, and packed sequences. We tested two representative jobs:

• Llama 3 8B, full fine-tune — every parameter trainable, AdamW, sequence length 4,096.
• Llama 3 70B, LoRA r=16 — adapters on all linear projections, tested at both 4,096 and 8,192 sequence length.

Numbers are median tokens/second per node over five runs after warmup. Both node types sat in the same rack family with identical storage and dataloader configurations.

Results

Reading the numbers

The 8B full fine-tune is the compute-bound case, and it behaves like one: +7%, almost entirely attributable to memory bandwidth shaving time off attention and optimizer steps. If your whole world is models that fit comfortably in 80 GB, the H200 is a modest upgrade.

The 70B runs are more interesting. On the H100 we needed full activation checkpointing to fit 8k sequences; on the H200, 141 GB per GPU let us checkpoint only every other layer and double the micro-batch size. Less recomputation plus better arithmetic intensity is where the 1.32× comes from. The H200 isn't computing faster — it's recomputing less.

The cost math

On Reviosa, an 8×H100 node is $18.32/hr ($2.29 per GPU-hour), and H200s are $3.69 per GPU-hour on demand ($29.52/hr for eight). Converting throughput to cost per million training tokens:

• 8B full fine-tune: $0.045/M tokens on H100 vs $0.068/M on H200 — H100 wins by a wide margin.
• 70B LoRA at 8k: $0.559/M on H100 vs $0.683/M on H200 — H100 still wins, but the gap narrows from 51% to 22%.

A 1.32× speedup doesn't overcome a 1.48× price ratio. For pure training throughput per dollar, the H100 node remains the default answer.

When the H200 wins anyway

Three situations flip the recommendation:

1. Capacity keeps you on one node. If 141 GB per GPU means single-node instead of 2-node training, you avoid interconnect overhead and a second node's bill entirely. That swing is far larger than 22%.
2. Wall-clock deadlines. If a run must finish before a launch review, 32% faster is worth 48% more per hour.
3. You'll serve from the same fleet. Inference decode is bandwidth-bound, and long-context KV caches love the extra 61 GB. H200s that train by night and serve by day amortize beautifully.

Recommendation

Default to 8×H100 nodes at $18.32/hr for training — the economics are hard to beat. Reach for H200 when memory capacity changes the shape of the job, not just its speed. And whatever you pick, measure your own workload: our benchmark harness took an afternoon to set up and has paid for itself on every hardware decision since.