FP8 training on Hopper and Blackwell: a practical guide

Why FP8, and what it really buys

On Hopper and Blackwell GPUs, FP8 tensor-core throughput is twice the BF16 figure on the spec sheet: roughly 1.98 PFLOPS dense FP8 versus 0.99 PFLOPS BF16 on an H100, and roughly 4.5 versus 2.25 on a B200. FP8 also halves the bytes moved for every tensor stored in it.

End-to-end training does not speed up 2×, because only the large matrix multiplications run in FP8. Attention softmax, normalization, optimizer steps, communication, and data loading do not. Publicly reported results for large transformer training typically land around 1.2-1.5× over a well-tuned BF16 baseline, with larger models gaining more because GEMMs dominate their step time. Treat that as an illustrative range, not a promise: a 1B model with short sequences may see almost nothing.

This guide covers the formats, the scaling recipes, what to keep in higher precision, how to verify you have not broken training, and what changes on Blackwell.

The two FP8 formats

E4M3 has more precision and less range; E5M2 has more range and less precision. Gradients have a wider dynamic range than activations, which is why the common "hybrid" recipe uses E4M3 forward and E5M2 backward. Some recent recipes use E4M3 everywhere with finer-grained scaling, which works because the scaling, not the format, provides the range.

With only 3 mantissa bits, E4M3 can represent only 8 distinct values per power of two. FP8 training works because each tensor is multiplied by a scale factor before casting, so its values occupy the representable range, and divided by the same factor afterward. Everything interesting about FP8 training is about choosing those scales.

Scaling recipes

Delayed scaling (per tensor). The original Transformer Engine recipe. For each FP8 tensor, keep a history of recent absolute-maximum (amax) values and compute the scale for the current step from that history. It is cheap because the amax for step N is reused at step N+1 rather than requiring an extra pass over the data. The risk is that a sudden outlier at step N overflows, because the scale was set from past steps. Overflowed values saturate, which usually shows up as a loss spike.

Current scaling (per tensor). Compute the amax of the tensor right now, then scale and cast. This is more robust to sudden shifts and simpler to reason about, at the cost of an extra reduction per tensor. Transformer Engine 2.x exposes this as a separate recipe, and torchao's "tensorwise" float8 recipe is in the same family.

Block or row scaling. Instead of one scale per tensor, use one scale per row or per block. DeepSeek-V3's technical report describes 1×128 tiles for activations and 128×128 blocks for weights. Finer scaling contains outliers to small regions so the rest of the tensor keeps precision. On Hopper this is done in software around the GEMM; on Blackwell, the MX formats move it into hardware.

A pragmatic order of preference on Hopper: start with current per-tensor scaling for simplicity, move to delayed scaling if the extra reduction shows up in profiles and your loss curves are stable, and use row or block scaling for very large models or when you see persistent overflow.

A minimal Transformer Engine setup

API names below follow Transformer Engine 2.x; check the release notes for your version, as recipe classes and context-manager names have evolved.

import transformer_engine.pytorch as te
from transformer_engine.common.recipe import DelayedScaling, Format

recipe = DelayedScaling(
    fp8_format=Format.HYBRID,   # E4M3 forward, E5M2 backward
    amax_history_len=16,
    amax_compute_algo="max",
)

# te.Linear / te.LayerNormMLP / te.TransformerLayer replace nn.Linear blocks
with te.fp8_autocast(enabled=True, fp8_recipe=recipe):
    logits = model(input_ids)
loss = loss_fn(logits.float(), labels)
loss.backward()

If you prefer to stay in plain PyTorch modules, torchao offers a conversion path that swaps nn.Linear layers for float8 equivalents and composes with FSDP2 and torch.compile:

from torchao.float8 import convert_to_float8_training, Float8LinearConfig

config = Float8LinearConfig.from_recipe_name("rowwise")
convert_to_float8_training(model, config=config, module_filter_fn=keep_in_bf16)

Here keep_in_bf16 is your own function that returns False for layers that should stay in BF16, which brings us to the most important decision.

What to keep in BF16 or FP32

FP8 belongs in the large linear layers of the transformer blocks. Keep these in higher precision:

• Master weights and optimizer state in FP32. FP8 is a compute format, not a storage format for the parameters you update.
• Embeddings and the output (LM head) projection. They are sensitive, and the LM head feeds the loss directly. Many recipes also keep the first and last one or two transformer blocks in BF16.
• Normalization, softmax, and the loss itself. Compute the loss in FP32.
• Attention score and value BMMs, at least initially. FP8 attention exists but is a separate, more aggressive step; validate it on its own.
• Small linear layers. If a GEMM's dimensions are small (for example, under ~1,024 on a side), casting overhead can exceed the gain. Leave it in BF16.
• Gradient reduction. Reduce gradients in BF16 or FP32. FP8 all-gather of weights for FSDP is a valid communication optimization; FP8 gradient reduction is riskier.

Sanity-checking the loss curve

Never switch a long run to FP8 on faith. Our standard procedure:

1. Run a BF16 baseline and an FP8 run from the same checkpoint, same data order, same seed, for at least 1,000-2,000 steps, or long enough to pass the early high-learning-rate phase.
2. Compare smoothed loss. The FP8 curve should track BF16 closely. A small, stable gap (on the order of a fraction of a percent of the loss value) is typical; a gap that widens over time is not.
3. Watch gradient norm and spikes. FP8 problems often show up as more frequent or larger loss spikes before they show up in the mean.
4. Log amax and overflow counts for a sample of layers. Repeated saturation in one layer points at a layer to exclude or a recipe to change.
5. Run your downstream evals at a few checkpoints. Loss parity is necessary, not sufficient.
6. Measure the speedup honestly. Compare tokens per second at matched batch size and sequence length. If FP8 is under ~1.1× faster, check whether the GEMMs you converted are actually large enough to matter.

If the curve diverges, bisect: move layers back to BF16 in groups, switch delayed to current scaling, then try finer-grained scaling. Most divergences we see trace to one of three causes: the LM head in FP8, a long amax history masking a distribution shift, or a single outlier-heavy layer.

What changes on Blackwell: MXFP8

Blackwell tensor cores natively support microscaling (MX) formats defined by the Open Compute Project. In MXFP8, every block of 32 consecutive elements shares an 8-bit power-of-two scale (E8M0), applied inside the tensor core. Consequences:

• Outliers are contained to 32 elements. That makes E4M3 viable for gradients as well as activations, so MXFP8 recipes commonly use E4M3 throughout.
• No amax history to tune. Scales are computed from the block itself, which removes the main failure mode of delayed scaling.
• Less code around the GEMM. Block scaling that required custom kernels on Hopper becomes a hardware feature.

Transformer Engine exposes MXFP8 as its own recipe on Blackwell. The validation procedure above still applies; you are changing numerics, so re-run the BF16 comparison when you migrate. Our B200 migration guide covers the rest of the stack changes, including the CUDA 12.8 requirement.

Blackwell also introduces FP4 formats (NVFP4 and MXFP4). FP4 training is an active research area; for most teams today, FP4 is an inference format, and MXFP8 is the training format to adopt.

Where to run it

FP8 training needs Hopper or newer: H100 or H200 for classic FP8, B200 for MXFP8. Single GPUs from our GPU cloud are fine for the BF16-versus-FP8 comparison runs, and 8×H100 nodes with InfiniBand handle production-scale training. Per-second billing on our pricing means a 2,000-step comparison costs exactly what it runs. For multi-node jobs, pair FP8 with the communication tuning in our NCCL guide, since faster GEMMs make exposed communication a larger fraction of each step.

FAQ

How much faster is FP8 training than BF16?

Spec-sheet FP8 throughput is 2× BF16, but end-to-end gains are typically around 1.2-1.5× for large transformers in public reports, because only the large GEMMs run in FP8. Small models may see little benefit.

Should I use E4M3 or E5M2?

The common Hopper recipe uses E4M3 for the forward pass and E5M2 for gradients. With fine-grained block scaling such as MXFP8 on Blackwell, E4M3 throughout is common.

Which layers should stay in BF16?

Keep embeddings, the LM head, normalization, softmax, the loss, small linear layers, and often the first and last blocks in BF16. Keep master weights and optimizer state in FP32.

What is the difference between delayed and current scaling?

Delayed scaling sets each tensor's scale from a history of past maxima, which is cheap but can overflow on sudden outliers. Current scaling computes the maximum at the current step, which is more robust at a small extra cost.