Fine-tuning Llama 3 70B on a single 8×H100 node

The case for one node

Teams routinely assume 70B fine-tuning means multi-node infrastructure, NCCL tuning, and a platform engineer on call. For LoRA — which is the right choice for the overwhelming majority of fine-tuning tasks — a single 8×H100 node is not just sufficient, it is comfortable. No inter-node fabric to tune, no stragglers, no distributed-filesystem checkpointing dance. On Reviosa the node is $18.32/hr, and the run in this post finished for about $64.

The memory budget

Start from what has to fit. In BF16, Llama 3 70B's weights are ~140 GB. Sharded with FSDP across 8 GPUs, that is 17.5 GB per GPU. LoRA with r=16 on all linear projections adds roughly 200M trainable parameters — about 0.3% of the model — so gradients and AdamW optimizer states for the adapters total under 4 GB per GPU even in FP32. That leaves ~55 GB per GPU for activations, which is enough to run sequence length 4,096 with micro-batch 2 and only selective activation checkpointing.

Compare that with full fine-tuning, where weights, gradients, and FP32 optimizer states cost ~16 bytes per parameter — 1.1 TB for 70B, before activations. That is the workload that needs a cluster. LoRA is not a compromise here; it is the reason the job fits.

The recipe

We use axolotl for runs like this; torchtune works equally well. The config that matters:

base_model: meta-llama/Meta-Llama-3-70B
adapter: lora
lora_r: 16
lora_alpha: 32
lora_target_linear: true
sequence_len: 4096
sample_packing: true
micro_batch_size: 2
gradient_accumulation_steps: 8
num_epochs: 3
learning_rate: 1.0e-4
lr_scheduler: cosine
warmup_ratio: 0.03
bf16: true
flash_attention: true
fsdp:
  - full_shard
  - auto_wrap
fsdp_config:
  fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer
  fsdp_state_dict_type: SHARDED_STATE_DICT

Micro-batch 2 × 8 GPUs × 8 accumulation steps gives a global batch of 128 packed sequences — roughly 500k tokens per step, which is a stable regime for LoRA at this learning rate. Sample packing matters more than people expect: on a chat dataset with median length ~700 tokens, packing lifted effective throughput by ~2.8× versus padded batches. Hold out a real eval split and log it every half epoch — LoRA runs are short enough that "train to completion, then evaluate" wastes your fastest feedback loop.

Throughput and the bill

On this configuration we sustain ~11,900 tokens/second across the node. For a 50M-token dataset over 3 epochs (150M tokens total):

• Wall clock: 150M ÷ 11,900 tok/s ≈ 3.5 hours
• Cost: 3.5 hrs × $18.32 ≈ $64

Even with a conservative 30-minute allowance for environment setup, model download (the 140 GB of weights pull in ~10 minutes on node-local NVMe), and a smoke-test batch, you are under $75. With per-second billing you pay for none of the padding an hourly meter would add.

Gotchas worth knowing

1. Checkpoint shards, then merge offline. Save SHARDED_STATE_DICT during training and merge the LoRA adapter afterward on a single GPU — gathering a full 140 GB state dict mid-run stalls training and can OOM host memory.
2. Validate on packed and unpacked batches. Packing changes attention masking; a quick unpacked eval catches config mistakes that packed loss curves hide.
3. Learning rate is per-adapter, not per-model. 1e-4 is a sane LoRA default; the 1e-5 you would use for full fine-tuning will look like the model is barely learning.
4. Watch step-time variance. On a single node it should be near-zero. If it isn't, the culprit is almost always the dataloader — raise worker count before touching anything else.

When you'd still go multi-node

Full-parameter fine-tuning, RLHF with separate reward and policy models, or LoRA over datasets in the billions of tokens where wall clock dominates cost. For everything else — domain adaptation, tone and format tuning, tool-use training — one node, one afternoon, one two-digit invoice.