Sizing GPUs for image and video generation models

Diffusion breaks the LLM sizing rules

For LLM serving, memory bandwidth and KV cache dominate the sizing conversation. Image and video generation models are different. Every denoising step runs the whole network over the whole latent, so the work is compute-bound; VRAM has to hold the weights, the text encoders, the VAE and one step's activations; and the sequence length is set by resolution and frame count, with attention cost growing with its square. Size for those three things and the choice between an RTX 4090 and a B200 becomes arithmetic.

From pixels to tokens

Modern diffusion transformers (DiTs) such as FLUX.1 and Stable Diffusion 3.5 compress the image 8x per side with a VAE, then group 2x2 latent patches into one token. Each token therefore covers a 16x16 pixel block. Video models add temporal compression: Wan 2.1-style VAEs compress time 4x and space 8x, then patch 2x2 spatially.

def dit_tokens(width, height, frames=1, spatial=8, temporal=4, patch=2):
    latent_frames = 1 if frames == 1 else (frames - 1) // temporal + 1
    return latent_frames * (height // (spatial * patch)) * (width // (spatial * patch))

base = dit_tokens(1024, 1024)
for name, (w, h, f) in {"512px image": (512, 512, 1), "1024px image": (1024, 1024, 1),
                        "2048px image": (2048, 2048, 1), "480p, 81 frames": (832, 480, 81),
                        "720p, 81 frames": (1280, 720, 81)}.items():
    n = dit_tokens(w, h, f)
    print(f"{name:>16}: {n:>7,} tokens  linear x{n / base:5.1f}  attention x{(n / base) ** 2:6.1f}")

A five-second 720p clip costs hundreds of times more attention compute per step than a single 1024px image. That one row explains why video generation lives on data center GPUs while image generation runs happily on an RTX card.

Memory is gentler than compute. FlashAttention-style kernels keep attention memory linear in sequence length, so inference activations scale with tokens times hidden size. For a 14B video DiT at 75,600 tokens, a single BF16 hidden-state tensor is around 0.8 GB; with several live intermediates and classifier-free guidance doubling the batch, budget roughly 5-15 GB of activations on top of weights. Treat that as an approximate planning figure and measure peak memory on your own pipeline.

VRAM: weights first

The denoiser is only part of the footprint. Large text encoders often add 10 GB or more.

Figures are approximate, from public model cards and parameter counts. Two techniques change the picture substantially:

• Encode, then evict. Text encoders run once per prompt. Encode, move the encoder to CPU or free it, then run the denoiser. For a service, cache embeddings for repeated prompts. This alone recovers roughly 10 GB on FLUX-class pipelines.
• FP8 weights. Quantizing the denoiser to FP8 halves its footprint with modest quality impact on most models. Ada (L40S, RTX 4090), Hopper and Blackwell GPUs all have FP8 tensor cores; Blackwell adds FP4.

Matching GPUs to workloads

The table below is rule-of-thumb guidance for single-GPU inference, not a benchmark. Every SKU in it is available on Reviosa GPU Cloud.

Why the large GPUs win for video

Once the job is compute-bound, dense tensor throughput sets the step time. Public spec figures for dense BF16 are roughly 360 TFLOPS on the L40S, roughly 990 on the H100 SXM and roughly 2,250 on the B200. The H200 has the same compute as the H100 but 141 GB of memory, which matters when you want to keep every component resident in BF16, generate longer clips, or batch the guidance pass. The B200 adds both: 180 GB and more than twice the H100's dense throughput.

For long or high-resolution clips, sequence parallelism (Ulysses or ring attention, as used by xDiT and several video model repos) splits one generation across GPUs. That works well over NVLink inside an 8x H100 node and poorly over PCIe, so RTX and L40S instances are best used one generation per GPU.

Batch versus latency

LLM decode gets huge wins from batching because it is bandwidth-bound. Diffusion mostly does not. A 1024px FLUX step already keeps a large GPU busy at batch size 1, so batching typically improves throughput modestly while multiplying latency. Smaller models such as SDXL on large GPUs are the exception, where batching pays.

The bigger levers are elsewhere:

• Steps. Cost scales linearly with step count. Moving from 50 steps to 28, or to 4-8 with a distilled or turbo variant, beats any hardware change.
• Guidance. Classifier-free guidance doubles the work per step. Guidance-distilled models skip the second pass.
• Compilation and attention kernels. torch.compile and FP8 attention typically trim step time further. Measure on your pipeline.

Compare GPUs on cost per output: cost per image equals the hourly rate divided by images per hour, and cost per video second equals the hourly rate divided by generated seconds per hour. Pull current rates from /pricing, run 50 generations at your production settings, and compute both.

Fine-tuning LoRAs

Training adds gradients, optimizer state for the adapter weights, and stored activations. Approximate VRAM ranges reported by common open-source trainers:

• SDXL LoRA: about 12-16 GB with gradient checkpointing. An RTX 4090 is plenty.
• FLUX.1 [dev] LoRA: possible in about 24 GB with a quantized base model, gradient checkpointing and cached text embeddings. An RTX 5090 or L40S gives headroom for 1024px and batch sizes above 1; an H100 shortens wall-clock time.
• 14B video model LoRA: 48 GB at low resolution and short clips at minimum; 80 GB or more is realistic. Use H100 or H200. Full fine-tunes belong on an 8x H100 node or B200s.

Three habits cut training cost regardless of GPU:

1. Precompute latents and text embeddings once. VAE encoding of video is expensive, and redoing it every epoch wastes most of the budget.
2. Bucket by aspect ratio and resolution so batches stay dense without padding.
3. Train low-resolution first. Learn the concept at 512px or 480p, then finish briefly at full resolution.

Keep datasets and cached latents on persistent block volumes or object storage, so a dev box and a training node can reuse them across runs. For the memory arithmetic behind training footprints, see Why VRAM, not FLOPS, limits your batch size.

FAQ

Is an RTX 4090 enough for FLUX image generation?

Yes, with an FP8 or otherwise quantized denoiser and the text encoder offloaded after encoding. If you want BF16 weights resident or larger batches, a 32 GB RTX 5090 or a 48 GB L40S is more comfortable.

How much VRAM do I need for AI video generation?

It depends on model size, resolution and clip length. Small 1-2B video models run at 480p in under 24 GB. 13-14B models at 720p need roughly 45-80 GB in practice, which means an H100 or larger unless you accept heavy offloading and slow generation.

Should I batch diffusion requests to save money?

Less than you might expect. Large diffusion models are compute-bound, so batching gives modest throughput gains while increasing latency. Reducing steps, using guidance-distilled models and compiling the pipeline usually save more.

Can I train a FLUX LoRA on an RTX 5090?

Yes. With 32 GB, a quantized base model, gradient checkpointing and cached text embeddings, FLUX LoRA training at 1024px fits with headroom. Video model LoRAs at 14B scale need 48-80 GB or more.