L40S vs H100 vs RTX 5090: inference cost per million tokens

Why cost per million tokens is the number that matters

Hourly price tells you what a GPU costs to hold. Cost per million tokens tells you what it costs to do the job. A card that costs half as much per hour but delivers a third of the throughput is a worse deal, and the reverse happens just as often. This post is a method, not a leaderboard: how to compute the number, what drives it on the L40S, H100 and RTX 5090 for 8B to 70B class models, and how to measure it on your own traffic in about an hour.

We deliberately do not quote dollar figures here. Rates change, and the current ones are published at /pricing and are identical in every region. Plug today's hourly rate into the formula below and you get a number that is true today, rather than one that was true when this post was written.

The formula

Cost per million tokens is price divided by throughput, with the units pinned down:

def cost_per_million_tokens(price_per_hour, tokens_per_second):
    tokens_per_hour = tokens_per_second * 3600
    return price_per_hour / tokens_per_hour * 1_000_000

# Shortcut: price_per_hour / (tokens_per_second * 0.0036)

Three details decide whether the result is honest:

1. Use aggregate throughput, not single-stream speed. A server running 48 concurrent requests at 40 tok/s each produces 1,920 tok/s. That aggregate is what you are paying for.
2. Count the tokens you care about. Prefill (input) tokens are cheap to process; decode (output) tokens are expensive. Report them separately, or report output tokens at a fixed input-to-output ratio that matches production, such as 4:1 for retrieval-augmented generation or 1:1 for chat.
3. Measure at the concurrency you will actually run while meeting your latency target. Throughput at concurrency 256 is meaningless if p95 time-to-first-token (TTFT) breaks your SLO at 64.

What drives the number

Memory bandwidth sets the decode ceiling

Decode is bandwidth-bound: every step streams the model weights and the active KV cache through memory. The public spec sheets are the starting point:

A useful upper bound for single-stream decode is bandwidth divided by the bytes read per step. An 8B model in FP8 is about 8 GB of weights, so the theoretical ceiling is roughly 100 tok/s per stream on the L40S, 220 on the RTX 5090 and 400 on the H100. Real numbers land below these bounds because of kernel overhead and KV reads, but the ratios between cards hold up well.

VRAM fit sets the batch, and the batch sets throughput

Batching amortizes each weight read across every sequence in flight, so aggregate throughput climbs with concurrency until the KV cache runs out of room. That makes VRAM left over after weights the second big term. With Llama 3-style grouped-query attention, an 8B model needs about 128 KB of KV cache per token in BF16 and a 70B needs about 320 KB. The full arithmetic is in Why VRAM, not FLOPS, limits your batch size.

For an 8B model in FP8 serving 2,048-token sequences (about 256 MB of BF16 KV cache each):

• RTX 5090: 32 GB minus 8 GB of weights minus ~3 GB of runtime overhead leaves ~21 GB, or roughly 80 concurrent sequences.
• L40S: ~37 GB of cache, or roughly 145 sequences.
• H100 SXM: ~68 GB of cache, or roughly 265 sequences.

Note the inversion: the L40S can hold more requests in flight than the RTX 5090, but each step is slower. Which one wins depends on whether your traffic is latency-bound or throughput-bound.

Quantization moves both terms at once

Going from BF16 to FP8 halves the weight bytes, which raises the bandwidth ceiling and frees cache. 4-bit weight-only formats (AWQ, GPTQ) roughly quarter them. For larger models, quantization decides whether the model fits at all:

"Tight" means the model loads but leaves too little KV cache for useful concurrency. A 4-bit 70B on an L40S leaves well under 10 GB for cache, which makes it a demo rather than a serving configuration.

Always validate quality on your own eval set after quantizing. FP8 is close to lossless for most modern models and 4-bit usually holds up too, but domain-specific regressions happen and are cheap to catch before launch.

Batch size and latency are the same dial

Raising the maximum number of concurrent sequences raises throughput and lowers cost per token until latency breaks. The right operating point is the highest concurrency that keeps p95 TTFT and inter-token latency inside your SLO. Two deployments of the same model on the same GPU can differ by 2-3x in cost per token purely because one team tuned this dial and the other left it at a default.

A worked example with illustrative numbers

The table shows how the method plays out. The throughput ranges are illustrative order-of-magnitude estimates derived from the bandwidth and fit arithmetic above. They are not Reviosa lab measurements. They assume vLLM with continuous batching, roughly 1,000 input and 250 output tokens per request, and concurrency chosen to keep p95 TTFT under about one second.

To make the arithmetic concrete without inventing a price: at 2,000 tok/s an instance produces 7.2 million output tokens per hour, so cost per million is the hourly rate divided by 7.2. At 1,000 tok/s divide by 3.6; at 4,000 tok/s divide by 14.4. With current rates from /pricing, the usual pattern looks like this:

• 8B class: the smaller cards tend to win on cost per token when the latency target is relaxed, because their hourly rate falls faster than their throughput. The H100 wins when you need low latency at high concurrency, or when one H100 replaces several small replicas and the overhead that comes with them.
• 32B class: the RTX 5090 at 4-bit and the L40S at 4-bit or FP8 are both viable. Check KV headroom at your real context length before committing.
• 70B class: a single H100, or an H200 for long context, is the practical floor. Splitting a 70B across smaller cards with tensor parallelism adds communication cost, because those instances talk over PCIe and Ethernet rather than NVLink.

Measure it yourself

An hour of measurement beats any table, including ours. Every SKU in the comparison is available on demand on Reviosa GPU Cloud, and per-second billing means a 20-minute sweep costs a fraction of one hourly rate.

1. Launch the server with explicit settings

vllm serve meta-llama/Llama-3.1-8B-Instruct --quantization fp8 --max-model-len 8192 --max-num-seqs 128 --gpu-memory-utilization 0.90

Pin the model revision and the vLLM version so reruns and cross-GPU comparisons stay apples to apples.

2. Sweep concurrency with a load generator

vLLM ships a serving benchmark that speaks the OpenAI API. Any load generator that records per-request timings works, but this one reports throughput and latency percentiles directly:

for c in 8 16 32 64 128; do
  vllm bench serve --backend openai-chat --base-url http://localhost:8000 --endpoint /v1/chat/completions --model meta-llama/Llama-3.1-8B-Instruct --dataset-name random --random-input-len 1000 --random-output-len 250 --num-prompts 600 --max-concurrency $c --percentile-metrics ttft,itl --metric-percentiles 95,99 --save-result --result-filename bench-c$c.json
done

Better still, replay a sample of real prompts. Random synthetic tokens understate prefix-cache hits and misrepresent the output-length distribution, which is where most of the cost lives.

3. Convert to cost and pick the operating point

import glob, json

RATE = float(input("Hourly rate from /pricing: "))
TTFT_SLO_MS = 1000

for path in sorted(glob.glob("bench-c*.json"), key=lambda p: int(p[7:-5])):
    r = json.load(open(path))
    tps = r["output_throughput"]
    ttft = r["p95_ttft_ms"]
    cost = RATE / (tps * 0.0036)
    flag = "ok" if ttft <= TTFT_SLO_MS else "breaks SLO"
    print(f"{path}: {tps:8.0f} tok/s  p95 TTFT {ttft:6.0f} ms  {cost:.4f} per 1M tokens  {flag}")

Choose the row with the lowest cost per million tokens that stays inside the SLO, then repeat on each candidate GPU.

4. Account for utilization

Benchmarks assume the GPU is busy. A replica that averages 30% load has an effective cost per token about 3.3 times the benchmark figure. Either autoscale replicas to follow traffic, or, for spiky and low-volume traffic, use serverless endpoints on Reviosa AI, which bill per request-second and scale to zero.

FAQ

How do I calculate GPU inference cost per million tokens?

Divide the hourly instance price by tokens per hour and multiply by one million: price ÷ (tokens per second × 3,600) × 1,000,000. Use aggregate throughput at your production concurrency and count output tokens separately from input tokens.

Is the RTX 5090 or the L40S cheaper for 8B model inference?

It depends on your latency target and current rates. The RTX 5090 has roughly twice the memory bandwidth of the L40S, so each stream decodes faster; the L40S has 16 GB more VRAM, so it holds more concurrent requests. Benchmark both at your SLO and divide by the rates on /pricing.

Can I serve a 70B model on a single RTX 5090 or L40S?

Not on an RTX 5090: even 4-bit weights (~38 GB) exceed its 32 GB. A 4-bit 70B loads on a 48 GB L40S but leaves little room for KV cache, so concurrency stays low. A single H100 or H200 is the practical floor for production 70B serving.

Why not just use the throughput numbers in this post?

They are illustrative estimates that demonstrate the method, not measurements. Prompt length, output length, quantization recipe and engine version can each move throughput by 2x or more, so an hour of measurement on your own traffic is worth more than any published table.