Cutting inference costs 60%: quantization, batching, right-sizing

Where inference money actually goes

When we audit customer inference deployments, the same three problems appear again and again: serving in BF16 when quality-neutral quantization exists, decoding one request at a time on hardware built for thousands, and renting 80 GB of HBM for a model that needs six. Each fix compounds with the others. This post walks through all three using a real (anonymized) deployment — Llama 3 8B behind a support chatbot at Quill & Query — that started at two on-demand H100s and $3,635/month.

Step 1: Quantize the weights

Weight-only 4-bit quantization (AWQ or GPTQ) takes Llama 3 8B from ~16 GB to ~5.8 GB. On Hopper, FP8 W8A8 is another strong option. In our evals — and in most published results — AWQ on 8B and 70B models costs under one point on standard benchmarks, and often nothing measurable on task-specific evals. Run your own eval suite before and after; if quality holds, you have just tripled your options.

The direct win is that the model now fits on far cheaper GPUs. The indirect win is bigger: every gigabyte freed from weights becomes KV cache, and KV cache is what lets you batch.

Step 2: Turn on continuous batching

A single decode stream uses a few percent of a modern GPU. vLLM's continuous batching admits and retires requests at token granularity, so the GPU stays saturated even with ragged request lengths. On an L40S serving Llama 3 8B AWQ, we measured ~48 tok/s for a single stream but over 400 tok/s aggregate at concurrency 32, with p95 time-to-first-token still under 300 ms. That is an 8× throughput multiplier from a config change.

vllm serve meta-llama/Meta-Llama-3-8B-Instruct   --quantization awq_marlin   --max-model-len 8192   --max-num-seqs 64   --gpu-memory-utilization 0.92

Set --max-model-len to what your traffic actually needs — every token of headroom you don't use is KV cache you can't allocate to another request.

Step 3: Right-size the hardware

With a quantized model and real batching numbers, match the GPU to the job instead of defaulting to the biggest one:

The L40S has no NVLink and modest bandwidth next to an H100 — and for an 8B model at moderate QPS, it does not matter. You are paying 72% less per replica.

Putting it together

The before and after for the deployment above:

• Before: 2× H100 on demand, BF16, naive batching → $3,635/mo (2 × $2.49 × 730 hrs).
• After: 3× L40S, AWQ, vLLM continuous batching, same peak QPS with headroom → $1,511/mo.
• Then: per-second billing let them drop to one replica overnight (roughly 12 hours at low traffic), bringing the effective bill to about $1,260/mo — a 65% reduction.

Latency got better, not worse: p95 end-to-end went from 2.4 s to 1.9 s, because continuous batching eliminated queueing behind long generations.

Measure, then cut

The order matters. Quantize first and verify quality with your own evals. Batch second and find your real throughput ceiling under production-shaped load. Only then choose hardware — because until you know your quantized, batched throughput per replica, you are sizing blind. Every deployment we have run through this sequence has landed between 55% and 75% savings, and none has given back quality to get there.