From notebook to production: vLLM endpoints on Reviosa

The gap is smaller than you think

The prototype works: a notebook, a model, some prompts, good outputs. Now it needs to be an endpoint your application can call — with real latency targets, restarts that don't page anyone, and a bill someone signed off on. Teams stall here for weeks because "production" sounds like a platform project. With vLLM's OpenAI-compatible server it is closer to an afternoon. Here is the path we walk customers through.

Step 1: Size the instance to the model

• ≤8B, quantized (AWQ/GPTQ): an L40S at $0.69/hr. 48 GB leaves ~40 GB of KV cache after weights — plenty for real concurrency.
• 70B-class, 4-bit: a single H100 at $2.49/hr fits the ~39 GB of weights with a workable cache budget.
• 70B at FP8 or above, or strict latency targets: 2× H100 with tensor parallelism, or an 8×H100 node at $18.32/hr for high-throughput serving.

Start smaller than your instinct says. Continuous batching makes one modest GPU go much further than notebook-style decoding suggests, and resizing later is a five-minute operation, not a migration.

Step 2: Launch the server

Launch an instance from the dashboard or API, then run the official container:

docker run --gpus all -p 8000:8000   -e HF_TOKEN=...   -v /mnt/models:/root/.cache/huggingface   vllm/vllm-openai:latest   --model meta-llama/Meta-Llama-3-8B-Instruct   --quantization awq_marlin   --max-model-len 8192   --max-num-seqs 64

Mounting the cache on the instance's local NVMe means restarts skip the model download. From here your application code barely changes: point any OpenAI client's base_url at the instance and keep the same chat-completions calls you prototyped with. That API compatibility is the whole trick — the notebook code is the production code.

Step 3: Watch the right metrics

vLLM exposes Prometheus metrics at /metrics, and three of them tell you nearly everything:

1. vllm:num_requests_waiting — sustained queueing means you need another replica or a smaller max-model-len.
2. vllm:gpu_cache_usage_perc — the KV cache is your real capacity gauge; sustained readings near 100% precede preemptions and latency spikes.
3. Time-to-first-token p95 — the metric users actually feel. Alert on it, not on averages.

Wire these into whatever you already run — a scrape config and one dashboard is enough to start.

Step 4: Load-test before your users do

Replay production-shaped traffic — your real prompt-length and output-length distributions, not synthetic 100-token pairs — at 1×, 2×, and 5× expected peak. You are looking for the concurrency where TTFT degrades and where the cache saturates. On the L40S configuration above, Llama 3 8B AWQ sustains ~400 tok/s aggregate around concurrency 32 while holding p95 TTFT under 300 ms; your numbers will differ with your traffic, which is exactly why you measure. This is also where per-second billing quietly shines: a 40-minute load test on that L40S costs $0.46.

Step 5: Production hygiene

• Put a reverse proxy with API-key auth in front — the vLLM port should never face the internet bare.
• Pin the model revision and the container tag so a redeploy serves the same weights you evaluated.
• Run a staging replica on the same instance type; with per-second billing, spinning it up only for deploy checks costs pennies.
• Script the launch (API or Terraform) so replica two is a variable change, not an archaeology project.

What it costs

One L40S replica around the clock is ~$504/month. At the measured 400 tok/s aggregate, that is 1.44M output tokens per hour — about $0.48 per million output tokens at full load, with your data never leaving your instance. Scale the replica count with traffic and the economics only improve. The afternoon you spend on this pays back the first time traffic spikes and the endpoint just... holds.