vLLM

Overview

vLLM serves LLMs with paged KV-cache memory (PagedAttention), continuous batching, prefix caching, chunked prefill, quantization and tensor/pipeline/data parallelism. vllm serve starts an OpenAI-compatible server (/v1/chat/completions, /v1/completions, /v1/embeddings, plus tokenization, health and Prometheus /metrics endpoints); the LLM class runs offline batch inference in Python. The engine is the V1 engine; older flags and environment variables from 2024-era guides may no longer exist, so check vllm serve --help for the installed version (0.30 at the time of writing).

Instructions

Install

uv venv --python 3.12 && source .venv/bin/activate
uv pip install vllm --torch-backend=auto     # picks the PyTorch/CUDA build for your driver
# plain pip alternative: pip install vllm

Use a fresh environment: the wheel pins its own PyTorch and CUDA kernels, and mixing in an existing PyTorch usually breaks. Blackwell GPUs need CUDA 12.8 or newer. Gated models (Llama, Gemma) need export HF_TOKEN=... and accepted licence terms on Hugging Face.

Serve a model

vllm serve Qwen/Qwen3-8B \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --api-key "$VLLM_API_KEY"

The model is a positional argument (--model also still works). Useful flags:

• --tensor-parallel-size 4 splits one model over 4 GPUs on one machine; --pipeline-parallel-size spans nodes; --data-parallel-size runs replicas.
• --max-model-len caps context length; lowering it is the first fix for out-of-memory at start-up because the KV cache is sized from it.
• --max-num-seqs caps concurrent sequences; --gpu-memory-utilization is the fraction of each GPU vLLM may use (default about 0.9).
• --served-model-name chat-prod sets the name clients must send as model.
• --quantization is normally unnecessary: pre-quantized checkpoints (for example Qwen/Qwen2.5-7B-Instruct-AWQ, GPTQ, FP8) are detected from their config. Passing awq for a non-AWQ model fails. Quantization needs a checkpoint that is already quantized (or an FP8 option vLLM applies on load); it does not quantize an arbitrary model for you.
• --enable-auto-tool-choice --tool-call-parser hermes turns on tool calling (pick the parser that matches the model family); --reasoning-parser is for reasoning models.
• Prefix caching is on by default in V1 for models that support it; --enable-prefix-caching / --no-enable-prefix-caching override.
• VLLM_API_KEY in the environment is equivalent to --api-key.

Docker

docker run --runtime nvidia --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 127.0.0.1:8000:8000 \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-8B --max-model-len 8192

Arguments after the image name are the same as for vllm serve. --ipc=host (or a large --shm-size) is required for multi-GPU. Pin a version tag such as vllm/vllm-openai:v0.11.0 instead of latest in production.

Call it with the OpenAI SDK

import os
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key=os.environ["VLLM_API_KEY"])

resp = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[{"role": "user", "content": "Write a Python function that returns the nth Fibonacci number."}],
    temperature=0.7,
    max_tokens=512,
)
print(resp.choices[0].message.content)

stream=True works as with OpenAI. List what the server exposes with curl -H "Authorization: Bearer $VLLM_API_KEY" localhost:8000/v1/models.

Embeddings

Serve an embedding model on its own server; a pooling model such as BAAI/bge-large-en-v1.5 is detected automatically (--runner pooling forces it):

vllm serve BAAI/bge-large-en-v1.5 --port 8001

emb = OpenAI(base_url="http://localhost:8001/v1", api_key="unused").embeddings.create(
    model="BAAI/bge-large-en-v1.5", input=["How do I reset my password?"])
print(len(emb.data[0].embedding))   # 1024

(If the server was started with --api-key, send that key instead of "unused".)

Offline batch inference

from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen3-8B", max_model_len=8192, gpu_memory_utilization=0.90)
params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=256)

outputs = llm.generate(["Summarize PagedAttention in two sentences.",
                        "Write a haiku about GPUs."], params)
for o in outputs:
    print(o.prompt, "->", o.outputs[0].text.strip())

# chat-formatted input applies the model's chat template for you
chat = llm.chat([{"role": "user", "content": "What is continuous batching?"}], params)
print(chat[0].outputs[0].text)

Examples

Example 1: Serve a 70B model on four GPUs

Request: "Host Llama 3.3 70B on our 4xA100 box so the team can use the OpenAI SDK."

export HF_TOKEN=...   # from your Hugging Face account, licence accepted
vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 4 --max-model-len 16384 --max-num-seqs 128 \
  --served-model-name llama-70b --api-key "$VLLM_API_KEY" --host 127.0.0.1

Weights download on first start, then the log prints the KV cache size and Application startup complete. A 70B bf16 model needs about 140 GB of GPU memory in total, so 4x80 GB fits with room for cache; on smaller cards use an FP8 or AWQ checkpoint. Clients use model="llama-70b". Put nginx or another gateway with TLS in front before exposing it.

Example 2: Fix an out-of-memory error at start-up

Request: "vllm crashes at startup on my 24 GB RTX 4090 with an out-of-memory error."

Lower the context and, if needed, use a quantized checkpoint: vllm serve Qwen/Qwen2.5-14B-Instruct-AWQ --max-model-len 8192 --gpu-memory-utilization 0.90. A 14B bf16 model (about 28 GB) cannot fit in 24 GB at all; the 4-bit version is about 9 GB, leaving the rest for KV cache. Check GPU use with nvidia-smi and make sure no other process holds memory.

Guidelines

• --api-key protects only the /v1-style inference paths; other endpoints on the same port (/invocations, for one, exposes the same capability as /v1) are not covered. Bind to 127.0.0.1 or put a reverse proxy in front; never publish port 8000 to the internet.
• --trust-remote-code runs code shipped with a model repository; use it only for repositories you trust.
• Do not run vLLM for a single low-traffic user on a laptop; Ollama or llama.cpp are simpler. vLLM pays off with concurrent requests on server GPUs.
• Throughput numbers in blog posts ("24x") depend on workload; benchmark your own prompts with vllm bench serve.
• Embedding and generation models are separate servers; one vllm serve process serves one model.
• Keep the vLLM version, PyTorch and CUDA driver in step; upgrading usually means recreating the environment.