Set Up vLLM on a Single GPU: From Drivers to an OpenAI-Compatible Server

A single NVIDIA GPU is enough to run a useful local LLM server, provided the model, precision, context length, and traffic fit its VRAM. vLLM is a strong choice when you want an OpenAI-compatible API rather than a desktop chat application. It handles continuous batching and paged attention, so one process can serve several clients efficiently.

This guide assumes Ubuntu or another mainstream Linux distribution, an NVIDIA GPU, and shell access. Commands may differ on other distributions. Do not install a random CUDA toolkit first and troubleshoot backward: verify the driver, Python, and current vLLM compatibility requirements before choosing package versions.

Before installing: match the model to the GPU

VRAM, not the marketing tier, is the first constraint. Model weights consume roughly parameter count multiplied by bytes per weight, before adding KV cache and runtime overhead. Quantization reduces weight memory, but does not make context or concurrency free.

GPU class	Typical VRAM	Sensible single-GPU starting point	Operator note
NVIDIA L4	24 GB	7B–8B models; some larger quantized models	Efficient and compact, but long contexts can consume the remaining VRAM quickly.
NVIDIA A10	24 GB	7B–8B models; some larger quantized models	Similar capacity planning to L4; benchmark your exact precision and prompt length.
NVIDIA A100	40 or 80 GB	14B–32B models, or smaller models with more cache	Capacity varies by SKU. Confirm VRAM with nvidia-smi, not the product name alone.
NVIDIA H100	80 GB in common PCIe/SXM variants	32B-class models or high-throughput smaller models	Fast, but a single card still has finite KV cache and can OOM under uncontrolled bursts.

These are planning ranges, not guarantees. Architecture, dtype, quantization format, maximum sequence length, multimodal inputs, and vLLM version all change the fit. Start conservatively and measure peak memory with realistic prompts.

The five-step installation

1. Install and verify the NVIDIA driver

Use the driver installation method recommended for your operating system or cloud image. Reboot if the installer requires it, then verify that the host can see the card:

nvidia-smi

The command should identify the GPU and report its driver version and total memory. If it fails, stop here. Python package changes cannot repair a missing kernel driver or an inaccessible GPU.

2. Confirm CUDA compatibility

For a standard pip install, vLLM wheels include compiled CUDA components, so a full system CUDA toolkit is not always necessary. You still need a compatible NVIDIA driver. Install the toolkit only when your chosen build path or extension requires it, following NVIDIA’s Linux guide rather than mixing repositories.

If you already have the compiler toolkit, inspect it with:

nvcc --version

nvidia-smi and nvcc report different things: the former reflects driver capability; the latter reflects an installed toolkit. Check the current vLLM installation matrix before pinning Python, PyTorch, or CUDA versions.

3. Create an isolated Python environment

python3 -m venv ~/.venvs/vllm
source ~/.venvs/vllm/bin/activate
python -m pip install --upgrade pip

An isolated environment prevents an unrelated PyTorch package from silently replacing vLLM’s expected dependency set.

4. Install vLLM

pip install vllm

Then verify imports and GPU visibility:

python -c "import torch, vllm; print(vllm.__version__); print(torch.cuda.is_available())"

The final value should be True. If installation fails, consult the current GPU installation page for the supported wheel path rather than forcing incompatible PyTorch packages into the environment.

5. Choose and authorize the model

Pick a Hugging Face model whose license you accept and whose weights fit the card with room for cache. For gated repositories, accept the model terms in your Hugging Face account and authenticate:

pip install -U huggingface_hub
hf auth login

You can let vLLM download weights on first launch. For predictable deployments, pre-download them into the persistent Hugging Face cache:

hf download Qwen/Qwen2.5-7B-Instruct

Replace that example with the exact repository you intend to serve. Review the model card for required chat templates, custom-code requirements, dtype, and license restrictions.

Launch the server with one command

The basic launch shape is:

vllm serve Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --port 8000 --dtype auto --max-model-len 8192 --gpu-memory-utilization 0.90

Keep --max-model-len explicit. Advertising a context window you do not need reserves less useful cache capacity and increases OOM risk. The memory-utilization setting is a ceiling target, not a guarantee that every request mix will fit.

Test the OpenAI-compatible endpoint locally:

curl http://127.0.0.1:8000/v1/models

Do not expose port 8000 directly to the public internet. Put authentication, TLS, request-size limits, timeouts, and rate limiting in a reverse proxy or API gateway.

Three startup gotchas

1. Model download time looks like a hang

A multi-gigabyte repository can take minutes or hours to download, especially when several weight shards are involved. Watch disk and network activity and ensure the Hugging Face cache is on persistent storage with enough free space. Container-local caches force another download after replacement.

2. Kernel compilation delays the first boot

Depending on the GPU, model, attention backend, and installed packages, the first run may compile or tune kernels. CPU usage and sparse logs can make this look stuck. Preserve relevant caches between deployments and do not let an orchestration health check kill the process before initialization finishes.

3. A listening port is not a warmed server

After weights load, the first real generation can still be slower while execution paths and caches warm. Send a small representative completion before putting the instance behind production traffic. A successful /v1/models response proves routing, but not steady-state generation latency.

Three production gotchas

1. Bursts can OOM a server that passed a single-request test

KV cache grows with active tokens. Concurrent long prompts and generations can exceed capacity even when model weights fit comfortably. Cap input and output tokens, enforce concurrency or queue limits, and load-test with the burst shape you expect—not only average traffic.

2. CPU swap buys capacity at a latency cost

vLLM’s --swap-space can provide CPU memory per GPU for supported scheduling scenarios, but moving data across PCIe is far slower than keeping it in VRAM. Treat swap as an emergency margin or deliberate trade-off, not equivalent GPU capacity. Watch tail latency and host RAM before increasing it.

3. VRAM fragmentation makes failures intermittent

Repeatedly loading models, sharing a GPU with other processes, and changing workload shapes can leave memory unavailable in useful contiguous blocks. Give the server exclusive GPU access where possible, monitor nvidia-smi, and restart the worker during a controlled drain if fragmentation accumulates. Do not run an unbounded notebook or display workload on the same production card.

A practical readiness checklist

Before declaring the endpoint ready, confirm that the driver survives reboot, caches are persistent, the model license is acceptable, and the service starts without an interactive login. Then warm it, run concurrent prompts at your maximum allowed context and output lengths, and record throughput, first-token latency, tail latency, and peak VRAM.

The safe single-GPU pattern is simple: choose a model below the card’s theoretical limit, leave room for KV cache, constrain context and concurrency, and treat startup warm-up as part of deployment. vLLM makes the API easy; capacity discipline keeps it alive.