19 Nov 2026 · gpu-sizing · vllm · kv-cache · l40s
A 32B model on a 48 GB GPU: what actually fits
Qwen3-32B on one NVIDIA L40S: why 16-bit weights don’t fit, why 8-bit weights still serve less than one user at 32K, and the two settings that change everything.
“32 billion parameters, 48 GB of GPU memory. Should be fine.”
It isn’t, and the reason is worth five minutes, because the same arithmetic decides every self-hosting budget. Here’s Qwen3-32B on a single NVIDIA L40S, served with vLLM, worked through step by step. Every number links to a live calculation you can change.
The budget is smaller than the card
An L40S has 48 GB, which nvidia-smi reports as 45.0 GiB. With vLLM’s default budget
(gpu_memory_utilization 0.92, vLLM 0.29 to 0.33), that’s 41.4 GiB. Out of that come three things, in this order:
- Weights. Fixed by the model and its precision.
- Runtime overhead. Activations, CUDA graphs, buffers. About 2 GiB for a text model.
- KV cache. Whatever is left. This decides how long a request can be and how many run at once.
16-bit weights: doesn’t fit at all
At 2 bytes per parameter, Qwen3-32B’s weights are 61.0 GiB. The budget is 41.4 GiB. That’s 19.6 GiB short before a single token of context. No vLLM setting makes it fit on one card: you need more GPU memory, two GPUs or fewer bits (offloading weights to the CPU works, but much more slowly). See it.
8-bit weights: it loads… and serves less than one user
The official FP8 checkpoint cuts the weights to about 32.0 GiB. On one L40S:
- Budget: 41.4 GiB
- Weights: 32.0 GiB
- Runtime overhead: 2.1 GiB
- Left for KV cache: 7.4 GiB
Now the part that decides how many users fit. Every token in every active request needs KV cache. For this model that’s 256 KiB per token:
2 (key + value) × 64 layers × 8 KV heads × 128 dims × 2 bytes = 262,144 bytes
7.4 GiB ÷ 256 KiB ≈ 30,000 tokens in total, shared by everyone. One request at 32K context
doesn’t fit: 0.9 requests. vLLM checks this at startup and refuses to start with
--max-model-len 32768; on one card you’d have to cap requests at about 30K tokens.
See it.
Two settings that change the answer
1. An 8-bit KV cache doubles the KV capacity. --kv-cache-dtype fp8 stores keys and values in one byte
instead of two: 60,000 tokens, 1.8 requests at 32K. It can affect output quality, so
check it on your own workload.
2. A second GPU does more than double it. Split across two L40S
(--tensor-parallel-size 2), the weights drop to 16 GiB per card and each card holds half the
KV heads. Result: 191,000 tokens, 5.8 requests at 32K. Two cards give you six times the
capacity of one, because the fixed weight cost no longer eats most of the memory.
The context ceiling isn’t only memory
Qwen3-32B’s native window is 40,960 tokens. Going to 128K needs RoPE scaling (YaRN) and the memory to hold it. On one L40S with an FP8 KV cache, memory would allow about 60,000 tokens for a single request, so the model’s native 40,960-token window is the limit you reach first.
The surprise: a hybrid model on the same card
Newer “hybrid” models mix full attention with linear attention layers that keep a fixed-size state instead of a growing cache. Qwen3.8-27B-FP8 has 64 layers, but only 16 grow with context. Same L40S, and it holds about 134,000 tokens: four times the 32B’s capacity.
But start it with vLLM’s defaults and vLLM refuses to start. Each request also needs one
recurrent-state slot, and the default 256 request slots (--max-num-seqs) is more than the
roughly 160 this card can hold. The fix is one flag, --max-num-seqs 32, and it runs.
See it.
What to take away
- Subtract before you divide. Weights and overhead come off the top; the KV cache is the remainder.
- Count KV per token, not parameters. It depends on KV heads and layers, and that’s what limits users.
- FP8 weights make it load; an FP8 KV cache roughly doubles the context that fits.
- Two GPUs can hold about six times the context of one. Once weights stop dominating, capacity jumps.
- This hybrid model needs
--max-num-seqsset, or it won’t start.
How these numbers were made
With gpu-fit, a small open-source tool that reads a model’s real architecture from Hugging Face and mirrors vLLM’s memory budget. Weights and KV per token come straight from the model’s config. Runtime overhead is calibrated against real vLLM runs; the method and raw data are in the repository. Treat the output as a planning estimate and confirm on your own hardware before you spend money.
Try your own model and GPU in the browser calculator, or run
pip install gpu-fit.
Corrected 20 Nov 2026: vLLM’s default gpu_memory_utilization is 0.92, not 0.9, so the budget is
41.4 GiB and the runtime overhead about 2.1 GiB. The 32B token and request figures were already right
and match the calculator links (8 requests at once); the hybrid example uses 32. An earlier version
said the FP8 model would load at 32K; vLLM refuses to start when one full request doesn’t fit, and
the text now says so.