22 Nov 2026 · gpu-sizing · vllm · kv-cache · a10g · l4

What fits on a 24 GB GPU: measured on an A10G and an L4

Qwen3-4B and Qwen3-8B served with vLLM on two 24 GB GPUs from different generations. Why the 8B model won't start at 32K, why FP8 weights help more than a smaller context, and why both GPUs held the same amount of context.

A 24 GB GPU is a common place to start self-hosting: an A10G or an L4 on AWS, or something similar at home. The question is always the same: which model, and how much context will it really hold?

I measured it instead of estimating it. Two GPUs from different generations, an A10G (Ampere, g5) and an L4 (Ada, g6), the same three Qwen3 models, vLLM 0.31, 32K context, --max-num-seqs 32. For each run I recorded the three numbers vLLM prints at startup: weight memory, KV cache memory and how many tokens that cache holds. The raw data and the script are in gpu-fit’s bench/ folder.

The budget is about 20.7 GiB, not 24

Both cards report about 22.5 GiB to the driver. vLLM takes 92% of that by default (--gpu-memory-utilization 0.92): about 20.7 GiB. Weights come out first, then a runtime overhead (activations, CUDA graphs, buffers), and the KV cache gets whatever is left. That remainder decides how long a request can be and how many run at once.

The results

Model KV cache KV tokens (A10G) KV tokens (L4) Full 32K requests at once
Qwen3-4B 16-bit 85,712 85,632 2.6
Qwen3-4B FP8 178,736 178,448 5.4–5.5
Qwen3-8B (16-bit weights) 16-bit won’t start won’t start 0
Qwen3-8B-FP8 16-bit 72,496 72,432 2.2
Qwen3-8B-FP8 FP8 158,240 158,080 4.8

Four things stand out.

1. Qwen3-8B in 16-bit won’t start at 32K

The 16-bit weights take 15.3 GiB. After the overhead, about 3.6 GiB is left for the KV cache: roughly 26,000 tokens. That’s less than one 32,768-token request, and vLLM checks this at startup and refuses to start. The model loads; the server doesn’t come up.

The obvious fix is a shorter context. At 16K it starts, with room for 1.6 requests at full length. That works, but it gives up half the context to save memory.

2. FP8 weights beat a smaller context

The official FP8 checkpoint cuts the weights from 15.3 to 8.9 GiB. The KV cache grows from 3.6 to about 10 GiB: 2.8 times as many tokens (72,496 against 26,256), and that’s at the full 32K context. Same model family, same card, and it now serves two long requests instead of none.

If a model doesn’t quite fit, reach for 8-bit weights before you cut the context. Check the quality on your own task first, but on capacity alone it isn’t close.

3. An FP8 KV cache slightly more than doubles capacity

Storing the cache in 8 bits halves the bytes per token, so you’d expect twice the tokens. vLLM reported 2.09× for the 4B model and 2.18× for the 8B: the KV cache was also a little larger with FP8 (12.27 against 11.77 GiB for the 4B). gpu-fit assumes exactly double, so its FP8 estimates are cautious here, by up to 9%.

The cost: an 8-bit cache can change output quality. Test it on your own prompts before you rely on it.

4. Two generations, the same capacity

The A10G and the L4 agree within 0.4% on KV tokens in every run, and the runtime overhead differed by at most 0.03 GiB. On the same software, memory size decides how much context fits, not the GPU generation. FP8 weights and the FP8 KV cache both ran on the Ampere A10G.

What this test doesn’t tell you is speed. I measured memory, not tokens per second, and the two cards differ there (the A10G has more memory bandwidth). That’s the next measurement.

A detail worth knowing

Qwen3-4B and Qwen3-8B need the same KV cache per token: 36 layers, 8 KV heads of 128 dimensions, 144 KiB a token in 16-bit. The 4B model holds more context mainly because its weights are smaller (7.6 against 15.3 GiB), and its runtime overhead is a little lower too. A bigger model in the same family doesn’t always cost more per token; it costs more weights, and that comes straight out of the context.

Takeaways

You can run these cases in the gpu-fit calculator or locally:

pip install gpu-fit
gpu-fit Qwen/Qwen3-8B-FP8 --gpu L4 --context 32k --kv-cache-dtype fp8 --max-num-seqs 32

Keep --max-num-seqs the same as the server’s: more request slots reserve more runtime memory, which leaves less for the KV cache.

Measured 20 Nov 2026 on AWS g5 and g6 instances with vLLM 0.31.0. Results can change with other vLLM versions; the accuracy table says what has been measured.

← All notes