<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://kirantallapragada.com/feed.xml" rel="self" type="application/atom+xml" /><link href="https://kirantallapragada.com/" rel="alternate" type="text/html" hreflang="en" /><updated>2026-11-22T17:17:08+00:00</updated><id>https://kirantallapragada.com/feed.xml</id><title type="html">Kiran Tallapragada</title><subtitle>Systems where accuracy matters: self-hosted LLMs on H100, L40 and L40S, document AI for regulated industries, market-making bots in Go and Rust, smart contracts in Solidity.</subtitle><author><name>Kiran Tallapragada</name></author><entry><title type="html">What fits on a 24 GB GPU: measured on an A10G and an L4</title><link href="https://kirantallapragada.com/2026/11/what-fits-on-a-24gb-gpu/" rel="alternate" type="text/html" title="What fits on a 24 GB GPU: measured on an A10G and an L4" /><published>2026-11-22T00:00:00+00:00</published><updated>2026-11-22T00:00:00+00:00</updated><id>https://kirantallapragada.com/2026/11/what-fits-on-a-24gb-gpu</id><content type="html" xml:base="https://kirantallapragada.com/2026/11/what-fits-on-a-24gb-gpu/"><![CDATA[<p>A 24 GB GPU is a common place to start self-hosting: an A10G or an L4 on AWS, or something
similar at home. The question is always the same: <em>which model, and how much context will it
really hold?</em></p>

<p>I measured it instead of estimating it. Two GPUs from different generations, an <strong>A10G</strong>
(Ampere, g5) and an <strong>L4</strong> (Ada, g6), the same three Qwen3 models, vLLM 0.31, 32K context,
<code>--max-num-seqs 32</code>. For each run I recorded the three numbers vLLM prints at startup: weight
memory, KV cache memory and how many tokens that cache holds. The raw data and the script are in
<a href="https://github.com/KiranTallapragada/gpu-fit/tree/main/bench">gpu-fit’s <code>bench/</code> folder</a>.</p>

<h2 id="the-budget-is-about-207-gib-not-24">The budget is about 20.7 GiB, not 24</h2>

<p>Both cards report about 22.5 GiB to the driver. vLLM takes 92% of that by default
(<code>--gpu-memory-utilization 0.92</code>): <strong>about 20.7 GiB</strong>. Weights come out first, then a runtime
overhead (activations, CUDA graphs, buffers), and the <strong>KV cache gets whatever is left</strong>. That
remainder decides how long a request can be and how many run at once.</p>

<h2 id="the-results">The results</h2>

<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>KV cache</th>
      <th>KV tokens (A10G)</th>
      <th>KV tokens (L4)</th>
      <th>Full 32K requests at once</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Qwen3-4B</td>
      <td>16-bit</td>
      <td>85,712</td>
      <td>85,632</td>
      <td>2.6</td>
    </tr>
    <tr>
      <td>Qwen3-4B</td>
      <td>FP8</td>
      <td>178,736</td>
      <td>178,448</td>
      <td>5.4–5.5</td>
    </tr>
    <tr>
      <td>Qwen3-8B (16-bit weights)</td>
      <td>16-bit</td>
      <td>won’t start</td>
      <td>won’t start</td>
      <td>0</td>
    </tr>
    <tr>
      <td>Qwen3-8B-FP8</td>
      <td>16-bit</td>
      <td>72,496</td>
      <td>72,432</td>
      <td>2.2</td>
    </tr>
    <tr>
      <td>Qwen3-8B-FP8</td>
      <td>FP8</td>
      <td>158,240</td>
      <td>158,080</td>
      <td>4.8</td>
    </tr>
  </tbody>
</table>

<p>Four things stand out.</p>

<h2 id="1-qwen3-8b-in-16-bit-wont-start-at-32k">1. Qwen3-8B in 16-bit won’t start at 32K</h2>

<p>The 16-bit weights take <strong>15.3 GiB</strong>. After the overhead, about <strong>3.6 GiB</strong> is left for the KV
cache: roughly 26,000 tokens. That’s less than one 32,768-token request, and vLLM checks this at
startup and refuses to start. The model loads; the server doesn’t come up.</p>

<p>The obvious fix is a shorter context. At 16K it starts, with room for <strong>1.6 requests</strong> at full
length. That works, but it gives up half the context to save memory.</p>

<h2 id="2-fp8-weights-beat-a-smaller-context">2. FP8 weights beat a smaller context</h2>

<p>The official FP8 checkpoint cuts the weights from 15.3 to <strong>8.9 GiB</strong>. The KV cache grows from
3.6 to about <strong>10 GiB</strong>: <strong>2.8 times as many tokens</strong> (72,496 against 26,256), and that’s at
the full 32K context. Same model family, same card, and it now serves two long requests
instead of none.</p>

<p>If a model doesn’t quite fit, reach for 8-bit weights before you cut the context. Check the
quality on your own task first, but on capacity alone it isn’t close.</p>

<h2 id="3-an-fp8-kv-cache-slightly-more-than-doubles-capacity">3. An FP8 KV cache slightly more than doubles capacity</h2>

<p>Storing the cache in 8 bits halves the bytes per token, so you’d expect twice the tokens. vLLM
reported <strong>2.09×</strong> for the 4B model and <strong>2.18×</strong> for the 8B: the KV cache was also a little
larger with FP8 (12.27 against 11.77 GiB for the 4B). gpu-fit assumes exactly double, so its
FP8 estimates are cautious here, by up to 9%.</p>

<p>The cost: an 8-bit cache can change output quality. Test it on your own prompts before you
rely on it.</p>

<h2 id="4-two-generations-the-same-capacity">4. Two generations, the same capacity</h2>

<p>The A10G and the L4 agree <strong>within 0.4%</strong> on KV tokens in every run, and the runtime overhead
differed by at most 0.03 GiB. On the
same software, memory size decides how much context fits, not the GPU generation. FP8 weights
and the FP8 KV cache both ran on the Ampere A10G.</p>

<p>What this test doesn’t tell you is <strong>speed</strong>. I measured memory, not tokens per second, and the
two cards differ there (the A10G has more memory bandwidth). That’s the next measurement.</p>

<h2 id="a-detail-worth-knowing">A detail worth knowing</h2>

<p>Qwen3-4B and Qwen3-8B need <strong>the same KV cache per token</strong>: 36 layers, 8 KV heads of 128
dimensions, 144 KiB a token in 16-bit. The 4B model holds more context mainly because its weights
are smaller (7.6 against 15.3 GiB), and its runtime overhead is a little lower too. A bigger model
in the same family doesn’t always cost more per token; it costs more weights, and that comes
straight out of the context.</p>

<h2 id="takeaways">Takeaways</h2>

<ul>
  <li>On a 24 GB card, the budget is about <strong>20.7 GiB</strong>; plan from that, not from 24.</li>
  <li><strong>Qwen3-8B in 16-bit won’t start at 32K.</strong> Use the FP8 checkpoint (2.8× the context) before
you cut the context in half.</li>
  <li>An <strong>FP8 KV cache roughly doubles</strong> the context that fits; check quality on your workload.</li>
  <li><strong>A10G or L4:</strong> the same context fits, so memory won’t decide between them. Speed and price
are a separate comparison, and the next thing I’ll measure.</li>
</ul>

<p>You can run these cases in the <a href="/gpu-fit/">gpu-fit calculator</a> or locally:</p>

<pre><code>pip install gpu-fit
gpu-fit Qwen/Qwen3-8B-FP8 --gpu L4 --context 32k --kv-cache-dtype fp8 --max-num-seqs 32
</code></pre>

<p>Keep <code>--max-num-seqs</code> the same as the server’s: more request slots reserve more runtime memory,
which leaves less for the KV cache.</p>

<p><em>Measured 20 Nov 2026 on AWS g5 and g6 instances with vLLM 0.31.0. Results can change with
other vLLM versions; the <a href="https://github.com/KiranTallapragada/gpu-fit#accuracy-measured">accuracy table</a>
says what has been measured.</em></p>]]></content><author><name>Kiran Tallapragada</name></author><category term="gpu-sizing" /><category term="vllm" /><category term="kv-cache" /><category term="a10g" /><category term="l4" /><summary type="html"><![CDATA[Qwen3-4B and Qwen3-8B served with vLLM on two 24 GB GPUs from different generations. Why the 8B model won't start at 32K, why FP8 weights help more than a smaller context, and why both GPUs held the same amount of context.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://kirantallapragada.com/assets/img/og.png" /><media:content medium="image" url="https://kirantallapragada.com/assets/img/og.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">A 32B model on a 48 GB GPU: what actually fits</title><link href="https://kirantallapragada.com/2026/11/a-32b-model-on-a-48gb-gpu/" rel="alternate" type="text/html" title="A 32B model on a 48 GB GPU: what actually fits" /><published>2026-11-19T00:00:00+00:00</published><updated>2026-11-19T00:00:00+00:00</updated><id>https://kirantallapragada.com/2026/11/a-32b-model-on-a-48gb-gpu</id><content type="html" xml:base="https://kirantallapragada.com/2026/11/a-32b-model-on-a-48gb-gpu/"><![CDATA[<p>“32 billion parameters, 48 GB of GPU memory. Should be fine.”</p>

<p>It isn’t, and the reason is worth five minutes, because the same arithmetic decides every
self-hosting budget. Here’s Qwen3-32B on a single NVIDIA L40S, served with vLLM, worked
through step by step. Every number links to a live calculation you can change.</p>

<h2 id="the-budget-is-smaller-than-the-card">The budget is smaller than the card</h2>

<p>An L40S has 48 GB, which <code>nvidia-smi</code> reports as <strong>45.0 GiB</strong>. With vLLM’s default budget
(<code>gpu_memory_utilization 0.92</code>, vLLM 0.29 to 0.33), that’s <strong>41.4 GiB</strong>. Out of that come three things, in this order:</p>

<ol>
  <li><strong>Weights.</strong> Fixed by the model and its precision.</li>
  <li><strong>Runtime overhead.</strong> Activations, CUDA graphs, buffers. About 2 GiB for a text model.</li>
  <li><strong>KV cache.</strong> Whatever is left. This decides how long a request can be and how many run at once.</li>
</ol>

<h2 id="16-bit-weights-doesnt-fit-at-all">16-bit weights: doesn’t fit at all</h2>

<p>At 2 bytes per parameter, Qwen3-32B’s weights are <strong>61.0 GiB</strong>. The budget is 41.4 GiB. That’s
<strong>19.6 GiB short</strong> before a single token of context. No vLLM setting makes it fit on one card:
you need more GPU memory, two GPUs or fewer bits (offloading weights to the CPU works, but much
more slowly). <a href="/gpu-fit/?model=Qwen/Qwen3-32B&amp;gpu=L40S">See it</a>.</p>

<h2 id="8-bit-weights-it-loads-and-serves-less-than-one-user">8-bit weights: it loads… and serves less than one user</h2>

<p>The official FP8 checkpoint cuts the weights to about <strong>32.0 GiB</strong>. On one L40S:</p>

<ul>
  <li>Budget: 41.4 GiB</li>
  <li>Weights: 32.0 GiB</li>
  <li>Runtime overhead: 2.1 GiB</li>
  <li><strong>Left for KV cache: 7.4 GiB</strong></li>
</ul>

<p>Now the part that decides how many users fit. Every token in every active request needs KV cache. For this
model that’s <strong>256 KiB per token</strong>:</p>

<pre><code>2 (key + value) × 64 layers × 8 KV heads × 128 dims × 2 bytes = 262,144 bytes
</code></pre>

<p>7.4 GiB ÷ 256 KiB ≈ <strong>30,000 tokens in total, shared by everyone</strong>. One request at 32K context
doesn’t fit: <strong>0.9 requests</strong>. vLLM checks this at startup and refuses to start with
<code>--max-model-len 32768</code>; on one card you’d have to cap requests at about 30K tokens.
<a href="/gpu-fit/?model=Qwen/Qwen3-32B-FP8&amp;gpu=L40S">See it</a>.</p>

<h2 id="two-settings-that-change-the-answer">Two settings that change the answer</h2>

<p><strong>1. An 8-bit KV cache doubles the KV capacity.</strong> <code>--kv-cache-dtype fp8</code> stores keys and values in one byte
instead of two: <strong>60,000 tokens, 1.8 requests at 32K</strong>. It can affect output quality, so
check it on your own workload.</p>

<p><strong>2. A second GPU does more than double it.</strong> Split across two L40S
(<code>--tensor-parallel-size 2</code>), the weights drop to 16 GiB per card and each card holds half the
KV heads. Result: <strong>191,000 tokens, 5.8 requests at 32K</strong>. Two cards give you six times the
capacity of one, because the fixed weight cost no longer eats most of the memory.</p>

<h2 id="the-context-ceiling-isnt-only-memory">The context ceiling isn’t only memory</h2>

<p>Qwen3-32B’s native window is <strong>40,960 tokens</strong>. Going to 128K needs RoPE scaling (YaRN) <em>and</em>
the memory to hold it. On one L40S with an FP8 KV cache, memory would allow about 60,000 tokens
for a single request, so the model’s native 40,960-token window is the limit you reach first.</p>

<h2 id="the-surprise-a-hybrid-model-on-the-same-card">The surprise: a hybrid model on the same card</h2>

<p>Newer “hybrid” models mix full attention with linear attention layers that keep a fixed-size
state instead of a growing cache. Qwen3.8-27B-FP8 has 64 layers, but <strong>only 16 grow with
context</strong>. Same L40S, and it holds <strong>about 134,000 tokens</strong>: four times the 32B’s capacity.</p>

<p>But start it with vLLM’s defaults and <strong>vLLM refuses to start</strong>. Each request also needs one
recurrent-state slot, and the default 256 request slots (<code>--max-num-seqs</code>) is more than the
roughly 160 this card can hold. The fix is one flag, <code>--max-num-seqs 32</code>, and it runs.
<a href="/gpu-fit/?model=Qwen/Qwen3.8-27B-FP8&amp;gpu=L40S">See it</a>.</p>

<h2 id="what-to-take-away">What to take away</h2>

<ul>
  <li><strong>Subtract before you divide.</strong> Weights and overhead come off the top; the KV cache is the remainder.</li>
  <li><strong>Count KV per token, not parameters.</strong> It depends on KV heads and layers, and that’s what limits users.</li>
  <li><strong>FP8 weights make it load; an FP8 KV cache roughly doubles the context that fits.</strong></li>
  <li><strong>Two GPUs can hold about six times the context of one.</strong> Once weights stop dominating, capacity jumps.</li>
  <li><strong>This hybrid model needs <code>--max-num-seqs</code> set</strong>, or it won’t start.</li>
</ul>

<h2 id="how-these-numbers-were-made">How these numbers were made</h2>

<p>With <a href="https://github.com/KiranTallapragada/gpu-fit">gpu-fit</a>, a small open-source tool that reads
a model’s real architecture from Hugging Face and mirrors vLLM’s memory budget. Weights and KV
per token come straight from the model’s config. Runtime overhead is calibrated against real
vLLM runs; the method and raw data are in the repository. Treat the output as a planning
estimate and confirm on your own hardware before you spend money.</p>

<p><strong>Try your own model and GPU in the <a href="/gpu-fit/">browser calculator</a></strong>, or run
<code>pip install gpu-fit</code>.</p>

<p><em>Corrected 20 Nov 2026: vLLM’s default <code>gpu_memory_utilization</code> is 0.92, not 0.9, so the budget is
41.4 GiB and the runtime overhead about 2.1 GiB. The 32B token and request figures were already right
and match the calculator links (8 requests at once); the hybrid example uses 32. An earlier version
said the FP8 model would load at 32K; vLLM refuses to start when one full request doesn’t fit, and
the text now says so.</em></p>]]></content><author><name>Kiran Tallapragada</name></author><category term="gpu-sizing" /><category term="vllm" /><category term="kv-cache" /><category term="l40s" /><summary type="html"><![CDATA[Qwen3-32B on one NVIDIA L40S: why 16-bit weights don’t fit, why 8-bit weights still serve less than one user at 32K, and the two settings that change everything.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://kirantallapragada.com/assets/img/og.png" /><media:content medium="image" url="https://kirantallapragada.com/assets/img/og.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>