What fits on a GPU

Open weights vs closed APIs said that open weights buy you control. Control starts with a smaller question: will the model fit on the card you have? The answer is a sum you can do before you download anything. Running a model yourself picks up after that, with the tools and the cost.

1.Weights: count the parameters, then the bytes

A GPU has a fixed amount of fast memory, its VRAM. The model has to sit in it. If it does not, you do not get a slower answer. You get an out-of-memory error.

The weights claim the first share, and the sum is plain:

memory for weights ≈ parameters × bytes per parameter

At 16-bit that is 2 bytes, at 8-bit it is 1. At 4-bit, Epoch AI's recipe uses 0.54 bytes on average (as of Aug 2025), a little above the 0.5 the bit count suggests.

These are decimal gigabytes, which is how cards are sold. Many tools print GiB (powers of 1024), which are about 7 percent smaller, so a "24 GB" card shows as roughly 22.4 GiB. Mix the two units and you lose more than a gigabyte.

One catch changes the count. A mixture-of-experts model splits its layers into many experts and sends each token to only a few of them. Mixtral 8x7B, a 2023 example, holds about 47B parameters but runs at roughly the speed of a 12B dense model, according to the Hugging Face explainer (Dec 2023). Its memory need is still the big number: the post says "all parameters need to be loaded in RAM". So memory follows the total, and speed follows the active part. Epoch counted only models whose full weights fit on the card, and says this skews its study away from mixture-of-experts models.

2.The KV cache and overhead take the rest

While a model writes, it keeps a KV cache: the attention keys and values for every token it has seen, so it need not recompute them. The tokens are the same ones counted in Context windows and their limits. The cache cannot be skipped, and it grows with context length and with the number of requests in flight at once:

KV cache ≈ bytes per token × tokens in context × users

bytes per token = 2 × layers × KV heads × head size × bytes per value

The 2 is one set of keys and one of values. You read the other three off the model's config.json, under num_hidden_layers, num_key_value_heads and head_dim. For Qwen3-32B, whose card lists 32.8B parameters and an Apache-2.0 licence (as of Oct 2026), the config gives 64 layers, 8 KV heads and a head size of 128. At 16-bit values that is 256 KiB per token, or 2 GiB for one 8K-token conversation.

Last comes overhead: the GPU's runtime context and memory fragmentation. Epoch allows about 2 GiB.

Weights are paid once. The KV cache is paid per token, per user, for as long as the conversation stays open.

Epoch's own worked sum is a good one to repeat for your model. It takes a 27B model at 4-bit on a 24 GB card with 8K context and gets 13.6 GiB of weights, 6.6 GiB of KV cache and 2.0 GiB of overhead.

Epoch's 6.6 GiB is a cautious allowance. It does not reproduce from a modern grouped-query config like the one above, which needs far less, so use your model's own numbers. Long outputs push the other way: Epoch estimates that 20K-token reasoning outputs lift its KV figure to about 15 GiB. Its sum also uses 99 percent of the card, which is a limit for a study, not a place to run. No standard safety margin was confirmed in the sources used, so leave one.

GIB = 1024**3

def fits(params_b, bytes_per_param, layers, kv_heads, head_dim,
         context, users, kv_bytes, card_gb, overhead_gib=2.0):
    weights = params_b * 1e9 * bytes_per_param / GIB
    kv = 2 * layers * kv_heads * head_dim * kv_bytes * context * users / GIB
    total, card = weights + kv + overhead_gib, card_gb * 1e9 / GIB
    print(f"weights {weights:.1f} + KV {kv:.1f} + overhead {overhead_gib:.1f}"
          f" = {total:.1f} GiB of {card:.1f} -> {'fits' if total <= card else 'NO FIT'}")

# Qwen3-32B from its config.json. kv_bytes: 2 = 16-bit cache, 1 = 8-bit cache.
fits(32.8, 0.54, 64, 8, 128, context=8192,  users=1, kv_bytes=2, card_gb=24)
fits(32.8, 0.54, 64, 8, 128, context=32768, users=1, kv_bytes=2, card_gb=24)
fits(32.8, 0.54, 64, 8, 128, context=32768, users=1, kv_bytes=1, card_gb=24)

Run it and you see three lines. The first, at 8K context, says "fits" at 20.5 GiB of 22.4. The second is the failure from the opening: the same model and card at 32K context come to 26.5 GiB, so "NO FIT". Nothing about the weights changed. The third halves the cache to 8-bit and still lands at 22.5, just over the limit. vLLM can store the cache in FP8 (as of Oct 2026), but its quantized KV cache page recommends calibrating the scales, gives no accuracy figure, and the quantisation study below did not test it.

3.Quantisation buys room, and you check the fee

Quantisation stores each weight in fewer bits. The model keeps its shape and parameter count, and each number gets less precise. It is the biggest lever on the weights line of the sum.

The best evidence in the sources used is a study of Llama-3.1 models from 8B to 405B, run in late 2024 (the paper's current version is dated May 2026). Its authors report that FP8 was near-lossless and that well-tuned INT8 lost about 1 to 3 percent. They also report that 4-bit weight-only quantisation, calibrated on real instruction data, kept about 96 to 99 percent of the 16-bit score across their tests. The smallest reasoning model they tried, at 1.5B, kept about 93.5 percent. The title asks the question: "Give Me BF16 or Give Me Death". The authors work on quantisation tooling, so they want it to look good. Those results hold for the Llama-3.1 and DeepSeek-R1-distilled models and the quantiser they used.

Three things they did not test matter to you. The common GGUF files that Ollama and llama.cpp use come from a different quantiser. Formats below 4-bit were not tried, and the common claim that they degrade quickly is not confirmed in the sources used. The KV cache was left at full precision. So the study tells you where to start, and nothing more.

For a feel of the ceiling, Epoch's study assumed 4-bit weights and found that a 24 GB card ran models up to about 27 to 28B parameters, and a 32 GB card up to about 40B (as of Aug 2025). That is more than a year old and the hardware has moved on, so check the Epoch page before you quote it. Epoch also found that models runnable on one consumer card matched the frontier of 6 to 12 months earlier on its benchmarks, and warns that small models are often tuned for those tests.

When this does not apply. The sum assumes the whole model sits on one card. It stops being the limit if you spread the weights over several GPUs. The study found its 405B model needed 4 GPUs at 4-bit where 16 were used at full precision, so quantising changes the GPU count too. Offloading layers to system memory also runs bigger models, at a cost in speed or quality, as Epoch notes. And if you call an API, none of this is your problem. For when the sums favour owning the GPU, read Running a model yourself.