Scope first, because it is the assumption everything else rests on: one stream, batch size one, ordinary autoregressive decoding. That is the shape of a person typing at a chat window on their own machine. Serving many users at once, speculative decoding and batched throughput all change the arithmetic, and none of them are modelled here.
What has to fit
A model runs locally when three things fit in memory at the same time — and the memory that counts is what the driver, the OS and the backend leave you, not the number on the box.
- The weights. Every parameter, at whatever precision you run it in.
- The KV cache. Attention keeps a key and a value for every token in the conversation, at every layer. It grows as you talk, which is why a model that loads fine can run out of memory an hour later.
- Room to work. Activations, buffers and the backend's own allocations, charged as a floor plus a fraction of the weights.
The cache is the part people underestimate. Its size per token depends on the layer count, how many key/value heads the architecture shares, and how many bytes each cached element takes:
| Cache precision | Bytes per element | Relative size |
|---|---|---|
| fp16 | 2 | 1× |
| q8 | 1 | 0.5× |
| q4 | 0.5 | 0.25× |
Models whose layers only attend to a fixed window cap that growth: past the window, older tokens stop being read, and the cache stops growing with them.
What sets the speed
Generating one token means reading the weights that token needs, plus the whole resident cache, out of memory. At batch size one there is almost no arithmetic reuse to hide that read behind, which is why memory bandwidth — not core count, not clock speed — is usually the number that decides how fast a model feels.
time per token = max(memory time, compute time) + fixed per-layer overhead memory time = bytes read per token / (bandwidth × efficiency) compute time = arithmetic per token / (peak throughput × utilisation)
The max matters. Dividing bytes by bandwidth and stopping there quietly
asserts that arithmetic never binds — untrue for small models on fast hardware, and for
long-context attention. Where a configuration approaches the point at which computation
binds instead, the site says so rather than reporting a bandwidth figure it no longer
believes.
Alongside every estimate sits the weights-only ceiling: peak bandwidth divided by the size of the weights read per token. No efficiency factor, no cache traffic, no overhead. Nothing can beat it on that hardware, and it is the one figure on the page that is almost free of assumptions — so a reader who distrusts every coefficient still leaves with something solid.
Speed also depends on where you are in the conversation. A rate quoted at the first message is not the rate at 100,000 tokens, because the cache being read has grown. Every rate on this site is labelled with the position it was computed at: a short-context figure at 1,024 tokens, and the figure at the full context you selected.
Why every figure is a range
No accelerator reaches its rated bandwidth in practice, and how close a backend gets depends on the kernels it uses, the quantization format and the hardware. Rather than invent a single efficiency number, each backend carries a declared band, and the published range is that band propagated through the formula above. The range is a sensitivity range over a stated assumption — not a measurement, and not a claim about how often reality lands inside it.
| Backend path | Runs on | Bandwidth efficiency | Compute utilisation |
|---|---|---|---|
| llama.cpp cuda |
NVIDIA — Consumer GPU, Workstation GPU, Datacenter GPU, Unified APU |
K-quant 60%–82% I-quant 52%–76%
|
20%–45% |
| llama.cpp rocm |
AMD — Consumer GPU, Datacenter GPU, Unified APU |
K-quant 52%–76% I-quant 45%–68%
|
18%–40% |
| llama.cpp vulkan |
AMD / Intel — Consumer GPU, Unified APU |
K-quant 42%–67% I-quant 35%–58%
|
15%–35% |
| llama.cpp metal |
Apple — Apple Silicon |
K-quant 58%–80% I-quant 50%–73%
|
18%–40% |
| MLX LM metal |
Apple — Apple Silicon |
FP16 62%–84% BF16 62%–84%
|
20%–48% |
| vLLM cuda |
NVIDIA — Consumer GPU, Workstation GPU, Datacenter GPU, Unified APU |
FP16 58%–80% BF16 58%–80% FP8 52%–76% AWQ 48%–72% GPTQ 48%–72%
|
25%–55% |
| vLLM rocm |
AMD — Consumer GPU, Datacenter GPU |
FP16 50%–73% BF16 50%–73% FP8 45%–68% AWQ 40%–64% GPTQ 40%–64%
|
22%–50% |
Each band is declared valid for a range of model sizes and context lengths. Ask a question outside that range and the estimate still renders, with its confidence lowered, because the band is being stretched past what it was stated for.
None of these bands is measured yet. They are engineering assumptions, and the fixed per-layer overhead is currently zero for every backend for the same reason. That is why no decode figure on this site carries the top confidence grade — a badge that claimed otherwise before a benchmark existed would be decoration, not evidence.
What quantization costs
Quantization is the main lever between "does not fit" and "fits comfortably". Weight bytes are charged at the effective bits per weight — including the scales and block metadata the format carries, not the number in its name — and where a model publishes the exact size of a real file, that size is used instead of the arithmetic.
Quality is ordered separately from bit width, because it is not monotonic in bits across formats. The table is ranked by quality, not by size:
| Quantization | Format | Effective bits/weight | Quality |
|---|---|---|---|
| BF16 | BF16 | 16 | excellent |
| FP16 | FP16 | 16 | excellent |
| Q8_0 | K-quant | 8.5 | excellent |
| Q6_K | K-quant | 6.56 | excellent |
| FP8 | FP8 | 8 | excellent |
| Q5_K_M | K-quant | 5.67 | good |
| Q5_K_S | K-quant | 5.52 | good |
| IQ4_NL | I-quant | 4.5 | good |
| Q4_K_M | K-quant | 4.83 | good |
| IQ4_XS | I-quant | 4.25 | good |
| Q4_K_S | K-quant | 4.58 | good |
| AWQ 4-bit | AWQ | 4.25 | acceptable |
| GPTQ 4-bit | GPTQ | 4.25 | acceptable |
| Q3_K_L | K-quant | 4.27 | acceptable |
| IQ3_M | I-quant | 3.66 | acceptable |
| Q3_K_M | K-quant | 3.91 | acceptable |
| IQ3_S | I-quant | 3.44 | acceptable |
| IQ3_XXS | I-quant | 3.06 | acceptable |
| Q3_K_S | K-quant | 3.5 | degraded |
| IQ2_M | I-quant | 2.7 | degraded |
| Q2_K | K-quant | 3.35 | degraded |
| IQ2_S | I-quant | 2.5 | degraded |
| IQ2_XS | I-quant | 2.31 | degraded |
| IQ2_XXS | I-quant | 2.06 | degraded |
The quality column is a stated ordering, not a measurement: this site does not publish perplexity deltas it has not sourced.
How far to trust a figure
Every estimate carries a grade, resolved per question rather than stored on a model or a device. A means nothing about the query weakened it. B means something about it is modelled but weakly constrained. C means an approximation with bounded error is in play, and the range is widened to match. The worst thing that applies wins — grades do not average out.
| What happened | Fit | Speed | Prompt |
|---|---|---|---|
| Compressed attention, reconstructed The backend rebuilds per-head keys and values each step. The cost of that rebuild is modelled with bounded error, so the range is widened rather than withheld. |
C | C | C |
| Sliding-window layers Some layers only attend to a fixed window, which caps how much cache is read per token. The interaction is modelled but weakly constrained, so the estimate carries less confidence. |
A | B | B |
| Split across devices, by layer One stream runs on one device at a time, so bandwidth does not add up. The estimate is the sum of each device's segment. |
A | B | B |
| Split across devices, by tensor Local bandwidth roughly adds up, but every layer pays a collective communication cost that depends on the interconnect. That cost is assumed from a topology table, not measured. |
A | B | B |
| Partly in host memory The model does not fit in device memory, and this backend has a sourced profile for what happens then. The result is an approximation with bounded error, so the range widens. |
A | C | C |
| Outside the range the assumption was stated for Each efficiency band is declared valid for a range of model sizes and context lengths. Your query sits outside it, so the band is being extrapolated. |
A | B | B |
| No measured per-layer overhead Real inference pays a fixed cost per layer that no bandwidth figure predicts. Nothing has been measured on this hardware yet, so that term contributes zero — a stated assumption rather than a silent default, and the reason every decode figure on this site stops short of the top confidence grade. |
A | B | B |
| Approaching compute-bound Computed arithmetic time is more than half the memory time here, so bandwidth is no longer the whole story and the estimate is treated as less certain. |
A | B | A |
| Weight size computed, not published No exact file size has been recorded for this model at this quantization, so the size comes from the parameter count and the effective bits per weight. Real files vary a little around that. |
B | B | A |
When we show nothing
Some uncertainty cannot be expressed as a wider range. When the honest span between two possible answers is a factor rather than a margin, this site publishes no number — and says which input is missing, in the place the number would have been.
It is never a dash, a zero or an empty cell. A blank in a tokens-per-second column reads as slow, and a blank fit verdict reads as does not fit; both are confidently wrong in a direction you cannot detect. Fit, speed and prompt processing are decided independently, so one can be withheld while another stands.
| Why a number is withheld | Fit | Speed | Prompt |
|---|---|---|---|
| No backend runs this combination None of the catalogued backends supports that quantization format on that hardware. This is a known-unsupported path rather than a failure to estimate, and it is worth knowing before you buy anything. |
withheld | withheld | withheld |
| Compressed attention, path unknown The two ways a backend can execute this attention differ in cache traffic by roughly ten times. Picking one at random would not widen the range — it would invalidate the answer, so no number is shown. |
withheld | withheld | withheld |
| Incomplete architecture record This model's record is missing a dimension both candidate cache formulas need. The estimator never falls back to the other formula, so nothing is published for it. |
withheld | withheld | withheld |
| An attention mechanism we cannot size yet Several recent open-weight families use hybrid, linear or unified-key-value attention whose cache behaviour has no reviewed formula here. Their weights and architecture are catalogued; their speed and fit are not. |
withheld | withheld | withheld |
| Partly in host memory, behaviour unknown It does not fit, and nothing sourced says what this backend does about it. "Layers compute in host RAM" and "weights cross the bus every step" differ by a factor, not a margin, so no rate is shown. |
A | withheld | withheld |
| No sourced arithmetic throughput Time-to-first-token needs the hardware's throughput at the precision this backend actually computes in. Where that figure is not published, it is never borrowed from a related precision, so the prompt-processing estimate is withheld while the token rate stands. |
— | — | withheld |
A withheld model is still worth having in the catalogue: its architecture, parameter counts and weight sizes at each quantization are all published, because those depend only on sourced facts. It is the speed and the fit verdict that wait for a defensible formula.
What this does not model
- Serving many users at once. Batching changes the arithmetic-to-bandwidth ratio completely, and every figure here assumes a single stream.
- Speculative decoding and other tricks. They change how many bytes are read per accepted token, which is the quantity the whole model is built on.
- Spilling to host memory, unless a backend has a sourced profile for it. Otherwise the difference between plausible behaviours is a factor, and the rate is withheld.
- Fine-tuning, training or image generation. Different workloads, different bottlenecks.
- Measured benchmarks. Nothing here is a stopwatch reading. Everything is an engineering estimate from published specifications, labelled as one.
Where the inputs come from
Memory capacity and bandwidth are the two figures the entire site rests on, so neither may be recorded without a primary source and the date it was checked. Prices are optional but sourced when present, and go stale on a schedule rather than sitting there indefinitely. Arithmetic throughput is optional too — where it is absent, prompt-processing estimates are withheld rather than inferred from a related precision.
Bandwidth, capacity, price and availability are recorded per configuration, never per device family. A 48 GB card and a 24 GB card of the same name are different products with different bandwidth, and averaging them would corrupt every figure downstream. Only released or officially announced hardware is catalogued — never a rumoured configuration.
Found something wrong? The estimate for any combination shows the inputs it used, so a disagreement can be pinned to a specific number rather than to the conclusion.