MB Memory Bandwidth
Explorer

Model · llama · released 2024-07

What runs Llama 3.1 8B Instruct

Capacity is charged on 8 B total parameters. Per-token traffic is charged on 8 B active parameters. Every rate below is an estimate at a 1,024-token position, batch 1, single stream.

Open in the calculator → Compare the two ends of the table

Sourced architecture

Parameter accounting

Capacity is charged on total parameters. Traffic is charged on active parameters.

FactValue
ArchitectureDense
Total parameters8 B
Active parameters per token8 B
Attention GQA
Layer groups 32 layers · 32 heads, 8 KV heads, head dim 128, full attention
Vocabulary128,256 tokens
Tied embeddingsNo
Maximum context131,072 tokens
Native precisionbf16

Weight accounting

Size at each quantization

An exact published artifact size always beats parameters × effective bits, and the basis column says which was used. Quality is an ordinal stated in the quant table, never derived from bit width.

Quantization Effective bits per weight Quality Weights (capacity) Active weights (per-token traffic) Size basis
BF16bf16 · quant ID bf16 16 excellentRank 100 16.1 GB 16.1 GB Computed
FP16fp16 · quant ID fp16 16 excellentRank 100 16.1 GB 16.1 GB Computed
Q8_0gguf_kquant · quant ID q8_0 8.5 excellentRank 96 8.5 GB 8.5 GB Computed
Q6_Kgguf_kquant · quant ID q6_k 6.56 excellentRank 92 6.6 GB 6.6 GB Computed
FP8fp8 · quant ID fp8 8 excellentRank 90 8 GB 8 GB Computed
Q5_K_Mgguf_kquant · quant ID q5_k_m 5.67 goodRank 86 5.7 GB 5.7 GB Computed
Q5_K_Sgguf_kquant · quant ID q5_k_s 5.52 goodRank 83 5.5 GB 5.5 GB Computed
IQ4_NLgguf_iquant · quant ID iq4_nl 4.5 goodRank 76 4.5 GB 4.5 GB Computed
Q4_K_Mgguf_kquant · quant ID q4_k_m 4.83 goodRank 74 4.8 GB 4.8 GB Computed
IQ4_XSgguf_iquant · quant ID iq4_xs 4.25 goodRank 73 4.3 GB 4.3 GB Computed
Q4_K_Sgguf_kquant · quant ID q4_k_s 4.58 goodRank 70 4.6 GB 4.6 GB Computed
AWQ 4-bitawq · quant ID awq_4bit 4.25 acceptableRank 68 4.3 GB 4.3 GB Computed
GPTQ 4-bitgptq · quant ID gptq_4bit 4.25 acceptableRank 64 4.3 GB 4.3 GB Computed
Q3_K_Lgguf_kquant · quant ID q3_k_l 4.27 acceptableRank 58 4.3 GB 4.3 GB Computed
IQ3_Mgguf_iquant · quant ID iq3_m 3.66 acceptableRank 56 3.7 GB 3.7 GB Computed
Q3_K_Mgguf_kquant · quant ID q3_k_m 3.91 acceptableRank 52 3.9 GB 3.9 GB Computed
IQ3_Sgguf_iquant · quant ID iq3_s 3.44 acceptableRank 51 3.5 GB 3.5 GB Computed
IQ3_XXSgguf_iquant · quant ID iq3_xxs 3.06 acceptableRank 45 3.1 GB 3.1 GB Computed
Q3_K_Sgguf_kquant · quant ID q3_k_s 3.5 degradedRank 44 3.5 GB 3.5 GB Computed
IQ2_Mgguf_iquant · quant ID iq2_m 2.7 degradedRank 34 2.7 GB 2.7 GB Computed
Q2_Kgguf_kquant · quant ID q2_k 3.35 degradedRank 32 3.4 GB 3.4 GB Computed
IQ2_Sgguf_iquant · quant ID iq2_s 2.5 degradedRank 30 2.5 GB 2.5 GB Computed
IQ2_XSgguf_iquant · quant ID iq2_xs 2.31 degradedRank 25 2.3 GB 2.3 GB Computed
IQ2_XXSgguf_iquant · quant ID iq2_xxs 2.06 degradedRank 18 2.1 GB 2.1 GB Computed

Estimated ranges · not measured

Configurations ranked by decode range

Every range is a sensitivity band over the selected backend target's declared bandwidth-efficiency interval, at s = 1,024, batch 1, single stream. None of it is measured. The recommended quantization is the highest-ranked one that fits at that context, with the KV cache held at fp16. The next column re-runs the same choice at this model's full declared context of 131,072 tokens, where the cache is far larger.

Rank Configuration Recommended quantization Still recommended at 131,072 tokens Memory required Decode range (estimate) Weights-only ceiling Confidence Deep link
1 NVIDIA Rubin GPU 288 GB · config ID rubin-sxm-288gb BF16 vllm-cuda · runner-up FP16 BF16 17.7 GB of 288 GB usable 787.9–1,086.8 tok/s estimate at s = 1,024 1,369.9 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
2 NVIDIA B200 SXM 180 GB · config ID b200-sxm-180gb BF16 vllm-cuda · runner-up FP16 BF16 17.7 GB of 180 GB usable 286.5–395.2 tok/s estimate at s = 1,024 498.1 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
3 NVIDIA B300 SXM 288 GB · config ID b300-sxm-288gb BF16 vllm-cuda · runner-up FP16 BF16 17.7 GB of 288 GB usable 286.5–395.2 tok/s estimate at s = 1,024 498.1 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
4 AMD Instinct MI300X 192 GB · config ID instinct-mi300x-192gb BF16 vllm-rocm · runner-up FP16 BF16 17.7 GB of 192 GB usable 163.6–238.9 tok/s estimate at s = 1,024 330.0 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
5 NVIDIA H200 SXM 141 GB · config ID h200-sxm-141gb BF16 vllm-cuda · runner-up FP16 BF16 17.7 GB of 141 GB usable 171.9–237.1 tok/s estimate at s = 1,024 298.9 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
6 NVIDIA H100 SXM 80 GB · config ID h100-sxm-80gb BF16 vllm-cuda · runner-up FP16 BF16 17.7 GB of 80 GB usable 120.0–165.5 tok/s estimate at s = 1,024 208.6 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
7 NVIDIA A100 SXM 80 GB · config ID a100-sxm-80gb BF16 vllm-cuda · runner-up FP16 BF16 17.7 GB of 80 GB usable 71.6–98.8 tok/s estimate at s = 1,024 124.5 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
8 NVIDIA GeForce RTX 5090 32 GB · config ID geforce-rtx-5090-32gb BF16 vllm-cuda · runner-up FP16 Q8_0 drops from BF16 17.7 GB of 32 GB usable 64.2–88.5 tok/s estimate at s = 1,024 111.6 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
9 NVIDIA RTX PRO 6000 Blackwell Workstation Edition 96 GB · config ID rtx-pro-6000-blackwell-96gb BF16 vllm-cuda · runner-up FP16 BF16 17.7 GB of 96 GB usable 64.2–88.5 tok/s estimate at s = 1,024 111.6 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
10 NVIDIA GeForce RTX 4080 SUPER 16 GB · config ID geforce-rtx-4080-super-16gb Q8_0 llama-cpp-cuda · runner-up Q6_K None fitsthe cache alone exhausts usable memory 9.7 GB of 16 GB usable 51.0–69.6 tok/s estimate at s = 1,024 86.3 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
11 NVIDIA GeForce RTX 4080 16 GB · config ID geforce-rtx-4080-16gb Q8_0 llama-cpp-cuda · runner-up Q6_K None fitsthe cache alone exhausts usable memory 9.7 GB of 16 GB usable 49.6–67.8 tok/s estimate at s = 1,024 84.0 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
12 NVIDIA GeForce RTX 4070 Ti SUPER 16 GB · config ID geforce-rtx-4070-ti-super-16gb Q8_0 llama-cpp-cuda · runner-up Q6_K None fitsthe cache alone exhausts usable memory 9.7 GB of 16 GB usable 46.5–63.6 tok/s estimate at s = 1,024 78.8 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
13 Apple M5 Ultra 256 GB · config ID apple-m5-ultra-256gb BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 204.8 GB usable 45.9–62.2 tok/s estimate at s = 1,024 74.7 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
14 Apple M5 Ultra 512 GB · config ID apple-m5-ultra-512gb BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 409.6 GB usable 45.9–62.2 tok/s estimate at s = 1,024 74.7 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
15 Apple M5 Ultra 96 GB · config ID apple-m5-ultra-96gb BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 76.8 GB usable 45.9–62.2 tok/s estimate at s = 1,024 74.7 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
16 NVIDIA GeForce RTX 4090 24 GB · config ID geforce-rtx-4090-24gb BF16 vllm-cuda · runner-up FP16 Q5_K_M drops from BF16 17.7 GB of 24 GB usable 36.1–49.8 tok/s estimate at s = 1,024 62.8 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
17 NVIDIA GeForce RTX 4070 12 GB · config ID geforce-rtx-4070-12gb Q8_0 llama-cpp-cuda · runner-up Q6_K None fitsthe cache alone exhausts usable memory 9.7 GB of 12 GB usable 34.9–47.7 tok/s estimate at s = 1,024 59.1 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
18 NVIDIA GeForce RTX 4070 SUPER 12 GB · config ID geforce-rtx-4070-super-12gb Q8_0 llama-cpp-cuda · runner-up Q6_K None fitsthe cache alone exhausts usable memory 9.7 GB of 12 GB usable 34.9–47.7 tok/s estimate at s = 1,024 59.1 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
19 NVIDIA GeForce RTX 4070 Ti 12 GB · config ID geforce-rtx-4070-ti-12gb Q8_0 llama-cpp-cuda · runner-up Q6_K None fitsthe cache alone exhausts usable memory 9.7 GB of 12 GB usable 34.9–47.7 tok/s estimate at s = 1,024 59.1 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
20 NVIDIA RTX 6000 Ada Generation 48 GB · config ID rtx-6000-ada-48gb BF16 vllm-cuda · runner-up FP16 BF16 17.7 GB of 48 GB usable 34.4–47.4 tok/s estimate at s = 1,024 59.8 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
21 NVIDIA GeForce RTX 3090 24 GB · config ID geforce-rtx-3090-24gb BF16 vllm-cuda · runner-up FP16 Q5_K_M drops from BF16 17.7 GB of 24 GB usable 33.5–46.2 tok/s estimate at s = 1,024 58.3 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
22 AMD Instinct MI50 32 GB · config ID instinct-mi50-32gb BF16 vllm-rocm · runner-up FP16 Q8_0 drops from BF16 17.7 GB of 32 GB usable 31.6–46.2 tok/s estimate at s = 1,024 63.8 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
23 AMD Radeon RX 7900 XTX 24 GB · config ID radeon-rx-7900-xtx-24gb BF16 vllm-rocm · runner-up FP16 Q5_K_M drops from BF16 17.7 GB of 24 GB usable 29.6–43.3 tok/s estimate at s = 1,024 59.8 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
24 NVIDIA L40S 48 GB · config ID l40s-48gb BF16 vllm-cuda · runner-up FP16 BF16 17.7 GB of 48 GB usable 30.9–42.7 tok/s estimate at s = 1,024 53.8 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
25 Apple M3 Ultra 512 GB · config ID apple-m3-ultra-512gb BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 409.6 GB usable 31.4–42.5 tok/s estimate at s = 1,024 51.0 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
26 Apple M3 Ultra 96 GB · config ID apple-m3-ultra-96gb BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 76.8 GB usable 31.4–42.5 tok/s estimate at s = 1,024 51.0 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
27 Apple M1 Ultra 128 GB · config ID apple-m1-ultra-128gb BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 102.4 GB usable 30.6–41.5 tok/s estimate at s = 1,024 49.8 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
28 Apple M1 Ultra 64 GB · config ID apple-m1-ultra-64gb BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 51.2 GB usable 30.6–41.5 tok/s estimate at s = 1,024 49.8 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
29 Apple M2 Ultra 192 GB · config ID apple-m2-ultra-192gb BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 153.6 GB usable 30.6–41.5 tok/s estimate at s = 1,024 49.8 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
30 Intel Arc B580 12 GB · config ID arc-b580-12gb Q8_0 llama-cpp-vulkan · runner-up Q6_K None fitsthe cache alone exhausts usable memory 9.7 GB of 12 GB usable 22.1–35.3 tok/s estimate at s = 1,024 53.4 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
31 Apple M5 Max 128 GB / 614 GB/s · config ID apple-m5-max-128gb-614 BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 102.4 GB usable 23.5–31.8 tok/s estimate at s = 1,024 38.2 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
32 Apple M5 Max 48 GB / 614 GB/s · config ID apple-m5-max-48gb-614 BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 38.4 GB usable 23.5–31.8 tok/s estimate at s = 1,024 38.2 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
33 Apple M5 Max 64 GB / 614 GB/s · config ID apple-m5-max-64gb-614 BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 51.2 GB usable 23.5–31.8 tok/s estimate at s = 1,024 38.2 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
34 Apple M4 Max 128 GB / 546 GB/s · config ID apple-m4-max-128gb-546 BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 102.4 GB usable 20.9–28.3 tok/s estimate at s = 1,024 34.0 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
35 Apple M4 Max 48 GB / 546 GB/s · config ID apple-m4-max-48gb-546 BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 38.4 GB usable 20.9–28.3 tok/s estimate at s = 1,024 34.0 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
36 Apple M4 Max 64 GB / 546 GB/s · config ID apple-m4-max-64gb-546 BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 51.2 GB usable 20.9–28.3 tok/s estimate at s = 1,024 34.0 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
37 NVIDIA RTX PRO 2000 Blackwell 16 GB · config ID rtx-pro-2000-blackwell-16gb Q8_0 llama-cpp-cuda · runner-up Q6_K None fitsthe cache alone exhausts usable memory 9.7 GB of 16 GB usable 19.9–27.3 tok/s estimate at s = 1,024 33.8 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
38 Apple M5 Max 36 GB / 460 GB/s · config ID apple-m5-max-36gb-460 BF16 mlx-lm-metal · runner-up FP16 Q8_0 drops from BF16 16.9 GB of 28.8 GB usable 17.6–23.9 tok/s estimate at s = 1,024 28.6 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
39 AMD Ryzen AI Max+ 395 Developer Platform 128 GB · config ID ryzen-ai-max-395-128gb Q8_0 llama-cpp-rocm · runner-up Q6_K Q8_0 9.7 GB of 96 GB usable 15.4–22.5 tok/s estimate at s = 1,024 30.0 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
40 Apple M4 Max 36 GB / 410 GB/s · config ID apple-m4-max-36gb-410 BF16 mlx-lm-metal · runner-up FP16 Q8_0 drops from BF16 16.9 GB of 28.8 GB usable 15.7–21.3 tok/s estimate at s = 1,024 25.5 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
41 NVIDIA Tesla P40 24 GB · config ID tesla-p40-24gb BF16 vllm-cuda · runner-up FP16 Q5_K_M drops from BF16 17.7 GB of 24 GB usable 12.4–17.1 tok/s estimate at s = 1,024 21.5 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
42 Apple M5 Pro 24 GB · config ID apple-m5-pro-24gb BF16 mlx-lm-metal · runner-up FP16 None fitsthe cache alone exhausts usable memory 16.9 GB of 19.2 GB usable 11.8–15.9 tok/s estimate at s = 1,024 19.1 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
43 Apple M5 Pro 48 GB · config ID apple-m5-pro-48gb BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 38.4 GB usable 11.8–15.9 tok/s estimate at s = 1,024 19.1 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
44 Apple M5 Pro 64 GB · config ID apple-m5-pro-64gb BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 51.2 GB usable 11.8–15.9 tok/s estimate at s = 1,024 19.1 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
45 Apple M4 Pro 24 GB · config ID apple-m4-pro-24gb BF16 mlx-lm-metal · runner-up FP16 None fitsthe cache alone exhausts usable memory 16.9 GB of 19.2 GB usable 10.5–14.2 tok/s estimate at s = 1,024 17.0 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
46 Apple M4 Pro 48 GB · config ID apple-m4-pro-48gb BF16 mlx-lm-metal · runner-up FP16 BF16 16.9 GB of 38.4 GB usable 10.5–14.2 tok/s estimate at s = 1,024 17.0 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
47 Apple M5 16 GB · config ID apple-m5-16gb Q8_0 llama-cpp-metal · runner-up Q6_K None fitsthe cache alone exhausts usable memory 9.4 GB of 12.8 GB usable 10.2–14.1 tok/s estimate at s = 1,024 17.9 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
48 Apple M6 16 GB · config ID apple-m6-16gb-153 Q8_0 llama-cpp-metal · runner-up Q6_K None fitsthe cache alone exhausts usable memory 9.4 GB of 12.8 GB usable 10.2–14.1 tok/s estimate at s = 1,024 17.9 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
49 NVIDIA DGX Spark 128 GB · config ID dgx-spark-128gb BF16 vllm-cuda · runner-up FP16 BF16 17.7 GB of 115.2 GB usable 9.8–13.5 tok/s estimate at s = 1,024 17.0 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
50 Apple M4 16 GB · config ID apple-m4-16gb Q8_0 llama-cpp-metal · runner-up Q6_K None fitsthe cache alone exhausts usable memory 9.4 GB of 12.8 GB usable 8.0–11.1 tok/s estimate at s = 1,024 14.1 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
51 Apple M6 24 GB · config ID apple-m6-24gb-170 BF16 mlx-lm-metal · runner-up FP16 None fitsthe cache alone exhausts usable memory 16.9 GB of 19.2 GB usable 6.5–8.8 tok/s estimate at s = 1,024 10.6 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
52 Apple M6 32 GB · config ID apple-m6-32gb-170 BF16 mlx-lm-metal · runner-up FP16 Q6_K drops from BF16 16.9 GB of 25.6 GB usable 6.5–8.8 tok/s estimate at s = 1,024 10.6 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
53 Apple M5 24 GB · config ID apple-m5-24gb BF16 mlx-lm-metal · runner-up FP16 None fitsthe cache alone exhausts usable memory 16.9 GB of 19.2 GB usable 5.9–7.9 tok/s estimate at s = 1,024 9.5 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
54 Apple M5 32 GB · config ID apple-m5-32gb BF16 mlx-lm-metal · runner-up FP16 Q6_K drops from BF16 16.9 GB of 25.6 GB usable 5.9–7.9 tok/s estimate at s = 1,024 9.5 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
55 Apple M4 24 GB · config ID apple-m4-24gb BF16 mlx-lm-metal · runner-up FP16 None fitsthe cache alone exhausts usable memory 16.9 GB of 19.2 GB usable 4.6–6.2 tok/s estimate at s = 1,024 7.5 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open
56 Apple M4 32 GB · config ID apple-m4-32gb BF16 mlx-lm-metal · runner-up FP16 Q6_K drops from BF16 16.9 GB of 25.6 GB usable 4.6–6.2 tok/s estimate at s = 1,024 7.5 tok/s Tier B matched target, exposed latency uncalibrated, computed artifact size Open

Why these tiers

What each confidence reason means

Confidence is resolved per operation, per configuration, per backend target.

  • computed artifact size - No exact published artifact size for BF16; the size is computed from parameter count and effective bits per weight (§4.3.1).
  • exposed latency uncalibrated - No exposed per-layer latency has been measured for this target, so the fixed overhead term contributes zero. That is a stated assumption, not a measurement (§2, §14.3).
  • matched target - Standard attention on a matched backend target (vllm-cuda).

Where to start

Minimum viable and cheapest qualifying

A callout appears only where a sourced price supports it.

  • The smallest configuration that runs Llama 3.1 8B Instruct is Intel Arc B580 - 12 GB. It has 12 GB of usable memory. The recommended quantization there is Q8_0, whose stated quality tier is excellent.
  • The cheapest configuration whose whole estimated range clears 10 tokens per second is Intel Arc B580 - 12 GB at $249.
  • The cheapest configuration whose whole estimated range clears 20 tokens per second is Intel Arc B580 - 12 GB at $249.
  • The cheapest configuration whose whole estimated range clears 40 tokens per second is Apple M5 Ultra - 96 GB at $5,499.

Not estimated

Configurations with no publishable result

These rows carry a reason instead of a number. They are not a slow result. Where a reason names a quantization, that is the preferred format the resolver probed with, not the only one it tried.

No publishable result Intel Arc Pro B60 - 24 GB
Known inputs: 24 GB usable memory and 456 GB/s of sourced peak bandwidth. No backend target supports BF16 on 24 GB (Intel Arc Pro B60). This is a known-unsupported path, not a failure to estimate. Choose a quant format one of the available backends supports, or add a backend target whose applies_to covers arc-pro-b60-24gb.
No publishable result NVIDIA Jetson AGX Orin - 64 GB
Known inputs: 54.4 GB usable memory and 204.8 GB/s of sourced peak bandwidth. No backend target supports BF16 on 64 GB (NVIDIA Jetson AGX Orin). This is a known-unsupported path, not a failure to estimate. Choose a quant format one of the available backends supports, or add a backend target whose applies_to covers jetson-agx-orin-64gb.
No publishable result Raspberry Pi 5 - 8 GB
Known inputs: 6.4 GB usable memory and 17 GB/s of sourced peak bandwidth. No backend target supports BF16 on 8 GB (Raspberry Pi 5). This is a known-unsupported path, not a failure to estimate. Choose a quant format one of the available backends supports, or add a backend target whose applies_to covers raspberry-pi-5-8gb.
No publishable result AMD Ryzen Threadripper PRO 7995WX Platform - 512 GB
Known inputs: 460.8 GB usable memory and 332.8 GB/s of sourced peak bandwidth. No backend target supports BF16 on 512 GB (AMD Ryzen Threadripper PRO 7995WX Platform). This is a known-unsupported path, not a failure to estimate. Choose a quant format one of the available backends supports, or add a backend target whose applies_to covers threadripper-pro-7995wx-512gb.

Audit trail

Primary sources

Parameter counts and architecture come from the model publisher.

Keep exploring

Other model pages

Each linked model passed the generated-page admission gate.