No publishable result
Intel Arc Pro B60 - 24 GB
Known inputs: 24 GB usable memory and
456 GB/s of sourced peak bandwidth.
No backend target supports BF16 on 24 GB (Intel Arc Pro B60). This is a known-unsupported path, not a failure to estimate. Choose a quant format one of the available backends supports, or add a backend target whose applies_to covers arc-pro-b60-24gb.
Model · llama · released 2024-09
What runs Llama 3.2 1B Instruct
Capacity is charged on 1.2 B total parameters. Per-token traffic is charged on 1.2 B active parameters. Every rate below is an estimate at a 1,024-token position, batch 1, single stream.
Sourced architecture
Parameter accounting
Capacity is charged on total parameters. Traffic is charged on active parameters.
| Fact | Value |
|---|---|
| Architecture | Dense |
| Total parameters | 1.2 B |
| Active parameters per token | 1.2 B |
| Attention | GQA |
| Layer groups |
16 layers ·
32 heads, 8 KV heads, head dim 64, full attention
|
| Vocabulary | 128,256 tokens |
| Tied embeddings | Yes |
| Maximum context | 131,072 tokens |
| Native precision | bf16 |
Weight accounting
Size at each quantization
An exact published artifact size always beats parameters × effective bits, and the basis column says which was used. Quality is an ordinal stated in the quant table, never derived from bit width.
| Quantization | Effective bits per weight | Quality | Weights (capacity) | Active weights (per-token traffic) | Size basis |
|---|---|---|---|---|---|
| BF16bf16 · quant ID bf16 | 16 | excellentRank 100 | 2.5 GB | 2.5 GB | Computed |
| FP16fp16 · quant ID fp16 | 16 | excellentRank 100 | 2.5 GB | 2.5 GB | Computed |
| Q8_0gguf_kquant · quant ID q8_0 | 8.5 | excellentRank 96 | 1.3 GB | 1.3 GB | Computed |
| Q6_Kgguf_kquant · quant ID q6_k | 6.56 | excellentRank 92 | 1 GB | 1 GB | Computed |
| FP8fp8 · quant ID fp8 | 8 | excellentRank 90 | 1.2 GB | 1.2 GB | Computed |
| Q5_K_Mgguf_kquant · quant ID q5_k_m | 5.67 | goodRank 86 | 0.9 GB | 0.9 GB | Computed |
| Q5_K_Sgguf_kquant · quant ID q5_k_s | 5.52 | goodRank 83 | 0.9 GB | 0.9 GB | Computed |
| IQ4_NLgguf_iquant · quant ID iq4_nl | 4.5 | goodRank 76 | 0.7 GB | 0.7 GB | Computed |
| Q4_K_Mgguf_kquant · quant ID q4_k_m | 4.83 | goodRank 74 | 0.7 GB | 0.7 GB | Computed |
| IQ4_XSgguf_iquant · quant ID iq4_xs | 4.25 | goodRank 73 | 0.7 GB | 0.7 GB | Computed |
| Q4_K_Sgguf_kquant · quant ID q4_k_s | 4.58 | goodRank 70 | 0.7 GB | 0.7 GB | Computed |
| AWQ 4-bitawq · quant ID awq_4bit | 4.25 | acceptableRank 68 | 0.7 GB | 0.7 GB | Computed |
| GPTQ 4-bitgptq · quant ID gptq_4bit | 4.25 | acceptableRank 64 | 0.7 GB | 0.7 GB | Computed |
| Q3_K_Lgguf_kquant · quant ID q3_k_l | 4.27 | acceptableRank 58 | 0.7 GB | 0.7 GB | Computed |
| IQ3_Mgguf_iquant · quant ID iq3_m | 3.66 | acceptableRank 56 | 0.6 GB | 0.6 GB | Computed |
| Q3_K_Mgguf_kquant · quant ID q3_k_m | 3.91 | acceptableRank 52 | 0.6 GB | 0.6 GB | Computed |
| IQ3_Sgguf_iquant · quant ID iq3_s | 3.44 | acceptableRank 51 | 0.5 GB | 0.5 GB | Computed |
| IQ3_XXSgguf_iquant · quant ID iq3_xxs | 3.06 | acceptableRank 45 | 0.5 GB | 0.5 GB | Computed |
| Q3_K_Sgguf_kquant · quant ID q3_k_s | 3.5 | degradedRank 44 | 0.5 GB | 0.5 GB | Computed |
| IQ2_Mgguf_iquant · quant ID iq2_m | 2.7 | degradedRank 34 | 0.4 GB | 0.4 GB | Computed |
| Q2_Kgguf_kquant · quant ID q2_k | 3.35 | degradedRank 32 | 0.5 GB | 0.5 GB | Computed |
| IQ2_Sgguf_iquant · quant ID iq2_s | 2.5 | degradedRank 30 | 0.4 GB | 0.4 GB | Computed |
| IQ2_XSgguf_iquant · quant ID iq2_xs | 2.31 | degradedRank 25 | 0.4 GB | 0.4 GB | Computed |
| IQ2_XXSgguf_iquant · quant ID iq2_xxs | 2.06 | degradedRank 18 | 0.3 GB | 0.3 GB | Computed |
Estimated ranges · not measured
Configurations ranked by decode range
Every range is a sensitivity band over the selected backend target's declared bandwidth-efficiency interval, at s = 1,024, batch 1, single stream. None of it is measured. The recommended quantization is the highest-ranked one that fits at that context, with the KV cache held at fp16. The next column re-runs the same choice at this model's full declared context of 131,072 tokens, where the cache is far larger.
| Rank | Configuration | Recommended quantization | Still recommended at 131,072 tokens | Memory required | Decode range (estimate) | Weights-only ceiling | Confidence | Deep link |
|---|---|---|---|---|---|---|---|---|
| 1 | NVIDIA Rubin GPU 288 GB · config ID rubin-sxm-288gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 288 GB usable | 5,092.6–7,024.3 tok/s estimate at s = 1,024 | 8,899.7 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 2 | NVIDIA B200 SXM 180 GB · config ID b200-sxm-180gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 180 GB usable | 1,851.9–2,554.3 tok/s estimate at s = 1,024 | 3,236.2 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 3 | NVIDIA B300 SXM 288 GB · config ID b300-sxm-288gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 288 GB usable | 1,851.9–2,554.3 tok/s estimate at s = 1,024 | 3,236.2 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 4 | AMD Instinct MI300X 192 GB · config ID instinct-mi300x-192gb | BF16 vllm-rocm · runner-up FP16 | BF16 | 4 GB of 192 GB usable | 1,057.6–1,544.1 tok/s estimate at s = 1,024 | 2,144.0 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 5 | NVIDIA H200 SXM 141 GB · config ID h200-sxm-141gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 141 GB usable | 1,111.1–1,532.6 tok/s estimate at s = 1,024 | 1,941.7 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 6 | NVIDIA H100 SXM 80 GB · config ID h100-sxm-80gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 80 GB usable | 775.5–1,069.6 tok/s estimate at s = 1,024 | 1,355.2 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 7 | NVIDIA A100 SXM 80 GB · config ID a100-sxm-80gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 80 GB usable | 463.0–638.6 tok/s estimate at s = 1,024 | 809.1 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 8 | NVIDIA GeForce RTX 5090 32 GB · config ID geforce-rtx-5090-32gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 32 GB usable | 414.8–572.2 tok/s estimate at s = 1,024 | 724.9 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 9 | NVIDIA RTX PRO 6000 Blackwell Workstation Edition 96 GB · config ID rtx-pro-6000-blackwell-96gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 96 GB usable | 414.8–572.2 tok/s estimate at s = 1,024 | 724.9 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 10 | Apple M5 Ultra 256 GB · config ID apple-m5-ultra-256gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 204.8 GB usable | 296.9–402.3 tok/s estimate at s = 1,024 | 485.4 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 11 | Apple M5 Ultra 512 GB · config ID apple-m5-ultra-512gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 409.6 GB usable | 296.9–402.3 tok/s estimate at s = 1,024 | 485.4 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 12 | Apple M5 Ultra 96 GB · config ID apple-m5-ultra-96gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 76.8 GB usable | 296.9–402.3 tok/s estimate at s = 1,024 | 485.4 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 13 | NVIDIA GeForce RTX 4090 24 GB · config ID geforce-rtx-4090-24gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 24 GB usable | 233.3–321.8 tok/s estimate at s = 1,024 | 407.8 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 14 | NVIDIA RTX 6000 Ada Generation 48 GB · config ID rtx-6000-ada-48gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 48 GB usable | 222.2–306.5 tok/s estimate at s = 1,024 | 388.3 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 15 | NVIDIA GeForce RTX 3090 24 GB · config ID geforce-rtx-3090-24gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 24 GB usable | 216.7–298.9 tok/s estimate at s = 1,024 | 378.7 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 16 | AMD Instinct MI50 32 GB · config ID instinct-mi50-32gb | BF16 vllm-rocm · runner-up FP16 | BF16 | 4 GB of 32 GB usable | 204.3–298.3 tok/s estimate at s = 1,024 | 414.2 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 17 | AMD Radeon RX 7900 XTX 24 GB · config ID radeon-rx-7900-xtx-24gb | BF16 vllm-rocm · runner-up FP16 | BF16 | 4 GB of 24 GB usable | 191.6–279.7 tok/s estimate at s = 1,024 | 388.3 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 18 | NVIDIA L40S 48 GB · config ID l40s-48gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 48 GB usable | 200.0–275.9 tok/s estimate at s = 1,024 | 349.5 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 19 | Apple M3 Ultra 512 GB · config ID apple-m3-ultra-512gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 409.6 GB usable | 202.7–274.6 tok/s estimate at s = 1,024 | 331.3 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 20 | Apple M3 Ultra 96 GB · config ID apple-m3-ultra-96gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 76.8 GB usable | 202.7–274.6 tok/s estimate at s = 1,024 | 331.3 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 21 | Apple M1 Ultra 128 GB · config ID apple-m1-ultra-128gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 102.4 GB usable | 198.0–268.2 tok/s estimate at s = 1,024 | 323.6 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 22 | Apple M1 Ultra 64 GB · config ID apple-m1-ultra-64gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 51.2 GB usable | 198.0–268.2 tok/s estimate at s = 1,024 | 323.6 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 23 | Apple M2 Ultra 192 GB · config ID apple-m2-ultra-192gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 153.6 GB usable | 198.0–268.2 tok/s estimate at s = 1,024 | 323.6 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 24 | NVIDIA GeForce RTX 4080 SUPER 16 GB · config ID geforce-rtx-4080-super-16gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 16 GB usable | 170.4–235.0 tok/s estimate at s = 1,024 | 297.7 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 25 | NVIDIA GeForce RTX 4080 16 GB · config ID geforce-rtx-4080-16gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 16 GB usable | 165.9–228.9 tok/s estimate at s = 1,024 | 290.0 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 26 | Intel Arc B580 12 GB · config ID arc-b580-12gb | Q8_0 llama-cpp-vulkan · runner-up Q6_K | Q8_0 | 2.3 GB of 12 GB usable | 142.2–226.8 tok/s estimate at s = 1,024 | 347.2 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 27 | NVIDIA GeForce RTX 4070 Ti SUPER 16 GB · config ID geforce-rtx-4070-ti-super-16gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 16 GB usable | 155.6–214.6 tok/s estimate at s = 1,024 | 271.8 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 28 | Apple M5 Max 128 GB / 614 GB/s · config ID apple-m5-max-128gb-614 | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 102.4 GB usable | 151.9–205.8 tok/s estimate at s = 1,024 | 248.4 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 29 | Apple M5 Max 48 GB / 614 GB/s · config ID apple-m5-max-48gb-614 | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 38.4 GB usable | 151.9–205.8 tok/s estimate at s = 1,024 | 248.4 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 30 | Apple M5 Max 64 GB / 614 GB/s · config ID apple-m5-max-64gb-614 | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 51.2 GB usable | 151.9–205.8 tok/s estimate at s = 1,024 | 248.4 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 31 | Apple M4 Max 128 GB / 546 GB/s · config ID apple-m4-max-128gb-546 | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 102.4 GB usable | 135.1–183.0 tok/s estimate at s = 1,024 | 220.9 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 32 | Apple M4 Max 48 GB / 546 GB/s · config ID apple-m4-max-48gb-546 | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 38.4 GB usable | 135.1–183.0 tok/s estimate at s = 1,024 | 220.9 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 33 | Apple M4 Max 64 GB / 546 GB/s · config ID apple-m4-max-64gb-546 | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 51.2 GB usable | 135.1–183.0 tok/s estimate at s = 1,024 | 220.9 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 34 | NVIDIA GeForce RTX 4070 12 GB · config ID geforce-rtx-4070-12gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 12 GB usable | 116.7–160.9 tok/s estimate at s = 1,024 | 203.9 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 35 | NVIDIA GeForce RTX 4070 SUPER 12 GB · config ID geforce-rtx-4070-super-12gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 12 GB usable | 116.7–160.9 tok/s estimate at s = 1,024 | 203.9 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 36 | NVIDIA GeForce RTX 4070 Ti 12 GB · config ID geforce-rtx-4070-ti-12gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 12 GB usable | 116.7–160.9 tok/s estimate at s = 1,024 | 203.9 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 37 | Apple M5 Max 36 GB / 460 GB/s · config ID apple-m5-max-36gb-460 | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 28.8 GB usable | 113.8–154.2 tok/s estimate at s = 1,024 | 186.1 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 38 | AMD Ryzen AI Max+ 395 Developer Platform 128 GB · config ID ryzen-ai-max-395-128gb | Q8_0 llama-cpp-rocm · runner-up Q6_K | Q8_0 | 2.3 GB of 96 GB usable | 98.8–144.5 tok/s estimate at s = 1,024 | 194.9 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 39 | Apple M4 Max 36 GB / 410 GB/s · config ID apple-m4-max-36gb-410 | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 28.8 GB usable | 101.5–137.5 tok/s estimate at s = 1,024 | 165.9 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 40 | NVIDIA Tesla P40 24 GB · config ID tesla-p40-24gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 24 GB usable | 80.1–110.5 tok/s estimate at s = 1,024 | 140.0 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 41 | Apple M5 Pro 24 GB · config ID apple-m5-pro-24gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 19.2 GB usable | 76.0–102.9 tok/s estimate at s = 1,024 | 124.2 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 42 | Apple M5 Pro 48 GB · config ID apple-m5-pro-48gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 38.4 GB usable | 76.0–102.9 tok/s estimate at s = 1,024 | 124.2 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 43 | Apple M5 Pro 64 GB · config ID apple-m5-pro-64gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 51.2 GB usable | 76.0–102.9 tok/s estimate at s = 1,024 | 124.2 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 44 | NVIDIA RTX PRO 2000 Blackwell 16 GB · config ID rtx-pro-2000-blackwell-16gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 16 GB usable | 66.7–92.0 tok/s estimate at s = 1,024 | 116.5 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 45 | Apple M4 Pro 24 GB · config ID apple-m4-pro-24gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 19.2 GB usable | 67.6–91.5 tok/s estimate at s = 1,024 | 110.4 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 46 | Apple M4 Pro 48 GB · config ID apple-m4-pro-48gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 38.4 GB usable | 67.6–91.5 tok/s estimate at s = 1,024 | 110.4 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 47 | NVIDIA DGX Spark 128 GB · config ID dgx-spark-128gb | BF16 vllm-cuda · runner-up FP16 | BF16 | 4 GB of 115.2 GB usable | 63.2–87.2 tok/s estimate at s = 1,024 | 110.4 tok/s | Tier B matched target, outside efficiency regime, exposed latency uncalibrated, computed artifact size | Open |
| 48 | Apple M6 24 GB · config ID apple-m6-24gb-170 | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 19.2 GB usable | 42.1–57.0 tok/s estimate at s = 1,024 | 68.8 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 49 | Apple M6 32 GB · config ID apple-m6-32gb-170 | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 25.6 GB usable | 42.1–57.0 tok/s estimate at s = 1,024 | 68.8 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 50 | Apple M5 16 GB · config ID apple-m5-16gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 12.8 GB usable | 37.9–51.3 tok/s estimate at s = 1,024 | 61.9 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 51 | Apple M5 24 GB · config ID apple-m5-24gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 19.2 GB usable | 37.9–51.3 tok/s estimate at s = 1,024 | 61.9 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 52 | Apple M5 32 GB · config ID apple-m5-32gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 25.6 GB usable | 37.9–51.3 tok/s estimate at s = 1,024 | 61.9 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 53 | Apple M6 16 GB · config ID apple-m6-16gb-153 | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 12.8 GB usable | 37.9–51.3 tok/s estimate at s = 1,024 | 61.9 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 54 | Apple M4 16 GB · config ID apple-m4-16gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 12.8 GB usable | 29.7–40.2 tok/s estimate at s = 1,024 | 48.5 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 55 | Apple M4 24 GB · config ID apple-m4-24gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 19.2 GB usable | 29.7–40.2 tok/s estimate at s = 1,024 | 48.5 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
| 56 | Apple M4 32 GB · config ID apple-m4-32gb | BF16 mlx-lm-metal · runner-up FP16 | BF16 | 3.3 GB of 25.6 GB usable | 29.7–40.2 tok/s estimate at s = 1,024 | 48.5 tok/s | Tier B matched target, exposed latency uncalibrated, computed artifact size | Open |
Why these tiers
What each confidence reason means
Confidence is resolved per operation, per configuration, per backend target.
- computed artifact size - No exact published artifact size for BF16; the size is computed from parameter count and effective bits per weight (§4.3.1).
- exposed latency uncalibrated - No exposed per-layer latency has been measured for this target, so the fixed overhead term contributes zero. That is a stated assumption, not a measurement (§2, §14.3).
- matched target - Standard attention on a matched backend target (vllm-cuda).
- outside efficiency regime - This query falls outside the regime the efficiency band for bf16 was declared valid for, so the band is being extrapolated (§2.7.1).
Where to start
Minimum viable and cheapest qualifying
A callout appears only where a sourced price supports it.
- The smallest configuration that runs Llama 3.2 1B Instruct is Intel Arc B580 - 12 GB. It has 12 GB of usable memory. The recommended quantization there is Q8_0, whose stated quality tier is excellent.
- The cheapest configuration whose whole estimated range clears 10 tokens per second is Intel Arc B580 - 12 GB at $249.
- The cheapest configuration whose whole estimated range clears 20 tokens per second is Intel Arc B580 - 12 GB at $249.
- The cheapest configuration whose whole estimated range clears 40 tokens per second is Intel Arc B580 - 12 GB at $249.
Not estimated
Configurations with no publishable result
These rows carry a reason instead of a number. They are not a slow result. Where a reason names a quantization, that is the preferred format the resolver probed with, not the only one it tried.
No publishable result
NVIDIA Jetson AGX Orin - 64 GB
Known inputs: 54.4 GB usable memory and
204.8 GB/s of sourced peak bandwidth.
No backend target supports BF16 on 64 GB (NVIDIA Jetson AGX Orin). This is a known-unsupported path, not a failure to estimate. Choose a quant format one of the available backends supports, or add a backend target whose applies_to covers jetson-agx-orin-64gb.
No publishable result
Raspberry Pi 5 - 8 GB
Known inputs: 6.4 GB usable memory and
17 GB/s of sourced peak bandwidth.
No backend target supports BF16 on 8 GB (Raspberry Pi 5). This is a known-unsupported path, not a failure to estimate. Choose a quant format one of the available backends supports, or add a backend target whose applies_to covers raspberry-pi-5-8gb.
No publishable result
AMD Ryzen Threadripper PRO 7995WX Platform - 512 GB
Known inputs: 460.8 GB usable memory and
332.8 GB/s of sourced peak bandwidth.
No backend target supports BF16 on 512 GB (AMD Ryzen Threadripper PRO 7995WX Platform). This is a known-unsupported path, not a failure to estimate. Choose a quant format one of the available backends supports, or add a backend target whose applies_to covers threadripper-pro-7995wx-512gb.
Audit trail
Primary sources
Parameter counts and architecture come from the model publisher.
Model recordllama-3.2-1b
https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct/raw/main/config.json
Keep exploring
Other model pages
Each linked model passed the generated-page admission gate.