> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Hardware reference

> Compare hardware for local LLM inference at a glance. Specs, model capacity, recommended models, and trade-offs for every supported platform.

Pick the right hardware for local LLM inference. Every platform below is spec'd for memory, compute, bandwidth, and model capacity so you can compare at a glance and jump to the setup guide that matches.

> [!NOTE]
>
> Last verified: 2026-10-24. Prices and availability change; confirm with the manufacturer before buying.

> [!WARNING]
>
> **GPU prices are rising due to HBM supply shortages.** Samsung, SK Hynix, and Micron have sold out their entire 2027 HBM and DRAM production capacity. Samsung's HBM prices are expected to more than double in 2027 as the industry shifts to HBM4 (\~$4.50/Gb vs HBM3E at ~$1.50/Gb). The RTX 5090 already trades above $4,000 (vs $1,999 MSRP), and NVIDIA cut RTX 50-series production by 30-40% in H1 2026 due to GDDR7 shortages. GPU lead times exceed 30 weeks. **Apple Silicon and AMD Strix Halo (unified memory) are not affected** by HBM supply constraints — they use LPDDR5X, which is sourced separately. If you plan to buy, doing so sooner rather than later avoids further price increases. Sources: [Tom's Hardware](https://www.tomshardware.com/tech-industry/artificial-intelligence/samsung-and-sk-hynix-warn-ai-driven-memory-shortages-could-last-until-2027-and-beyond-as-hbm-demand-explodes-customers-already-reserving-supply-years-ahead-while-the-wider-dram-market-begins-to-tighten), [Reuters](https://www.reuters.com/world/asia-pacific/samsung-electronics-says-hbm-account-nearly-30-industry-dram-capacity-next-year-2026-09-29/).

If you already have hardware, start at the [quick comparison table](#quick-comparison) to see where your platform sits. If you are buying, use the [decision guide](#which-hardware-should-you-choose) to narrow down by budget and use case.

## Hardware platforms

> [card] [NVIDIA DGX Spark](https://docs.tokios.com/use-cases/dgx-spark.md)
>
> **Always-on inference server for large MoE models**
>
> | Spec | Value |
> | - | - |
> | Compute | Up to 1 PFLOP (FP4) |
> | Memory | 128 GB LPDDR5x (unified) |
> | Bandwidth | 273 GB/s |
> | Interface | 256-bit |
> | Power | 65W idle, 200W load |
> | Model capacity | 70B FP8, 120B MoE at 2-bit experts |
>
> Best for teams running large MoE models 24/7 on Grace Blackwell silicon.
>
> See [DGX Spark model compatibility](https://docs.tokios.com/guides/dgx-spark-model-compatibility.md) for the full model list.

> [card] [AMD Strix Halo](https://docs.tokios.com/use-cases/strix-halo.md)
>
> **Compact workstation with up to 192 GB unified memory**
>
> | Spec | Value |
> | - | - |
> | Compute | Up to 67 TFLOPS (FP16) |
> | Memory | Up to 128 GB unified (192 GB on developer platform) |
> | Bandwidth | 256 GB/s |
> | Interface | 256-bit |
> | Power | 15-54W TDP |
> | Model capacity | 70B Q4, 30B FP16 |
>
> Available with up to 192 GB unified memory on the AMD Ryzen AI Max+ Pro 495 developer platform. Found in Framework Desktop, mini-PCs, and laptops.
>
> > [!NOTE]
> >
> > ROCm support is limited. Use the Vulkan backend with `llama.cpp` for best results on consumer hardware.

> [card] [NVIDIA RTX 4090](https://docs.tokios.com/use-cases/rtx-workstation.md)
>
> **Desktop workstation, single-user high-speed inference**
>
> | Spec | Value |
> | - | - |
> | Compute | 83 TFLOPS (FP16), 165 TFLOPS (FP8) |
> | Memory | 24 GB GDDR6X |
> | Bandwidth | 1008 GB/s |
> | Interface | PCIe 4.0 x16 |
> | Power | 450W TDP |
> | Model capacity | 30B Q4, 14B FP16 |
>
> High memory bandwidth in a consumer GPU at a lower price than the 5090. Ideal for fast single-user inference.
>
> Pairs well with vLLM for high-throughput serving or Ollama for simple setup. See [RTX workstation setup](https://docs.tokios.com/use-cases/rtx-workstation.md).

> [card] NVIDIA RTX A6000
>
> **Professional workstation, multi-user serving**
>
> | Spec | Value |
> | - | - |
> | Compute | 38 TFLOPS (FP16) |
> | Memory | 48 GB GDDR6 |
> | Bandwidth | 768 GB/s |
> | Interface | PCIe 4.0 x16 |
> | Power | 300W TDP |
> | Model capacity | 70B Q4, 30B FP16 |
>
> Double the VRAM of a 4090 at lower peak bandwidth. Fits 70B models on 1 GPU.

> [card] NVIDIA RTX 5090
>
> **Consumer flagship with 32 GB GDDR7**
>
> | Spec | Value |
> | - | - |
> | Compute | Blackwell architecture, FP4/NVFP4 support |
> | Memory | 32 GB GDDR7 |
> | Bandwidth | 1,792 GB/s |
> | Interface | PCIe 5.0 x16 |
> | Power | 575W TDP |
> | Model capacity | 70B Q4, 32B dense |
>
> \~39% faster than the RTX 4090 on 8B models (145 vs 104 tok/s). The extra 8 GB of VRAM over the 4090 runs 70B Q4 models that do not fit on 24 GB.
>
> Estimated tok/s: 7B \~162, 14B \~90, 32B dense \~45, 70B \~23. \$1,999 MSRP.

> [card] NVIDIA RTX 5080
>
> **Mid-tier Blackwell with 16 GB GDDR7**
>
> | Spec | Value |
> | - | - |
> | Compute | Blackwell architecture, FP4/NVFP4 support |
> | Memory | 16 GB GDDR7 |
> | Bandwidth | 960 GB/s |
> | Interface | PCIe 5.0 x16 |
> | Power | 360W TDP |
> | Model capacity | 30B Q4, 14B Q8 |
>
> Best for 30B models at Q4. Gemma 4 E4B at Q8: \~141 tok/s. 14B models at Q4: \~83 tok/s. \~\$1,000 MSRP.

> [card] NVIDIA RTX 5070 Ti / 5070 / 5060
>
> **Entry-level Blackwell consumer cards**
>
> | Model | Memory | Bandwidth | Price | Best for |
> | - | - | - | - | - |
> | RTX 5070 Ti | 16 GB GDDR7 | 896 GB/s | \~\$750 | 30B Q4 (slower than 5080) |
> | RTX 5070 | 12 GB GDDR7 | 672 GB/s | \~\$550 | 14B Q4 |
> | RTX 5060 Ti | 16 GB GDDR7 | 448 GB/s | \~\$450 | 30B Q4 (bandwidth-limited) |
> | RTX 5060 | 8 GB GDDR7 | 448 GB/s | \~\$350 | 7B Q4 |
>
> All RTX 50 series have native FP4 tensor cores. Software support: Ollama, llama.cpp, vLLM, SGLang, LM Studio.

> [card] NVIDIA RTX 4080 Super / 4070 Ti Super
>
> **Previous-gen Ada Lovelace mid-tier**
>
> | Model | Memory | Bandwidth | Price (used) | Best for |
> | - | - | - | - | - |
> | RTX 4080 Super | 16 GB GDDR6X | 736 GB/s | \~\$800-900 | 30B Q4 |
> | RTX 4070 Ti Super | 16 GB GDDR6X | 672 GB/s | \~\$650-750 | 30B Q4 |
> | RTX 4070 Super | 12 GB GDDR6X | 504 GB/s | \~\$500-600 | 14B Q4 |
>
> Still strong performers with full CUDA support. 14-29% slower than RTX 50 series at same VRAM tier.

> [card] NVIDIA RTX 3090 / 3080 / 3070 / 3060
>
> **Previous-gen Ampere — new life in 2026**
>
> | Model | Memory | Bandwidth | Price (used) | Best for |
> | - | - | - | - | - |
> | RTX 3090 | 24 GB GDDR6X | 936 GB/s | \~\$700-900 | 70B Q4, \~118 tok/s on 7B |
> | RTX 3080 | 10 GB GDDR6X | 760 GB/s | \~\$400-500 | 13B Q4, \~80 tok/s on 7B |
> | RTX 3070 | 8 GB GDDR6 | 448 GB/s | \~\$300-400 | 7B Q4, \~61 tok/s |
> | RTX 3060 | 12 GB GDDR6 | 360 GB/s | \~\$200-250 | 13B Q4, \~50 tok/s |
>
> Why older cards have new life in 2026:
>
> 1. **Smaller, smarter models**: Qwen3.6-35B-A3B (3B active), Phi-4-mini (3.8B), Gemma 4 E4B
>
> 2. **Better quantization**: NVFP4, MXFP4, dynamic quants preserve quality at lower bits
>
> 3. **MoE architecture**: 35B total but only 3B active = fits in 16GB
>
> 4. **CPU offloading**: llama.cpp and KTransformers spill to system RAM when VRAM is tight
>
> 5. **Used market value**: RTX 3090 at \~\$800 is the best price-per-VRAM in 2026
>
> > [!TIP]
> >
> > The RTX 3090 (24GB) is the best value in 2026 for budget builders. It runs 70B Q4 models that newer 16GB cards cannot fit.

> [card] NVIDIA RTX PRO 6000 Blackwell
>
> **Professional workstation flagship — 96 GB GDDR7**
>
> | Spec | Value |
> | - | - |
> | Compute | Blackwell architecture, FP4/NVFP4 support |
> | Memory | 96 GB GDDR7 (ECC optional) |
> | Bandwidth | 1,792 GB/s |
> | Interface | PCIe 5.0 x16 |
> | Power | 300W TDP |
> | Model capacity | 120B MoE FP8, 70B FP16 |
>
> 3 variants:
>
> * **Workstation** (\~\$16,000): Full tower, active cooling, display outputs
> * **Max-Q** (\~\$8,500): Low-profile, 4x fit in one workstation (384 GB total), enables V4.1 Flash at high precision
> * **Server** (pricing varies): Rack-mount, datacenter cooling, no display outputs
>
> See [RTX PRO Blackwell variants](https://docs.tokios.com/guides/rtx-pro-blackwell-variants.md) for the full comparison.

> [card] NVIDIA RTX PRO 5000 / 4500 / 4000
>
> **Professional Blackwell mid-tier**
>
> | Model | Memory | Bandwidth | Price | Best for |
> | - | - | - | - | - |
> | RTX PRO 5000 | 48 GB or 72 GB GDDR7 | 1,344 GB/s | \~\$5,000-7,000 | 70B FP8, 120B MoE Q4 |
> | RTX PRO 4500 | 32 GB GDDR7 | 896 GB/s | \~\$3,000-4,000 | 70B Q4, 32B dense FP16 |
> | RTX PRO 4000 | 24 GB GDDR7 | 672 GB/s | \~\$2,000-2,500 | 30B Q4, 14B FP16 |
>
> Professional drivers, ECC memory support, and ISV certifications. Full CUDA support across all backends (Ollama, vLLM, llama.cpp, LM Studio).

> [card] NVIDIA RTX 6000D
>
> **China export-compliance variant — 84 GB**
>
> | Spec | Value |
> | - | - |
> | Compute | Blackwell architecture |
> | Memory | 84 GB GDDR7 |
> | Bandwidth | 1,597 GB/s |
> | Interface | PCIe 5.0 x16 |
> | Power | 300W TDP |
> | Model capacity | 70B FP8, 120B MoE Q4 |
>
> Cut-down RTX PRO 6000 for China market compliance. 84 GB instead of 96 GB. Performance is \~12% lower than the full 96 GB variant. Available through authorized China distributors.

> [card] CMP 170HX (unlocked)
>
> **Budget hacker tier — 64 GB HBM2e for \$250-800**
>
> | Spec | Value |
> | - | - |
> | Compute | Ampere GA100, 74 SMs (unlocked) |
> | Memory | 64 GB HBM2e (unlocked from 8 GB) |
> | Bandwidth | 732-1,100 GB/s |
> | Interface | PCIe 4.0 x16 |
> | Power | 250W TDP |
> | Model capacity | 70B Q4-Q8, 120B MoE Q4 |
>
> **What is it?** NVIDIA CMP 170HX is a mining card based on the Ampere GA100 GPU. It ships artificially capped at 8 GB VRAM, but the full 64 GB HBM2e can be unlocked.
>
> **Unlock methods:**
>
> * **Vix Loader (recommended)**: No-flashing unlock via Windows registry keys. Fully reversible. Requires driver 616.56+. See [CMP 170HX unlock guide](https://docs.tokios.com/guides/cmp-170hx-unlock.md).
> * **Linux kernel patching**: For server deployments. Requires Proxmox VE or bare-metal Linux.
>
> **Performance:**
>
> * Single card: 70B models at Q4/Q8, \~40-60 tok/s
>
> * 4x CMP 170HX: 816 tok/s aggregate on GLM-5.3-Flash
>
> * 8x CMP 170HX: \~30 tok/s on GLM-5.2 at 4-bit
>
> > [!WARNING]
> >
> > **Unsupported path**: NVIDIA may patch the unlock in future drivers. No display outputs. Passive cooling requires directed airflow. Best for tinkerers and budget builders who accept the risk.

> [card] Tesla P40 / V100 (used)
>
> **Datacenter cast-offs — cheap VRAM per GB**
>
> | Model | Memory | Bandwidth | Price (used) | Best for |
> | - | - | - | - | - |
> | Tesla P40 | 24 GB GDDR5X | 346 GB/s | \~\$150-250 | 70B Q4 (slow but cheap) |
> | Tesla V100 | 32 GB HBM2 | 900 GB/s | \~\$400-600 | 70B Q4, \~70 tok/s on 7B |
>
> **Pros:** Extremely cheap per GB ($4-12/GB vs $50-170/GB for pro cards). Full CUDA support.
> **Cons:** No display outputs. Passive cooling (requires server chassis or custom fans). Older architecture (Pascal/Volta) lacks FP8/FP4 tensor cores.
>
> Best for: Budget builders who need 24+ GB VRAM and accept the tinkering overhead.

> [card] Apple Mac Studio (M4 Ultra)
>
> **Large models on macOS, updated for 2025**
>
> | Spec | Value |
> | - | - |
> | Compute | Not publicly specified |
> | Memory | Up to 192 GB unified |
> | Bandwidth | \~820 GB/s |
> | Interface | Unified memory architecture |
> | Power | 60-150W |
> | Model capacity | 120B Q4, 70B Q5\_K\_M |
>
> Runs 70B Q4 at \~18-24 tok/s. Found in Mac Studio and Mac Pro. A mid-point between M2 Ultra and M5 Ultra for large-model inference on macOS.

> [card] Apple Mac Studio (M5 Ultra)
>
> **Largest unified memory desktop — up to 512 GB**
>
> | Spec | Value |
> | - | - |
> | Compute | Not publicly specified |
> | Memory | Up to 256 GB (or 512 GB) unified |
> | Bandwidth | 1,200 GB/s |
> | Interface | Unified memory architecture |
> | Power | 60-150W |
> | Model capacity | 120B+ Q4, DeepSeek V4-Flash at native precision (256 GB config) |
>
> 50% more bandwidth than M3 Ultra. Runs DeepSeek V4-Flash-0731 at native precision (155 GB) on the 256 GB configuration. \~20-25% faster than M2 Ultra on LLM inference. Mac Studio M5 Ultra from \$5,499.

> [card] Apple Mac Studio (M5 Max)
>
> **Mid-range Apple Silicon for 70B models**
>
> | Spec | Value |
> | - | - |
> | Compute | Not publicly specified |
> | Memory | Up to 128 GB unified |
> | Bandwidth | 460-614 GB/s |
> | Interface | Unified memory architecture |
> | Power | 30-90W |
> | Model capacity | 70B Q4, 30B FP16 |
>
> Runs 70B Q4 models with room to spare. Mac Studio M5 Max from \$2,499. A strong mid-point between M4 Max and M5 Ultra for macOS inference.

> [card] [Apple Mac Studio (M2 Ultra)](https://docs.tokios.com/use-cases/mac-studio.md)
>
> **Silent operation, large models on macOS**
>
> | Spec | Value |
> | - | - |
> | Compute | 27 TFLOPS (FP16, neural engine) |
> | Memory | Up to 192 GB unified |
> | Bandwidth | 800 GB/s |
> | Interface | Unified memory architecture |
> | Power | 60-150W |
> | Model capacity | 120B Q4, 70B Q5\_K\_M |
>
> 192 GB unified memory runs 120B models silently. Superseded by M4 Ultra and M5 Ultra but still a strong option for large-model inference on macOS.
>
> Use LM Studio or Ollama with Metal acceleration. See [Mac Studio setup](https://docs.tokios.com/use-cases/mac-studio.md).

> [card] AMD Ryzen AI Max+ Pro 495
>
> **Mini PC with up to 192 GB unified memory**
>
> | Spec | Value |
> | - | - |
> | Compute | Up to 67 TFLOPS (FP16) |
> | Memory | Up to 192 GB unified (160 GB GPU-accessible) |
> | Bandwidth | 256 GB/s |
> | Interface | 256-bit |
> | Power | 15-54W TDP |
> | Model capacity | 230B-284B MoE at Q4 (MiniMax M2.7 class) |
>
> Available in Framework Desktop, Minisforum MS-S1 MAX-P495, and other mini PCs. The 192 GB configuration runs MiniMax M2.7 (230B-284B MoE) at Q4 quantization. The 160 GB GPU-accessible portion is the key advantage over standard 128 GB systems.
>
> > [!NOTE]
> >
> > ROCm support is limited. Use Vulkan backend with `llama.cpp` for best results on consumer hardware.

> [card] NVIDIA RTX Spark
>
> **Laptop form factor with unified memory**
>
> | Spec | Value |
> | - | - |
> | Compute | Blackwell architecture |
> | Memory | 64 GB or 128 GB unified |
> | Bandwidth | 300 GB/s |
> | Interface | Unified memory architecture |
> | Power | Laptop-class TDP |
> | Model capacity | 70B Q4 (128 GB config) |
>
> RTX Spark combines a 20-core Grace CPU with a Blackwell RTX GPU in a laptop form factor. Windows-native (unlike DGX Spark's Linux). Runs 120B-parameter models according to NVIDIA. Available in 64 GB and 128 GB configurations.
>
> > [!NOTE]
> >
> > RTX Spark drops the ConnectX-7 networking found in DGX Spark, so clustering is not supported. Best for single-device inference.

> [card] [Multi-GPU (2x RTX 5090 / 2x RTX 4090 / 2x A6000)](https://docs.tokios.com/guides/model-sharding.md)
>
> **Models too large for 1 GPU**
>
> | Spec | Value |
> | - | - |
> | Compute | 2x single GPU |
> | Memory | 48-96 GB total (64 GB on 2x 5090) |
> | Bandwidth | Per-GPU (limited by interconnect) |
> | Interconnect | PCIe (slow) or NVLink (fast, A6000 only) |
> | Model capacity | 70B Q5\_K\_M (TP=2), 120B MoE Q4 |
>
> Use tensor parallelism across 2 GPUs when 1 GPU is not enough. See [1 vs 2 DGX Sparks](https://docs.tokios.com/guides/one-vs-two-dgx-sparks.md) for clustering patterns.

> [card] Consumer laptop (16 GB)
>
> **7B models, offline AI, travel**
>
> | Spec | Value |
> | - | - |
> | Memory | 16 GB system RAM |
> | Runtime | Ollama, llama.cpp |
> | Model capacity | 7B Q4\_K\_M, 14B Q3\_K\_M |
>
> Run 7B models on any laptop with no GPU required. Good for offline coding assistants and travel.
>
> Pair with [Ollama](https://docs.tokios.com/guides/ollama-remote-access.md) or llama.cpp for the simplest local setup, then connect through Tokios to reach the model from your phone or another device.

> [card] [Phone / edge device](https://docs.tokios.com/use-cases/local-model-on-your-phone.md)
>
> **On-device inference, maximum privacy**
>
> | Spec | Value |
> | - | - |
> | Memory | 8-16 GB |
> | Runtime | llama.cpp, MLC-LLM, MediaPipe |
> | Model capacity | 3B Q4, 7B Q3 |
>
> Run small models directly on your phone. No network, no cloud, no latency.
>
> See [From your phone](https://docs.tokios.com/use-cases/local-model-on-your-phone.md) for how to reach a home server model from iOS and Android chat apps.

> [!NOTE]
>
> `DeepSeek-V4-Flash-0731` is a special case: its roughly `162–167 GB` weight floor exceeds most single-GPU setups. The Mac Studio M5 Ultra (256 GB config) runs it at native precision. 2x `96 GB` RTX PRO 6000 Blackwell cards also have enough aggregate VRAM, but 0731 needs community validation on that configuration. [Compare the M5 Ultra, 2x RTX PRO 6000, 2x DGX Spark, and 2x Strix Halo paths](https://docs.tokios.com/blog/deepseek-v4-flash-local-hardware.md) before buying hardware.

## Quick comparison

| Platform | Memory | Max model (Q4) | Tok/s (7B est.) | Power | Price range |
| - | - | - | - | - | - |
| **Consumer Blackwell (RTX 50 series)** | | | | | |
| RTX 5090 | 32 GB GDDR7 | 70B | \~162 | 575W | \~\$1,999 |
| RTX 5080 | 16 GB GDDR7 | 30B | \~100 | 360W | \~\$1,000 |
| RTX 5070 Ti | 16 GB GDDR7 | 30B | \~90 | 300W | \~\$750 |
| RTX 5070 | 12 GB GDDR7 | 14B | \~75 | 250W | \~\$550 |
| RTX 5060 Ti | 16 GB GDDR7 | 30B | \~70 | 200W | \~\$450 |
| RTX 5060 | 8 GB GDDR7 | 7B | \~60 | 150W | \~\$350 |
| **Consumer Ada (RTX 40 series)** | | | | | |
| RTX 4090 | 24 GB GDDR6X | 30B | \~120 | 450W | \~\$1,600-2,000 |
| RTX 4080 Super | 16 GB GDDR6X | 30B | \~95 | 320W | \~\$800-1,000 (used) |
| RTX 4070 Ti Super | 16 GB GDDR6X | 30B | \~85 | 285W | \~\$650-800 (used) |
| RTX 4070 Super | 12 GB GDDR6X | 14B | \~70 | 220W | \~\$500-600 (used) |
| **Consumer Ampere (RTX 30 series)** | | | | | |
| RTX 3090 | 24 GB GDDR6X | 30B | \~118 | 350W | \~\$700-900 (used) |
| RTX 3080 | 10 GB GDDR6X | 13B | \~80 | 320W | \~\$400-500 (used) |
| RTX 3070 | 8 GB GDDR6 | 7B | \~61 | 220W | \~\$300-400 (used) |
| RTX 3060 | 12 GB GDDR6 | 13B | \~50 | 170W | \~\$200-250 (used) |
| **Professional Blackwell (RTX PRO series)** | | | | | |
| RTX PRO 6000 Workstation | 96 GB GDDR7 | 120B MoE FP8 | \~140 | 300W | \~\$16,000 |
| RTX PRO 6000 Max-Q | 96 GB GDDR7 | 120B MoE FP8 | \~135 | 250W | \~\$8,500 |
| RTX PRO 6000 Server | 96 GB GDDR7 | 120B MoE FP8 | \~140 | 300W | varies |
| RTX PRO 5000 | 48-72 GB GDDR7 | 70B FP8 | \~100 | 250W | \~\$5,000-7,000 |
| RTX PRO 4500 | 32 GB GDDR7 | 70B Q4 | \~85 | 200W | \~\$3,000-4,000 |
| RTX PRO 4000 | 24 GB GDDR7 | 30B Q4 | \~75 | 150W | \~\$2,000-2,500 |
| RTX 6000D (China) | 84 GB GDDR7 | 70B FP8 | \~120 | 300W | varies |
| RTX A6000 (Ada) | 48 GB GDDR6 | 70B Q4 | \~70 | 300W | \~\$4,000-6,800 |
| **Budget hacker tier** | | | | | |
| CMP 170HX (unlocked) | 64 GB HBM2e | 70B Q8 | \~50-60 | 250W | \~\$250-800 |
| Tesla V100 (used) | 32 GB HBM2 | 70B Q4 | \~70 | 250W | \~\$400-600 |
| Tesla P40 (used) | 24 GB GDDR5X | 70B Q4 | \~35 | 250W | \~\$150-250 |
| **Unified memory platforms** | | | | | |
| DGX Spark | 128 GB unified | 120B MoE | \~80 | 65-200W | \~\$3,000 |
| Mac Studio M5 Ultra | 256-512 GB unified | 120B+ | \~50 | 60-150W | \~\$5,499+ |
| Mac Studio M4 Ultra | 192 GB unified | 120B | \~45 | 60-150W | \~\$4,500-7,000 |
| Mac Studio M2 Ultra | 192 GB unified | 120B | \~40 | 60-150W | \~\$4,000-7,000 |
| Mac Studio M5 Max | 128 GB unified | 70B | \~35 | 30-90W | \~\$2,499+ |
| Strix Halo (128 GB) | 128 GB unified | 70B | \~40 | 15-54W | \~\$1,500-3,000 |
| Strix Halo (192 GB) | 192 GB unified | 120B Q4 | \~40 | 15-54W | \~\$3,000-5,000 |
| **Multi-GPU configurations** | | | | | |
| 2x RTX 5090 | 64 GB total | 120B MoE (TP=2) | \~162 per GPU | 1,150W | \~\$4,000 |
| 2x RTX 4090 | 48 GB total | 70B (TP=2) | \~120 per GPU | 900W | \~\$3,200-4,000 |
| 2x RTX PRO 6000 | 192 GB total | V4.1 Flash (TP=2) | \~140 per GPU | 600W | \~\$32,000 |
| 4x RTX PRO 6000 Max-Q | 384 GB total | V4.1 Flash (TP=4) | \~135 per GPU | 1,000W | \~\$34,000 |
| 2x DGX Spark | 256 GB total | V4.1 Flash (TP=2) | \~38-73 | 400W | \~\$6,000 |
| 4x DGX Spark | 512 GB total | V4.1 Flash (TP=4) | \~87-125 | 800W | \~\$12,000 |
| 8x CMP 170HX | 512 GB HBM2e | GLM-5.2 (TP=8) | \~30 | 2,000W | \~\$2,000-6,400 |
| 2x A6000 | 96 GB total | 120B MoE | \~70 per GPU | 600W | \~\$8,000-14,000 |
| **Entry-level** | | | | | |
| Consumer laptop | 16 GB RAM | 14B | \~15-30 | 15-65W | \~\$800-1,500 |
| Phone / edge | 8-16 GB | 7B | \~10-20 | 3-8W | \~\$500-1,200 |

> [!NOTE]
>
> Token speeds are rough estimates for 7B Q4 inference and vary by backend, quantization, and context length. Benchmark your own setup with [Test your deployment](https://docs.tokios.com/guides/benchmark-your-deployment.md).

## Runtime compatibility

Not every runtime works on every platform. Here is what each hardware tier supports today.

| Platform | Ollama | llama.cpp | vLLM | LM Studio | Notes |
| - | - | - | - | - | - |
| **Consumer GPUs (RTX 30/40/50)** | | | | | |
| RTX 5090/5080/5070/5060 | Yes | Yes (CUDA) | Yes | Yes | Full CUDA support; FP4/NVFP4 on Blackwell |
| RTX 4090/4080/4070 | Yes | Yes (CUDA) | Yes | Yes | Full CUDA support across all backends |
| RTX 3090/3080/3070/3060 | Yes | Yes (CUDA) | Yes | Yes | Ampere architecture; no FP4 support |
| **Professional GPUs** | | | | | |
| RTX PRO 6000/5000/4500/4000 | Yes | Yes (CUDA) | Yes | Yes | Professional driver; ECC memory support |
| RTX A6000 (Ada/Ampere) | Yes | Yes (CUDA) | Yes | Yes | Professional driver; NVLink on Ada generation only |
| CMP 170HX (unlocked) | Yes | Yes (CUDA) | Yes | No | Requires unlock; no display outputs |
| Tesla P40/V100 | Yes | Yes (CUDA) | Yes | No | No display outputs; passive cooling |
| **Unified memory platforms** | | | | | |
| DGX Spark | Yes | Yes | Yes | No | ARM64 Linux; vLLM needs DGX-specific wheels |
| Mac Studio (M2/M4/M5) | Yes | Yes (Metal) | No | Yes | Metal acceleration; vLLM has no Apple Silicon build |
| Strix Halo | Yes | Yes (Vulkan) | Limited | Yes | ROCm is unreliable; prefer Vulkan backend |
| **Entry-level** | | | | | |
| Consumer laptop | Yes | Yes (CPU) | No | Yes | CPU-only inference; expect 15-30 tok/s on 7B |
| Phone / edge | No | Yes | No | No | On-device only; use llama.cpp, MLC-LLM, or MediaPipe |

> [!TIP]
>
> If your hardware has limited runtime support, start with Ollama or llama.cpp — they have the broadest platform coverage.

## VRAM tier decision matrix

Not sure what fits your GPU? Start here. This table maps VRAM to the largest model class you can run at practical quantization levels.

| VRAM tier | Example GPUs | Best model class | Example models | Expected tok/s |
| - | - | - | - | - |
| **8 GB** | RTX 3070, 4060, 5060 | 7B models | Llama 3.1 8B, Qwen3 8B, Phi-4-mini | 50-75 tok/s |
| **10-12 GB** | RTX 3080, 3060, 4070, 5070 | 13B models | Qwen3 14B, Gemma 3 12B, Mistral 12B | 45-70 tok/s |
| **16 GB** | RTX 4080S, 5080, PRO 4500 | 30B models / 35B MoE | Qwen3.6-35B-A3B (3B active), Gemma 4 E4B | 70-120 tok/s |
| **24 GB** | RTX 3090, 4090, PRO 4000 | 70B models at Q4 | Qwen3 70B, Llama 3.1 70B | 20-120 tok/s |
| **32 GB** | RTX 5090, PRO 4500 | 70B models at Q5-Q8 | Qwen3 70B Q8, higher quality | 25-160 tok/s |
| **48-72 GB** | RTX PRO 5000, A6000 | 70B models at FP8 | Qwen3 70B FP8, better quality | 70-100 tok/s |
| **96 GB** | RTX PRO 6000 | 120B MoE at FP8 | Qwen3.5-122B, GLM-4.5-Air | 30-140 tok/s |
| **128 GB** | DGX Spark, Mac Studio M5 Max | 120B+ models | Qwen3.5-122B, DeepSeek V4-Flash (2x Spark) | 35-80 tok/s |
| **192-256 GB** | Mac Studio M5 Ultra, Strix Halo 192GB | 200B+ models | Qwen3 235B, MiniMax M2.7 | 30-50 tok/s |
| **384-512 GB** | 4x RTX PRO 6000 Max-Q, 4x DGX Spark | Frontier models | DeepSeek V4.1 Flash (763B), GLM-5.2 | 10-125 tok/s |

> [!TIP]
>
> **MoE models punch above their weight.** A 35B MoE model like Qwen3.6-35B-A3B has 35B total parameters but only 3B active per token. It fits in 16GB and runs at 120+ tok/s, but performs closer to a 35B dense model. Always check active parameter count, not just total.

> [!NOTE]
>
> **Frontier models need multi-GPU or unified memory.** DeepSeek V4.1 Flash (763B total params) requires at least 2x DGX Spark (256GB) or 2x RTX PRO 6000 (192GB) at mixed precision. See [DeepSeek V4.1 Flash on 2 DGX Sparks](https://docs.tokios.com/blog/deepseek-v41-flash-dgx-spark.md) for the full breakdown.

## Power and form factor

Local inference hardware ranges from a phone in your pocket to a rack-mount server. Power draw and noise matter if the box lives on your desk or in a shared office.

| Platform | Typical power | Form factor | Noise | Always-on? |
| - | - | - | - | - |
| DGX Spark | 65-200W | Small desktop | Low fan | Yes |
| RTX 5090 | 575W | Full tower | Loud under load | Possible |
| RTX 4090 | 450W | Full tower | Loud under load | Possible |
| RTX A6000 | 300W | Workstation tower | Moderate | Yes |
| Mac Studio (M2/M4/M5) | 30-150W | Compact desktop | Near-silent | Yes |
| Strix Halo | 15-54W | Mini PC / desktop | Quiet | Yes |
| Consumer laptop | 15-65W | Laptop | Varies | No |
| Phone / edge | 3-8W | Handheld | Silent | No |

If the machine runs 24/7 in a shared space, prioritize low idle power and quiet cooling. Mac Studio and DGX Spark are the best always-on options for office environments.

## Scaling beyond 1 machine

When 1 box is not enough, you have 3 options:

1. **Add a second GPU** in the same machine using tensor parallelism (TP=2). Works well with 2x RTX 5090 (64 GB total), 2x RTX 4090, or 2x A6000. See [Model sharding](https://docs.tokios.com/guides/model-sharding.md) for the configuration.
2. **Cluster multiple nodes** — for example, 2 DGX Sparks connected over the network can serve selected distributed models whose weights, cache, and runtime overhead fit below 256 GB. See [1 vs 2 DGX Sparks](https://docs.tokios.com/guides/one-vs-two-dgx-sparks.md) for the trade-offs.
3. **Offload to a remote server** — run the model on a dedicated GPU server and reach it through Tokios from any client. See [vLLM remote access](https://docs.tokios.com/guides/vllm-remote-access.md) and [Ollama remote access](https://docs.tokios.com/guides/ollama-remote-access.md).

For team serving, the key question is whether you need more throughput or more model capacity. Additional GPUs can improve either, but scaling depends on the model, parallelism strategy, and interconnect. Measure the full workload instead of assuming linear throughput.

## Which hardware should you choose?

> [accordion] Budget under \$500
>
> **RTX 3060** (12 GB, \~$200-250 used) runs 13B models at Q4 with ~50 tok/s. **RTX 3070** (8 GB, ~$300-400 used) runs 7B models at \~61 tok/s. Both are entry points into local LLM with full CUDA support. For the absolute minimum, a **consumer laptop** with 16 GB RAM runs 7B models through Ollama or llama.cpp at 15-30 tok/s.

> [accordion] Budget \$500-1,000
>
> **RTX 3090** (24 GB, \~$700-900 used) is the best value in 2026 for budget builders. It runs 70B Q4 models that newer 16GB cards cannot fit, with ~118 tok/s on 7B. **RTX 3080** (10 GB, ~$400-500 used) runs 13B models at \~80 tok/s. If you accept the tinkering overhead, a **CMP 170HX** (64 GB HBM2e after unlock, \~$250-800) or **Tesla P40** (24 GB, ~$150-250 used) offer extreme VRAM per dollar — see [Budget hacker tier](#budget-hacker-tier) below.

> [accordion] Budget hacker tier (\$250-800)
>
> **CMP 170HX** (64 GB HBM2e after unlock) is the most compelling budget story in local LLM. Originally an Ampere GA100 mining card capped at 8 GB, it unlocks to full 64 GB with 732-1,100 GB/s bandwidth. Single card runs 70B Q4-Q8 at \~40-60 tok/s. 4x CMP 170HX achieves 816 tok/s aggregate on GLM-5.3-Flash. **Tesla V100** (32 GB HBM2, \~$400-600 used) runs 70B Q4 at ~70 tok/s. **Tesla P40** (24 GB GDDR5X, ~$150-250 used) is slower (\~35 tok/s on 7B) but extremely cheap per GB.
>
> > [!WARNING]
> >
> > **Unsupported path**: These cards require tinkering. No display outputs. Passive cooling needs directed airflow. CMP 170HX unlock may break with future NVIDIA drivers. Best for tinkerers who accept the risk. See [CMP 170HX unlock guide](https://docs.tokios.com/guides/cmp-170hx-unlock.md) for the Vix Loader method.

> [accordion] Best single-GPU value (\$1,000-2,000)
>
> **RTX 5090** (32 GB, $1,999 MSRP) is the sweet spot. 32 GB GDDR7 fits 70B Q4 models that the 4090's 24 GB cannot, with ~39% faster inference on small models (~162 tok/s on 7B). **RTX 4090** (24 GB, ~$1,600-2,000) remains strong if you find one at lower price — highest consumer memory bandwidth at 1008 GB/s. **RTX 5080** (16 GB, \~\$1,000) is the mid-tier choice for 30B models at \~100 tok/s.

> [accordion] Best for large models (70B+)
>
> **Mac Studio M5 Ultra** (256-512 GB unified, \~$5,499+) fits the largest models of any desktop — 120B+ at Q4, and DeepSeek V4-Flash at native precision on the 256 GB config. **Mac Studio M4 Ultra** (192 GB) and **M2 Ultra** (192 GB) also run 120B models silently. **DGX Spark** (128 GB, ~$3,000) runs 70B FP8 natively and 120B MoE models. **Strix Halo** (up to 192 GB unified, \~$1,500-5,000) is a compact alternative with Vulkan-based inference. **RTX PRO 6000** (96 GB, ~$8,500-16,000) fits 120B MoE at FP8 on a single card.

> [accordion] Best for frontier models (200B+)
>
> **DeepSeek V4.1 Flash** (763B total params) requires multi-GPU or large unified memory. Options:
>
> * **2x DGX Spark** (256 GB total, \~\$6,000): Mixed FP8+MXFP4 precision, 38-73 tok/s. Community-tested. See [V4.1 Flash on 2 DGX Sparks](https://docs.tokios.com/blog/deepseek-v41-flash-dgx-spark.md).
> * **Mac Studio M5 Ultra 512 GB** (\~\$11,300+): V4.1 Flash fits at Q2\_K-Q4 (\~246-402 GiB), \~10-20 tok/s.
> * **4x RTX PRO 6000 Max-Q** (384 GB total, \~\$34,000): V4.1 Flash at high precision, \~135 tok/s per GPU.
> * **4x DGX Spark** (512 GB total, \~\$12,000): V4.1 Flash at NVFP4, 87-125 tok/s.
> * **8x CMP 170HX** (512 GB HBM2e, \~\$2,000-6,400): Budget path to 512 GB, \~30 tok/s on GLM-5.2.
>
> **GLM-5.2** (753B params) needs at least 4x DGX Spark at Q4 (\~377 GB weight floor). See [GLM-5.2 feasibility](https://docs.tokios.com/guides/dgx-spark-model-compatibility.md#glm-52-feasibility).

> [accordion] Best for team serving
>
> **2x RTX PRO 6000** (192 GB total with NVLink, \~$32,000) serves V4.1 Flash at FP8 with ~140 tok/s per GPU. **2x RTX 5090** (64 GB total, ~$4,000) is a faster consumer alternative for 120B MoE models. **2x DGX Spark** (256 GB total, \~$6,000) is a turnkey alternative with 128 GB unified memory per node and ConnectX-7 RoCE networking. For budget team serving, **4x CMP 170HX** (256 GB HBM2e total, ~$1,000-3,200) offers extreme value if you accept the unsupported path.

> [accordion] Silent / fanless
>
> **Mac Studio** (any Apple Silicon) runs at 30-150W with near-silent cooling. It is the quietest option for always-on inference in an office or home environment. The M5 Ultra tops out at 512 GB unified memory for the largest models. **DGX Spark** (65-200W) is also quiet and designed for always-on server use.

> [accordion] AMD ecosystem
>
> **Strix Halo** with up to 192 GB unified memory on the Ryzen AI Max+ Pro 495 developer platform. Available in Framework Desktop, mini-PCs, and laptops. Use `llama.cpp` with the Vulkan backend — ROCm support is limited on consumer hardware. See the [Strix Halo setup guide](https://docs.tokios.com/use-cases/strix-halo.md) for details.

> [accordion] Maximum portability
>
> **Phone / edge device** for on-device inference with no network required. Run 3B-7B models through llama.cpp, MLC-LLM, or MediaPipe.

## How memory maps to model size

The single biggest constraint for local LLM inference is memory. Here is how common model sizes map to the RAM or VRAM you need at different quantization levels.

| Model size | FP16 | Q8 | Q4\_K\_M | Q2\_K |
| - | - | - | - | - |
| 7B | 14 GB | 7 GB | 4 GB | 3 GB |
| 14B | 28 GB | 14 GB | 8 GB | 5 GB |
| 30B | 60 GB | 30 GB | 18 GB | 12 GB |
| 70B | 140 GB | 70 GB | 40 GB | 28 GB |
| 120B MoE | 240 GB | 120 GB | 70 GB | 45 GB |
| 2.8T MoE (Kimi K3) | 5.6 TB | 2.8 TB | 1.4 TB | 1.05 TB |

> [!NOTE]
>
> Always leave 2-4 GB free for the KV cache and runtime overhead. A 70B Q4\_K\_M model that "fits" in 40 GB of VRAM still needs headroom for context — plan for 44-48 GB total. See [KV cache and context](https://docs.tokios.com/concepts/kv-cache-and-context.md) for the math.

> [!WARNING]
>
> Frontier-scale models like Kimi K3 (2.8T parameters) need \~1.4 TB even at 4-bit quantization — no single desktop or prosumer setup fits them. Moonshot recommends a supernode with 64+ accelerators. See [Kimi K3 hardware requirements](https://docs.tokios.com/models/kimi-k3.md) for the full picture.

See [Find a model that fits your hardware](https://docs.tokios.com/guides/model-fit.md) for a deeper look at how to match model size to your hardware, and [Quantization concepts](https://docs.tokios.com/concepts/quantization.md) for what the quant levels mean.

## Common hardware mistakes

* **Buying for TFLOPS, not memory.** A 4090 or 5090 has more raw compute than an A6000, but the A6000's 48 GB VRAM fits 70B models that the 4090's 24 GB cannot. Even the 5090's 32 GB is tight for 70B at higher quantizations. Memory determines what runs; TFLOPS determines how fast.
* **Ignoring KV cache overhead.** A model that "fits" your VRAM at load time may still OOM during inference because the KV cache grows with context length. Budget 2-4 GB extra.
* **Assuming multi-GPU is free.** Tensor parallelism across 2 GPUs requires fast interconnect (NVLink or PCIe 4.0 x16). Over PCIe, expect 10-20% throughput loss versus a single larger GPU.
* **Overlooking power and cooling.** A 575W RTX 5090 or 450W RTX 4090 in a small office generates real heat and noise. If the box runs 24/7, factor in the electricity cost and acoustic impact.
* **Skipping the backend check.** Not every runtime supports every GPU. Confirm your chosen backend works on your hardware before buying — see [Runtime compatibility](#runtime-compatibility) above.

## Next steps

> [card] [Choose a local model](https://docs.tokios.com/guides/choose-a-local-model.md)
>
> Pick a model and backend based on your workflow, hardware, and latency needs.

> [card] [Model fit](https://docs.tokios.com/guides/model-fit.md)
>
> Calculate which model sizes and quantizations fit your GPU or unified memory.

> [card] [Model sharding](https://docs.tokios.com/guides/model-sharding.md)
>
> Split models across multiple GPUs when 1 GPU is not enough.

> [card] [DGX Spark model compatibility](https://docs.tokios.com/guides/dgx-spark-model-compatibility.md)
>
> See which models work on DGX Spark and at what quantization levels.

> [card] [Strix Halo model compatibility](https://docs.tokios.com/guides/strix-halo-model-compatibility.md)
>
> Check which models work on AMD Strix Halo with Vulkan and ROCm backends.

> [card] [Test your deployment](https://docs.tokios.com/guides/benchmark-your-deployment.md)
>
> Compare latency and throughput across deployments once traffic is flowing.

> [card] [Supported backends](https://docs.tokios.com/setup/supported-backends.md)
>
> Configure Ollama, LM Studio, llama.cpp, vLLM, or SGLang as your upstream.

> [card] [Quickstart](https://docs.tokios.com/quickstart.md)
>
> Go from zero to a working connector, deployment, and API key.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.