> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Find a model that fits your hardware

> Interactive compatibility checker: find which models run on your GPU or unified-memory system, with VRAM sizing math and quantization estimates.

Find which open-weight models run on your hardware, or which hardware can run your target model. Toggle between "I have this hardware" and "I want to run this model" modes, adjust quantization and context length, and see the memory math for every combination.

> [!TIP]
>
> Found a model that fits? [Register it with Tokios](https://docs.tokios.com/setup/register-model.md) to reach it as `https://api.tokios.com` from any OpenAI- or Anthropic-compatible client.

Model fit finder

Interactive model-fit finder — select hardware or model, adjust quantization and context.

## How it works

The widget estimates memory requirements using:

```text theme={"theme":{"light":"github-light","dark":"css-variables"}}
total = (params × bytes-per-param) + KV cache + 5% overhead
```

A model "fits" if the total estimate is within the hardware's usable memory:

| Status | Meaning |
| - | - |
| **Fits comfortably** | Uses ≤ 85% of available memory |
| **Fits (tight)** | Uses 85–95% of available memory |
| **Very tight** | Uses 95–100%, short context only |
| **Doesn't fit** | Exceeds available memory |

MoE models (like Qwen3.8-2.4T-A95B or DeepSeek-V3) load **all** expert weights into memory, so total parameter count determines memory usage — even though only the active subset runs per token.

## The sizing formula

Model weights take a fixed number of bytes per parameter, and that byte count depends entirely on precision:

| Precision | Bytes per parameter | Example: `8B` model |
| - | - | - |
| FP32 | `4` | \~32 GB |
| FP16 / BF16 | `2` | \~16 GB |
| 8-bit | `1` | \~8 GB |
| 4-bit (`Q4`) | `~0.5` | \~4 GB |

Raw weight size is not the whole story. You also need memory for the KV cache, activations, and framework overhead. The widget above adds a modest KV cache estimate (at the selected context length) plus a **5% overhead** factor on weights and KV cache combined:

```text theme={"theme":{"light":"github-light","dark":"css-variables"}}
VRAM(GB) ≈ (weights + KV cache) × 1.05
```

For quick mental math without the widget, a **`×1.2`** multiplier on raw weight size alone is a reasonable shortcut — it bakes in a rough KV cache estimate and overhead in one step, and is what the worked examples below use.

**Worked examples** at `~Q4` (4-bit), including the `~1.2` overhead:

| Model size | Estimated VRAM | Note |
| - | - | - |
| `7–8B` | `~4.5–5.5 GB` | |
| `13–14B` | `~8–10 GB` | |
| `30–32B` | `~19–22 GB` | |
| `70B` | `~42–48 GB` | Tight on `48 GB`, comfortable on `64 GB+` unified memory |
| `120B` MoE (for example `gpt-oss-120b`, `MXFP4`) | `~65–80 GB` | Needs `96 GB+` unified memory or multiple GPUs |

For reference, higher precision scales the same models far past what a single consumer card can hold: `70B` at FP16 is about `140 GB`, and `120B` at FP16 is about `230 GB`.

Hugging Face documents a similar pattern across model sizes: for example, about `29 GB` at `bf16`, `~15 GB` at 8-bit, and `~9.5 GB` at 4-bit for a large model. See the [Hugging Face guide to optimizing LLMs](https://huggingface.co/docs/transformers/llm_tutorial_optimization) and the [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) for the underlying numbers — `GGUF` `Q4_K_M` uses mixed precision that averages out to about `0.5–0.56` bytes per parameter.

Ternary weight compression goes further than any standard quantization. [Bonsai 2 27B](https://docs.tokios.com/models/qwen3.8.md#bonsai-2-27b-ternary-compression) stores every weight as `-1`, `0`, or `+1` — roughly 1.58 bits per weight — shrinking Qwen3.8-27B from 54 GB to 5.9 GB. That fits under 6 GB of VRAM on a consumer GPU. The calculator above does not model ternary format directly, since it is a training-time technique rather than a post-training quantization you can apply to any model.

> [!NOTE]
>
> These formulas estimate weight size. The actual number that matters is the `GGUF` file size listed on Hugging Face for the exact quantization you plan to run, plus the KV cache for your target context length. Always check that before committing to hardware.

## GPU VRAM at a glance

| Card / chip | Memory | Type |
| - | - | - |
| RTX 5090 | `32 GB` | GDDR7 dedicated, \~1792 GB/s |
| RTX 5080 | `16 GB` | GDDR7 dedicated, \~960 GB/s |
| RTX 5070 | `12 GB` | GDDR7 dedicated, \~672 GB/s |
| RTX 5070 Ti | `16 GB` | GDDR7 dedicated, \~896 GB/s |
| RTX 5060 Ti | `16 GB` | GDDR7 dedicated, \~448 GB/s |
| RTX 5060 | `8 GB` | GDDR7 dedicated, \~448 GB/s |
| RTX 4090 | `24 GB` | GDDR6X dedicated, \~1008 GB/s |
| RTX 4080 Super | `16 GB` | GDDR6X dedicated, \~736 GB/s |
| RTX 4070 Ti Super | `16 GB` | GDDR6X dedicated, \~672 GB/s |
| RTX 3090 | `24 GB` | GDDR6X dedicated, \~936 GB/s |
| RTX 3080 | `10 GB` | GDDR6X dedicated, \~760 GB/s |
| RTX 3060 | `12 GB` | GDDR6 dedicated, \~360 GB/s |
| RTX PRO 6000 | `96 GB` | GDDR7 ECC dedicated, \~1792 GB/s |
| RTX PRO 5000 | `48-72 GB` | GDDR7 ECC dedicated, \~1344 GB/s |
| RTX PRO 4500 | `32 GB` | GDDR7 ECC dedicated, \~896 GB/s |
| RTX PRO 4000 | `24 GB` | GDDR7 ECC dedicated, \~672 GB/s |
| RTX 6000 Ada | `48 GB` | GDDR6 ECC dedicated |
| RTX A6000 | `48 GB` | GDDR6 ECC dedicated (Ampere) |
| CMP 170HX (unlocked) | `64 GB` | HBM2e dedicated, \~732-1100 GB/s |
| Tesla V100 (used) | `32 GB` | HBM2 dedicated, \~900 GB/s |
| Tesla P40 (used) | `24 GB` | GDDR5X dedicated, \~346 GB/s |
| Apple M5 Ultra | `256-512 GB` | Unified, \~1200 GB/s |
| Apple M4 Ultra | `192 GB` | Unified, \~820 GB/s |
| Apple M4 Max | `36-128 GB` | Unified, up to \~546 GB/s |
| Apple M3 Max | `36-128 GB` | Unified, \~400 GB/s |
| Apple M2 Ultra | `192 GB` | Unified, \~800 GB/s |
| Ryzen AI Max+ 395 (Strix Halo) | up to `128 GB` (`≤96 GB` assignable as VRAM) | Unified LPDDR5X, \~256 GB/s |
| Ryzen AI Max+ Pro 495 | up to `192 GB` (`160 GB` GPU-accessible) | Unified LPDDR5X, \~256 GB/s |

Sources: [NVIDIA GeForce RTX 50 series](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/), [NVIDIA professional GPUs](https://www.nvidia.com/en-us/design-visualization/rtx-6000/), [Wikipedia: GeForce RTX 40 series](https://en.wikipedia.org/wiki/GeForce_RTX_40_series), [Wikipedia: GeForce RTX 30 series](https://en.wikipedia.org/wiki/GeForce_RTX_30_series), [AMD Ryzen AI Max](https://www.amd.com/en/products/processors/laptop/ryzen-pro/ai-max-pro-300-series.html), Apple technical specifications, [CMP 170HX community benchmarks](https://docs.tokios.com/guides/cmp-170hx-unlock.md).

Dedicated GDDR memory (the NVIDIA cards above) is fast and fixed in size — you cannot add more later. Unified memory (Apple Silicon, Strix Halo) shares one pool between CPU and GPU, so it can be much larger, but it runs at lower bandwidth than a dedicated card. HBM2e (CMP 170HX, Tesla V100) offers datacenter-class bandwidth at mining-card prices.

## What fits where

The table below estimates comfort level at `~Q4` quantization for a modest context length. Treat every entry as an **estimate** — verify against the actual `GGUF` file size plus KV cache before you commit.

| VRAM tier | Comfortable | Tight | Won't fit |
| - | - | - | - |
| `12 GB` (RTX 3060) | `7–8B` | `13–14B` | `30B+` |
| `16 GB` (RTX 5080 / 4080) | `7–14B` | `30–32B` (short context) | `70B+` |
| `24 GB` (RTX 4090 / 3090) | `7–32B` | `32B` at long context | `70B+` |
| `48 GB` (RTX 6000 Ada / A6000) | `7–32B`, `70B` possible | `70B` (tight, short context) | `120B+` |
| `64 GB` unified | `7–70B` | `70B` at long context | `120B+` usually |
| `96–128 GB` unified (M3/M4 Max, Strix Halo `≤96 GB` assignable) | `7–70B`, `120B` MoE | `120B` at long context | `200B+` dense |
| `256 GB+` multi-GPU | `120B` MoE, `552B` MoE (DeepSeek-V4.1-Flash, Q4 \~350 GB) | `552B` MoE at long context | `800B+` |
| `512 GB+` multi-device | `552B` MoE, `816B` MoE (Muse Spark Open, Q4 \~500 GB) | `816B` MoE at long context | |

Ground truth is always the `GGUF` file size on Hugging Face for the quantization you choose, plus the KV cache your context length requires.

> [!TIP]
>
> Most 30B-class agent models are MoE and need far more than their active-parameter count in memory. [Muse Glimmer 30B](https://docs.tokios.com/models/muse-glimmer.md) is dense instead, so the whole model — not just an active slice — fits in `18 GB` at 4-bit, comfortably inside the `24 GB` tier above.

> [!TIP]
>
> `DeepSeek-V4-Flash-0731` has a roughly `162–167 GB` weight floor in the published lossless GGUF and fused-weight formats. It needs a multi-device setup with measured KV-cache headroom; see [DeepSeek V4 Flash local hardware](https://docs.tokios.com/blog/deepseek-v4-flash-local-hardware.md) for the supported, community-tested, and experimental paths.

> [card] [Picked a model? Connect it](https://docs.tokios.com/quickstart.md)
>
> One authenticated endpoint at `https://api.tokios.com` — no inbound ports, about 5 minutes.

## Why unified memory loads big models but runs them slower

Loading a model only requires enough memory capacity to hold the weights. Running it fast requires enough memory bandwidth to keep feeding the compute units, because single-stream decoding is **memory-bandwidth-bound** — each token generated has to read the model's weights (or, for a mixture-of-experts model, the active experts) out of memory. Prefill (processing the prompt) is compute-bound instead, so it behaves differently.

This is why the two memory types trade off differently:

* **Dedicated GDDR** (RTX 4090, 5090, and similar): high bandwidth (roughly `1000–1800 GB/s`), but capacity is fixed by the card.
* **Unified memory** (Apple Silicon, Strix Halo): much larger capacity (up to `128 GB`), but lower bandwidth (roughly `256–546 GB/s`).

A `96 GB` unified-memory machine can *load* a `120B` model that would never fit on any single consumer GPU. It will still generate tokens more slowly than a dedicated GPU running a model that fits within the GPU's own VRAM, because it has less bandwidth to move weights through per token.

Do not rely on published tok/s numbers from other setups — hardware, driver versions, quantization, context length, and batch size all shift the result. Measure your own deployment with [`llama-bench`](https://github.com/ggml-org/llama.cpp) and see [Test your deployment](https://docs.tokios.com/guides/benchmark-your-deployment.md) for a repeatable way to compare deployments once your model is running.

## Next steps

> [card] [Get started with Tokios](https://docs.tokios.com/quickstart.md)
>
> Once a model fits your hardware, Tokios turns it into a stable API endpoint you can call from anywhere.

> [card] [Pick the right model for your task](https://docs.tokios.com/guides/model-by-task.md)
>
> Sizing narrows down what fits. Task fit narrows down which family to load in the first place.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.