> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# How to read a model name

> Decode open-weight model names, quantization labels, and architecture suffixes. Know what Q4_K_M, MoE, and -Instruct mean before you download.

## Why model names matter

Browse Hugging Face or Ollama's model library and you'll see names like `Qwen/Qwen3-235B-A22B-Instruct` or `bartowski/Llama-3.3-70B-Instruct-GGUF`. These aren't random strings — every segment encodes a decision you'll need to make before downloading.

The name tells you:

* **How big the model is** — and therefore how much VRAM or system memory it needs.
* **What it was trained for** — general chat, coding, reasoning, vision, or raw text completion.
* **How it was compressed** — and what quality you're trading off against size.
* **Who made it** — and whether the weights are original or a third-party repackaging.

Learning to read these conventions takes 5 minutes and saves you from downloading a 70B model that doesn't fit your GPU, or wasting VRAM on a base model that can't follow instructions.

## The anatomy of a model name

Most open-weight model names follow a pattern:

```
Organization / ModelName - Size - TrainingStage - Quantization
```

Not every name includes every segment, and publishers vary the separators — some use hyphens, others use slashes or nothing. But you can usually identify these parts:

| Segment | Example | What it tells you |
| - | - | - |
| **Organization** | `Qwen`, `mistralai`, `TheBloke` | Who published or quantized the model |
| **Model family** | `Qwen3`, `Mistral`, `Llama-2` | Architecture and generation |
| **Size** | `7B`, `70B`, `235B-A22B` | Parameter count (and active parameters for MoE) |
| **Training stage** | `Instruct`, `Chat`, `Base` | How the model was fine-tuned |
| **Quantization / format** | `Q4_K_M`, `GGUF`, `GPTQ` | Weight precision and file format |

Real examples, parsed segment by segment:

**`Qwen/Qwen3-235B-A22B-Instruct`**

| Segment | Value | Meaning |
| - | - | - |
| Organization | `Qwen` | Alibaba's Qwen team |
| Model family | `Qwen3` | Third generation of the Qwen architecture |
| Size | `235B-A22B` | 235B total parameters, 22B active (MoE) |
| Training stage | `Instruct` | Instruction-tuned for chat |

**`mistralai/Mistral-7B-Instruct-v0.3`**

| Segment | Value | Meaning |
| - | - | - |
| Organization | `mistralai` | Mistral AI |
| Model family | `Mistral` | Mistral architecture |
| Size | `7B` | 7B parameters (dense) |
| Training stage | `Instruct` | Instruction-tuned for chat |
| Version | `v0.3` | Third iteration |

**`TheBloke/Llama-2-7B-Chat-GGUF`** — TheBloke (third-party quantizer), Llama 2, 7B params, chat-tuned, GGUF format.

> [!NOTE]
>
> When the organization is a third-party quantizer like `TheBloke`, `bartowski`, or `unsloth`, the model itself was trained by someone else — the repo contains a repackaged or quantized version of the original weights.

## Size labels decoded

The number before `B` is the parameter count in billions. It's the single biggest driver of both capability and memory requirements.

| Size | Rough VRAM at `Q4_K_M` | Typical use |
| - | - | - |
| `3B` | 2 GB | Edge devices, phones, CPU inference |
| `7B`–`8B` | 4–5 GB | General chat on a single consumer GPU |
| `13B`–`14B` | 8–9 GB | Stronger reasoning, still single-GPU |
| `30B`–`34B` | 18–20 GB | High quality on a 24 GB GPU |
| `70B`–`72B` | 38–40 GB | Frontier-tier on workstation or dual-GPU |
| `405B` | 200 GB+ | Multi-GPU server only |

### MoE models: two numbers

MoE (mixture-of-experts) models list both total and active parameters, written as `235B-A22B` or `30B-A3B`:

* The first number is the **total** parameter count — what you need to store in memory or on disk.
* The second number (after `A`) is the **active** parameter count — how many parameters run per token.

A `235B-A22B` model needs enough memory to hold 235B parameters, but only computes with 22B at a time, so it generates tokens much faster than a dense 235B model. The trade-off: MoE models are larger on disk and in memory than a dense model of the same active size, but faster at inference.

> [!NOTE]
>
> A `235B-A22B` MoE model roughly matches a dense 22B model in per-token speed, but you still need VRAM or system RAM to store all 235B parameters. At `Q4_K_M`, that's roughly 120 GB — far more than a dense 22B model's 13 GB.

### How parameter count maps to capability

More parameters generally means more capability, but the relationship isn't linear. A 70B model isn't 10x better than a 7B model — it's noticeably better at reasoning, nuance, and complex tasks, but a well-tuned 7B model can still handle everyday chat and simple coding tasks well. The right size depends on your hardware and your task, not just a bigger-is-better rule.

## Architecture suffixes

Publishers append suffixes to signal architecture variants beyond the base model. These affect what the model can do and what hardware it needs.

> [accordion] Dense vs MoE
>
> A **dense** model uses every parameter on every token. An **MoE** (mixture-of-experts) model routes each token through a subset of "expert" subnetworks. MoE models are larger in total parameters but faster per token because only the active parameters compute. If a model name includes two size numbers (like `235B-A22B`), it's MoE.

> [accordion] VL (vision-language)
>
> Models labeled `-VL` accept images or video in addition to text. Examples: `Qwen3-VL-32B-Instruct`, `gemma-4-12b-it`. VL models need extra VRAM for the vision encoder on top of the language model weights.

> [accordion] Coder
>
> Models labeled `-Coder` are fine-tuned or trained primarily on code. They typically outperform general models of the same size on programming tasks but may be weaker at open-ended conversation. Example: `Qwen3-Coder-480B-A35B-Instruct`.

> [accordion] Math / reasoning
>
> Models with `-Math` or reasoning-specific labels are trained on mathematical problem-solving and chain-of-thought data. They're stronger at step-by-step logic than general models of the same size but narrower in scope.

> [accordion] "next" or preview
>
> Some publishers tag experimental releases as `next`, `preview`, or `experimental`. These may have incomplete training or untested chat templates — useful for evaluation, not production.

## Training stage labels

The training stage tells you whether the model can hold a conversation or just complete text. This is the single most important segment for deciding whether a model is usable out of the box.

| Label | What it means | When to use |
| - | - | - |
| `Base` | Raw pretrained model — predicts the next token with no instruction tuning | Fine-tuning, embedding, research |
| `Instruct` / `Chat` | Fine-tuned to follow instructions and hold conversations | Almost all local use cases |
| `RLHF` / `DPO` | Aligned using human feedback or preference data — a step beyond basic instruction tuning | When output tone and safety matter |
| `Thinking` / `Reasoning` | Trained to emit chain-of-thought reasoning before answering | Math, logic, multi-step problems |

`Instruct` and `Chat` are interchangeable for most purposes — both mean the model was fine-tuned to respond to user prompts. `Chat` is the older label (Llama 2 used it); `Instruct` is more common in newer families. Some publishers like Mistral use neither and instead rely on version tags (`v0.3`) alongside the base name.

> [!NOTE]
>
> For local use, you almost always want `Instruct` or `Chat`. A `Base` model can't follow instructions — it only completes text — and will give confusing outputs if you prompt it like a chatbot.

> [!WARNING]
>
> If you see a model with no training stage label and no `Base` tag, check the model card before assuming it's instruction-tuned. Some publishers release fine-tuned models without labeling them, but others release base models without labeling them either.

## Quantization labels (the GGUF world)

GGUF quantization labels follow a naming scheme documented in the [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md). The `Q` number is roughly how many bits per weight; higher means larger file and higher quality. K-quants add mixed precision — different tensors get different bit-widths depending on sensitivity — and the trailing `S`/`M`/`L` marks small/medium/large variants within that level.

Here's the full range you'll encounter, with practical guidance:

| Quant | Bits (approx) | Quality | Notes |
| - | - | - | - |
| `Q2_K` | 2–3 | Low | Smallest. Noticeable quality loss — use only as a last resort. |
| `Q3_K_S` | 3 | Low–medium | Small variant. Tight VRAM, degraded reasoning. |
| `Q3_K_M` | 3–4 | Medium | Best 3-bit option. Lets you fit a larger model in limited VRAM. |
| `Q3_K_L` | 3–4 | Medium | Larger 3-bit variant, slightly better quality. |
| `Q4_0` | 4 | Medium | Simple 4-bit, no mixed precision. Superseded by K-quants. |
| `Q4_K_S` | 4–5 | Good | Smaller K-quant at 4-bit. |
| `Q4_K_M` | 4–5 | Good | **Best size/quality balance.** The default most repos highlight. |
| `Q4_K_L` | 4–5 | Very good | Larger 4-bit variant. |
| `Q5_0` | 5 | Very good | Simple 5-bit. |
| `Q5_K_S` | 5–6 | Very good | Smaller K-quant at 5-bit. |
| `Q5_K_M` | 5–6 | Near-lossless | Strong choice when VRAM allows. |
| `Q6_K` | 6 | Near-lossless | Hard to distinguish from full precision for most tasks. |
| `Q8_0` | 8 | Near-lossless | Safest quality margin. Roughly half the size of FP16. |
| `F16` | 16 | Lossless | Full 16-bit precision. Largest file, reference quality. |

### Practical sweet spots

* **`Q4_K_M`** — Start here. Good quality, fits consumer hardware.
* **`Q5_K_M`** — Step up when your VRAM has headroom. Near-lossless for most tasks.
* **`Q8_0`** — Minimal loss. Use when quality is critical and memory is available.
* **`Q3_K_M`** — Drop to this only to fit a larger model that won't load at `Q4_K_M`. Expect a quality hit.

### Size comparison: a 7B model at each quant

| Quant | Approximate file size |
| - | - |
| `F16` | 14 GB |
| `Q8_0` | 7.5 GB |
| `Q6_K` | 5.9 GB |
| `Q5_K_M` | 5.1 GB |
| `Q4_K_M` | 4.4 GB |
| `Q3_K_M` | 3.7 GB |
| `Q2_K` | 3.0 GB |

> [!TIP]
>
> The file size of a GGUF quant *is* your ground-truth VRAM number for the weights. Add KV cache on top (see [KV cache and context](https://docs.tokios.com/concepts/kv-cache-and-context.md)) for the total memory requirement.

## I-quants (imatrix) vs K-quants

You may also see quant names starting with `IQ` — like `IQ4_XS` or `IQ3_XXS`. These are **imatrix quants**, built using an importance matrix: calibration data that tells the quantizer which weights matter most, so it can preserve them more carefully. I-quants can produce better quality than K-quants at the same bit-width, particularly at very low bit-widths, at the cost of a slower quantization process. If a repo offers both `Q4_K_M` and `IQ4_M`, the I-quant may be slightly better — but the difference is usually small enough that either works.

For a deeper explanation of how quantization works and why K-quants outperform simple quants, see [Quantization](https://docs.tokios.com/concepts/quantization.md).

## Other common suffixes

Beyond quantization labels, model names include format and precision tags that tell you which backend can load the file.

> [accordion] GGUF
>
> The format used by `llama.cpp`-based backends — Ollama, LM Studio, and any `llama.cpp` server. GGUF files bundle the weights, tokenizer, and chat template into 1 file. Most consumer hardware runs GGUF.

> [accordion] GPTQ
>
> A GPU quantization format (typically 4-bit or 8-bit) that runs on NVIDIA GPUs via `AutoGPTQ` or `vLLM`. Faster than GGUF on GPU but NVIDIA-only and not compatible with `llama.cpp` backends.

> [accordion] AWQ
>
> Activation-aware weight quantization — another GPU format similar to GPTQ but calibrated using activation statistics. Supported by `vLLM` and some other inference engines. NVIDIA GPUs.

> [accordion] EXL2
>
> The ExLlamaV2 format — supports mixed-ratio quantization where different layers use different bit-widths. Optimized for NVIDIA GPUs and the ExLlamaV2 inference engine.

> [accordion] FP16 / BF16
>
> Full 16-bit precision. `FP16` (float16) and `BF16` (bfloat16) are both 2 bytes per parameter. BF16 has a wider exponent range and is more numerically stable during training; for inference, both behave similarly. Requires `vLLM` or similar engines.

> [accordion] FP8
>
> 8-bit floating point — half the size of FP16 with minimal quality loss. Used by inference engines like `vLLM` and `SGLang` on GPUs that support FP8 (NVIDIA Ada Lovelace and newer, AMD MI300).

> [accordion] NVFP4
>
> NVIDIA's 4-bit floating-point format, available on Blackwell GPUs including DGX Spark. Offers aggressive size reduction with hardware-native support — see [DGX Spark model compatibility](https://docs.tokios.com/guides/dgx-spark-model-compatibility.md).

## Quantization from the publisher vs third-party

Multiple parties often publish quantized versions of the same model:

* **Publisher quants** — Qwen, Mistral, Google, and others increasingly ship their own GGUF files alongside full-precision weights. These are usually the safest choice because the publisher controls the original weights and chat template.
* **Unsloth** — Ships GGUF quants with optimized chat templates and dynamic quantization. Also offers 1.58-bit (`unsloth/...-1.58bit`) variants that are extremely small. Generally reliable and widely used.
* **TheBloke** — Prolific early quantizer whose repos helped define the GGUF distribution ecosystem. Well-organized but may lag behind the newest models or miss chat template updates for recent releases.
* **bartowski** — Active community quantizer with consistent naming and good coverage of new releases. Often among the first to publish GGUF variants when a new model drops.

### How to verify a third-party quant

1. Check that the repo links back to the original model.
2. Compare the file size to what you'd expect at that quant level (the table above is a reference).
3. Look for the chat template metadata — a missing or wrong template degrades output silently.
4. If perplexity benchmarks are published for the quant, compare them to the publisher's own numbers.

## Quick decision guide

Match your hardware to a model size and quantization level:

> [card] 8 GB VRAM
>
> `7B Q4_K_M` or `14B Q3_K_M`. Stick with smaller models at comfortable quants — a 7B at Q4\_K\_M outperforms a 14B at Q2\_K.

> [card] 16 GB VRAM
>
> `14B Q5_K_M` or `32B Q4_K_M`. The 14B at Q5 is near-lossless; the 32B at Q4 trades some quality for significantly more capability.

> [card] 24 GB VRAM
>
> `32B Q5_K_M` or `70B Q4_K_M`. The 32B at Q5 is the sweet spot for single-GPU quality. A 70B at Q4 pushes capability higher but fills most of the VRAM.

> [card] 48 GB+
>
> `70B Q5_K_M` or MoE models like `235B-A22B Q4_K_M` with offloading. You have room for high-quality quants of the largest models.

> [card] 128 GB (DGX Spark)
>
> `120B+` MoE models at `FP8` or `NVFP4`. Full precision or near-full precision of frontier-scale models. See [DGX Spark model compatibility](https://docs.tokios.com/guides/dgx-spark-model-compatibility.md).

> [card] CPU only
>
> `3B Q4_K_M` or `7B Q3_K_M`. CPU inference is slow — keep the model small. Expect 1–5 tokens per second.

> [!NOTE]
>
> These are starting points. Your actual VRAM usage also depends on context length (KV cache grows with context), batch size, and whether the backend offloads layers to system RAM. Use the [model fit guide](https://docs.tokios.com/guides/model-fit.md) for precise sizing.

## Naming red flags

Not every model name follows conventions, and some patterns warrant caution:

* **No size label** — A model name without a parameter count could be anything. Check the model card before downloading.
* **"Uncensored" variants** — Usually just safety training removed, not a capability improvement. Quality of the base model may be unchanged or worse if the removal broke alignment.
* **Merged models** — Combine weights from 2 or more models (for example, merging a coding model with a chat model). Quality varies wildly and is hard to predict from the name alone. Look for the merge method (SLERP, DARE, TIES) and whether evaluation results are published.
* **Extremely low quants** — `Q2_K` is almost always too lossy for anything beyond testing whether a model loads. Expect degraded reasoning, broken formatting, and hallucinations. If a model only ships Q2 quants, look for a different model at the same parameter count.
* **Version numbers without changelogs** — `v0.1` through `v0.9` with no clear description of what changed between versions. Use the latest unless you have a specific reason not to.
* **"AWSD" or similar hype tags** — Some repos add attention-grabbing suffixes that don't correspond to any standard technique. If you can't find the term elsewhere, treat it skeptically.

## Putting it all together

Take `unsloth/Qwen3-30B-A3B-Instruct-GGUF`. Reading left to right:

1. `unsloth` — quantized by Unsloth (third-party quantizer).
2. `Qwen3` — Qwen3 architecture, current generation from Alibaba.
3. `30B-A3B` — MoE model: 30B total parameters, 3B active per token.
4. `Instruct` — instruction-tuned, ready for chat.
5. `GGUF` — `llama.cpp` format, runs on Ollama, LM Studio, or any `llama.cpp` backend.

From the name alone you know: this is a small MoE model that runs fast, fits in 8 GB of VRAM at `Q4_K_M`, and can follow instructions. You don't need to open the model card to decide whether to try it.

Now compare `meta-llama/Llama-3.3-70B-Instruct`. That's Meta's Llama 3.3, 70B dense parameters, instruction-tuned — no GGUF tag means you're looking at full-precision weights that need `vLLM` or a similar engine. The name tells you it's a large, high-quality model that requires serious hardware.

## Next: read the model card

Now that you can decode the name, the [model card](https://docs.tokios.com/concepts/read-a-model-card.md) tells you everything the name leaves out — context length, license, chat template, and intended use.

> [card] [Read a model card](https://docs.tokios.com/concepts/read-a-model-card.md)
>
> Find the parameters, architecture, context length, and license that matter before you download.

> [card] [Quantization](https://docs.tokios.com/concepts/quantization.md)
>
> How quantization works and why K-quants outperform simple quants.

> [card] [Model fit guide](https://docs.tokios.com/guides/model-fit.md)
>
> Check whether a model-and-quant combination fits your hardware.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.