> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Quantization explained

> What quantization is, how to read GGUF quant names like Q4_K_M, and which one to pick.

Quantization shrinks a model by storing its weights at lower numeric precision, trading a small amount of quality for a much smaller memory footprint and often faster inference. It's the single biggest lever you have over whether a model fits on your hardware at all.

## Why it works

A model's weights are just numbers. Trained models are usually released at 16-bit precision (FP16 or BF16) — 2 bytes per parameter. Quantization rounds those numbers down to fewer bits: 8-bit uses 1 byte per parameter, and 4-bit (`Q4`) uses roughly half a byte. Fewer bits per weight means less data to store and less data to move through memory on every token generated, which is why quantized models are both smaller *and* often faster to run — recall from [How inference works](https://docs.tokios.com/concepts/how-inference-works.md) that decode speed is bound by how fast weights move through memory, not by raw compute.

The tradeoff is precision loss. Rounding weights to fewer bits introduces small errors that compound across a model's layers, which can measurably affect output quality at aggressive quantization levels. For exact bytes-per-parameter figures and the sizing math this feeds into, see [Memory and offloading](https://docs.tokios.com/concepts/memory-and-offloading.md).

## Reading GGUF quant names

Files distributed for `llama.cpp`-based backends (including Ollama and LM Studio) use the `GGUF` format, and quantized variants follow a naming convention documented in the [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) and the [GGUF format spec](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md). A typical filename looks like `Qwen3-8B-Instruct-Q4_K_M.gguf` — the last segment before `.gguf` is the quantization.

Quant names run roughly from `Q2_K` (smallest, lowest quality) up to `Q8_0` (largest, near-lossless):

| Quant family | What it means |
| - | - |
| `Q2_K` … `Q8_0` | The number after `Q` is roughly how many bits per weight; higher means larger file and higher quality. |
| **K-quants** (`Q4_K_M`, `Q5_K_S`, etc.) | Mixed-precision quantization — different tensors in the model get different bit-widths depending on how sensitive they are to precision loss. The trailing `S`/`M`/`L` marks small/medium/large variants within that quant level. |
| **I-quants** (imatrix) | Built using an *importance matrix* — calibration data that tells the quantizer which weights matter most, so it can preserve them more carefully. Can produce better quality than a K-quant at the same bit-width, particularly at very low bit-widths, at the cost of a slower quantization process. |

So `Q4_K_M` decodes as: 4-bit, K-quant family, medium variant. `GGUF` `Q4_K_M` is mixed precision that averages out to roughly `0.5–0.56` bytes per parameter in practice — not a flat 4 bits across every tensor.

> [!NOTE]
>
> Higher number after `Q` is *not* strictly "always better" in every dimension — it's a size/quality dial. `Q8_0` is closer to the original 16-bit weights than `Q2_K`, but it's also roughly four times the file size.

## The size/quality tradeoff

Every step down in quantization saves memory and often gains a little speed, at some cost to output quality. The relationship isn't linear — quality tends to hold up well down to around 4-bit, then degrades more noticeably below that:

* **`Q8_0`**: Very close to full 16-bit quality. Use when you have VRAM to spare and want the safest quality margin.
* **`Q4_K_M`**: The common default. Good balance of size and quality for most general use.
* **`Q2_K`–`Q3_K`**: Smallest, most VRAM-friendly, but noticeably lower quality — reserve for cases where nothing else fits.

## Which quant should you pick?

As a rule of thumb: **start with `Q4_K_M`**. It's the default most model repos highlight for a reason — it fits comfortably on consumer hardware while staying close to full-precision output quality for most tasks.

From there:

* **Go higher** (`Q5_K_M`, `Q6_K`, `Q8_0`) if your VRAM has headroom after accounting for the KV cache at your target context length.
* **Go lower** (`Q3_K`, `Q2_K`) only if `Q4_K_M` genuinely doesn't fit and you have no other option — expect a noticeable quality drop, especially on reasoning-heavy or coding tasks.

Use the [GPU model-fit guide](https://docs.tokios.com/guides/model-fit.md) to check whether a given model-and-quant combination fits your hardware before downloading it.

> [!TIP]
>
> Quantization also affects other formats beyond GGUF — NVFP4, MXFP4, AWQ, GPTQ, and MLX all compress weights, but each is built for specific hardware and backends. See [Quantization and hardware](https://docs.tokios.com/models/quantization-and-hardware.md) for which format runs where; this page covers GGUF naming and the general size/quality tradeoff only.

## Quantization-aware training (QAT)

Everything above describes **post-training quantization (PTQ)** — quantizing a finished model in one shot. **Quantization-aware training (QAT)** instead simulates low precision *during* training, so the weights learn to tolerate rounding and hold quality better at the same bit-width. Google ships QAT checkpoints for Gemma under an `-it-qat` tag — for example `ollama run gemma4:12b-it-qat` — that stay close to full-precision quality at roughly a quarter of the BF16 VRAM. A QAT checkpoint still ships as `GGUF`: QAT is *how the weights were made*, not a different format.

> [card] [Quantization-aware training, explained](https://docs.tokios.com/concepts/qat.md)
>
> How QAT differs from PTQ and GGUF, the Gemma QAT checkpoints to run, and fine-tuning your own with Unsloth.

## Go deeper

> [card] [Read a model card](https://docs.tokios.com/concepts/read-a-model-card.md)
>
> Find which quant variants a model repo actually offers.

> [card] [Memory and offloading](https://docs.tokios.com/concepts/memory-and-offloading.md)
>
> The full sizing formula, including bytes-per-parameter by precision.

> [card] [Quantization and hardware](https://docs.tokios.com/models/quantization-and-hardware.md)
>
> Match weight formats (GGUF, NVFP4, MXFP4, AWQ, GPTQ, MLX) to hardware.

## Once you've picked a quant and it's running

Choosing a quant gets a model to fit and run well on your hardware. Once it's answering requests locally, the next step is reaching it from somewhere other than that one machine.

> [card] [Give it a stable endpoint](https://docs.tokios.com/quickstart.md)
>
> Tokios doesn't pick, host, or fine-tune the model — once it runs locally, Tokios turns it into a stable, authenticated endpoint you can call from anywhere.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.