> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Quantization-aware training (QAT), explained

> What quantization-aware training (QAT) is, how it differs from post-training quantization, and how to run Google's Gemma QAT checkpoints at ~4-bit VRAM.

**Quantization-aware training (QAT)** produces low-bit weights that hold quality far better than a plain 4-bit conversion. Instead of shrinking a finished model in one shot, QAT simulates low precision *during* training, so the weights learn to tolerate rounding before they're ever cast down. The payoff for running models locally: near-full-quality output at roughly a quarter of the VRAM.

This page explains what QAT is, how it relates to the `GGUF` files you already download, and which QAT checkpoints you can run today. For the basics of quantization and how to read a name like `Q4_K_M`, start with [Quantization explained](https://docs.tokios.com/concepts/quantization.md).

## QAT vs post-training quantization (PTQ)

Almost every quantized model you download is made with **post-training quantization (PTQ)**: take a finished BF16 model, round its weights to fewer bits, ship it. It's fast and needs no retraining — but the model was never given a chance to adapt to the rounding error, so accuracy loss grows as you push the bit-width down.

**QAT** inserts "fake quantization" into the training or fine-tuning loop. The forward pass rounds and clamps weights to the target bit-width but keeps them in floating point, and gradients (via a straight-through estimator) teach the model to stay accurate under that rounding *before* it's actually cast to low precision. Because the weights were optimized to survive quantization, a QAT checkpoint beats a PTQ one at the same bit-width.

The gap is measurable:

* **PyTorch's** own numbers on Llama 3-8B at 4-bit weights: QAT recovered **96%** of the accuracy degradation on HellaSwag and **68%** of the perplexity degradation on WikiText versus PTQ. ([PyTorch blog](https://pytorch.org/blog/quantization-aware-training/))
* **Google's** Gemma 3 QAT int4 checkpoints cut the perplexity gap versus the non-quantized model by **54%** compared to a naive `Q4_0` conversion of the same model. ([Google Developers blog](https://developers.googleblog.com/en/gemma-3-quantized-aware-trained-state-of-the-art-ai-to-consumer-gpus/))

> [!NOTE]
>
> The recovery rate isn't a fixed constant — it varies by model, bit-width, and benchmark. Treat the figures above as illustrative of the direction and rough magnitude, not a guarantee for every model.

## QAT is a method, not a file format

A common mix-up is "QAT vs GGUF." They aren't alternatives — they answer different questions:

* **QAT and PTQ** are *how the low-bit weights were produced* (the training/quantization method).
* **`GGUF`** is *the container the weights ship in* for `llama.cpp`-based runtimes (Ollama, LM Studio, Jan).

A QAT checkpoint can still be exported to `GGUF` — and usually is. Google's release makes this explicit in its naming: [`google/gemma-4-12B-it-qat-q4_0-gguf`](https://huggingface.co/collections/google/gemma-4-qat-q4-0) is a `GGUF` file whose `Q4_0` weights came out of the QAT pipeline, as opposed to a plain `gemma-4-12b-it` that Ollama quantized to `Q4_0` with ordinary PTQ. Same format, same bit-width — different, higher-fidelity weights inside. So the real comparison is **QAT vs PTQ**, and `GGUF` is just how either one reaches your runtime.

## Gemma QAT: the checkpoints to actually run

Google is the standout here: it fine-tunes each Gemma model with QAT and publishes the results as ready-to-run int4 checkpoints. On Ollama they carry an `-it-qat` tag. For **Gemma 4** (the current generation — see [Gemma](https://docs.tokios.com/models/gemma.md)):

| Ollama tag | Download size |
| - | - |
| `gemma4:e2b-it-qat` | \~4.3 GB |
| `gemma4:e4b-it-qat` | \~6.1 GB |
| `gemma4:12b-it-qat` | \~7.2 GB |
| `gemma4:26b-a4b-it-qat` | \~16 GB |
| `gemma4:31b-it-qat` | \~19 GB |

Sizes are from [ollama.com/library/gemma4](https://ollama.com/library/gemma4/tags) — confirm the current tag list there before you standardize on one. Google reports that Gemma 4 QAT yields "higher overall quality compared to standard PTQ baselines." The well-quantified precedent is Gemma 3, where int4 QAT dropped VRAM roughly **3×** — the 27B model from \~54 GB at BF16 to \~14 GB — while staying close to full-precision quality, which is what brought 27B-class models onto a single consumer GPU.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Pull a Gemma 4 QAT checkpoint and run it locally
ollama run gemma4:12b-it-qat
```

> [!TIP]
>
> If a model offers both a QAT tag and a same-bit-width PTQ tag, prefer QAT — you get better quality at identical size and no extra cost at inference time. QAT pays its price once, during training.

## Fine-tuning your own QAT model

QAT isn't only for vendor checkpoints. [Unsloth](https://unsloth.ai/docs/blog/quantization-aware-training-qat) (with PyTorch's TorchAO team) added QAT to its fine-tuning library: pass a `qat_scheme` (`int4`, `int8-int4`, `fp8-int4`, or `fp8-fp8`) so fake-quantization runs during your LoRA/fine-tune loop, then export to a low-bit format. Unsloth reports recovering **up to \~70%** of the accuracy a naive quantization loses, with no added inference-time overhead and the same on-disk footprint as ordinary quantization.

This is the practical path for the popular open models that *don't* ship official QAT checkpoints — Qwen, for example, has no first-party QAT release, but its models are a common base for community QAT recipes built on Unsloth and TorchAO. If you fine-tune and serve your own QAT model, [serve your fine-tune](https://docs.tokios.com/guides/serve-your-fine-tune.md) covers exporting it to a runtime, and Tokios connects it from there.

## Run your QAT model from anywhere

QAT gets a high-quality model to fit on the hardware you actually own. Once it's running locally — on Ollama, LM Studio, vLLM, or llama.cpp — the next step is reaching it from your agents, apps, and teammates without exposing `localhost`.

> [card] [Give it a stable endpoint](https://docs.tokios.com/quickstart.md)
>
> Tokios doesn't pick, host, or fine-tune the model — once your QAT checkpoint runs locally, Tokios turns it into a stable, authenticated endpoint at `https://api.tokios.com` you can call from anywhere.

## Go deeper

> [card] [Quantization explained](https://docs.tokios.com/concepts/quantization.md)
>
> How quantization works and how to read `GGUF` names like `Q4_K_M`.

> [card] [Run Gemma locally](https://docs.tokios.com/models/gemma.md)
>
> Gemma 4's sizes, hardware fit, and its `-it-qat` quants.

> [card] [Quantization and hardware](https://docs.tokios.com/models/quantization-and-hardware.md)
>
> Which weight formats — GGUF, NVFP4, MXFP4, AWQ, MLX — run on your hardware.

> [card] [What size model fits your GPU?](https://docs.tokios.com/guides/model-fit.md)
>
> Check whether a model-and-quant combination fits before you download it.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.