> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Quantization formats and the hardware they need

> The format×hardware matrix for local model quantization — which formats (GGUF, AWQ, GPTQ, NVFP4, MXFP4, MLX) run on which GPUs, and which to pick.

> [!NOTE]
>
> Last verified: 2026-10-14

Quantization trades precision for size and speed, and it has two independent axes: the quant **name** (`Q4_K_M`, `INT4`, `FP8` — how many bits and how they're packed, covered in [Quantization explained](https://docs.tokios.com/concepts/quantization.md)) and the quant **format** (`GGUF`, `AWQ`, `NVFP4` — the file format and kernel it runs through, which decides what hardware can actually load it). This page owns the second axis: pick a row below, and you know what hardware it needs.

## The matrix

| Format | CPU | NVIDIA CUDA | NVIDIA Blackwell only | AMD ROCm | Apple Metal | Vulkan | Typical runtime |
| - | - | - | - | - | - | - | - |
| `GGUF` | Yes | Yes | — | Yes | Yes | Yes | llama.cpp, Ollama, LM Studio |
| `bf16` / `FP16` | Runtime dependent, usually impractical | Yes | — | Yes when the runtime supports it | Runtime dependent | Runtime dependent | Framework-dependent unquantized baseline |
| `AWQ` | No† | Yes | — | No‡ | No | No | vLLM |
| `GPTQ` | No† | Yes | — | No‡ | No | No | vLLM |
| `MXFP4` | Partial | Yes | Not required | Partial | Partial | — | llama.cpp, vLLM — gpt-oss's native format |
| `NVFP4` | No | No§ | Yes | No | No | No | TensorRT / TensorRT-LLM, vLLM (Blackwell) |
| `MLX` | No | No\*\* | — | No | Yes | — | MLX (Apple Silicon) |

† vLLM's own hardware-support table lists AWQ with x86/Arm CPU support via a separate code path, but not GPTQ; treat both as GPU-first formats in practice.
‡ vLLM's compatibility table marks AWQ and GPTQ as unsupported on AMD GPU in vLLM specifically — this is a vLLM-runtime limitation, not a statement about every AWQ/GPTQ kernel everywhere.
§ Pre-Blackwell NVIDIA GPUs don't have the Tensor Core support NVFP4 needs; it isn't a driver-only limitation.
\*\* MLX's core PyPI package targets Apple Silicon + macOS; the project also ships an experimental Linux CUDA backend for the framework itself, but MLX-quantized model files you'll find on Hugging Face are built for the Apple Metal path.

Sources for every claim above are cited in the notes below.

## Per-format notes

### GGUF — the portable default

`GGUF` is llama.cpp's model format, and it's what Ollama and LM Studio use under the hood. It runs on CPU, Apple Metal, NVIDIA CUDA, AMD ROCm (via HIP), and Vulkan — llama.cpp's own build docs list dedicated build targets for all of these backends, which makes `GGUF` the most portable format across consumer hardware. If you're not sure what your hardware supports, `GGUF` is the safe default. See [llama.cpp](https://github.com/ggml-org/llama.cpp) and the [GGUF spec](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md).

### bf16 / FP16 — the baseline

`bf16` and `FP16` store each parameter in 2 bytes with no compression. This is the precision quantization compresses *from* — every `Q4`, `AWQ`, or `NVFP4` file is a smaller, lossier version of a `bf16`/`FP16` original. Treat support as framework-dependent: CUDA and ROCm serving stacks commonly support these precisions when you have enough memory, while CPU, Metal, and Vulkan paths depend on the runtime and are often impractical for large models. At roughly 4x the footprint of a 4-bit quantization, `bf16`/`FP16` is rarely what you run locally on consumer hardware. See [GPU-to-model fit](https://docs.tokios.com/guides/model-fit.md) for the memory math.

### AWQ and GPTQ — GPU-oriented, vLLM's formats

`AWQ` and `GPTQ` are 4-bit quantization formats built for GPU inference, and vLLM is the runtime that documents and maintains support for both. Per vLLM's hardware-compatibility table, both run on NVIDIA GPUs (Volta through Hopper, with GPTQ additionally supported on Volta); vLLM's table shows neither as supported on AMD GPU. Treat them as an NVIDIA/CUDA-first pair unless you've confirmed otherwise for your specific runtime. See [vLLM's quantization docs](https://docs.vllm.ai/en/latest/features/quantization/index.html).

### MXFP4 — gpt-oss's native format

`MXFP4` is the Open Compute Project's Microscaling 4-bit floating-point format, and it's the format OpenAI shipped gpt-oss in natively — both `gpt-oss-120b` and `gpt-oss-20b` are distributed pre-quantized in `MXFP4` on Hugging Face, which is what lets `120b` run in about 80GB and `20b` in about 16GB. llama.cpp added native `MXFP4` support for gpt-oss, and vLLM supports it as well. See [OpenAI — Introducing gpt-oss](https://openai.com/index/introducing-gpt-oss/) and the [llama.cpp gpt-oss support notes](https://github.com/ggml-org/llama.cpp/pull/15091).

### NVFP4 — Blackwell only

`NVFP4` is NVIDIA's own 4-bit floating-point format, purpose-built for the fifth-generation Tensor Cores in the **Blackwell** architecture (`RTX 50` series consumer cards and Blackwell-generation datacenter GPUs). NVIDIA's own technical blog describes `NVFP4` as a Blackwell Tensor Core format using a two-level scaling scheme, and positions it alongside `MXFP4` and `FP4` as the three 4-bit formats Blackwell's Tensor Cores support natively. If your GPU predates Blackwell, `NVFP4` isn't the format for you — look for a `GGUF`, `AWQ`, or `GPTQ` build of the same model instead. See [NVIDIA — Introducing NVFP4](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/).

> [!NOTE]
>
> Many `NVFP4` repos on Hugging Face live under the `nvidia/` org (for example `...-NVFP4` suffixed models). That naming is a strong hint you need Blackwell-generation hardware to get the native speedup — check the model card before you pull one on an older card.

> [!WARNING]
>
> **DGX Spark / RTX Spark GB10 performance gap:** Community benchmarks (October 2026) report that the GB10 chip in DGX Spark and early RTX Spark systems delivers approximately half the promised NVFP4 throughput (\~0.5 PFLOP instead of 1 PFLOP). NVIDIA's published speedup figures are measured on B200 datacenter GPUs, not the GB10 hardware listed as a supported deployment target. If you're buying DGX Spark or RTX Spark specifically for NVFP4, benchmark your expected workloads before relying on the published numbers. See the [NVIDIA developer forum thread](https://forums.developer.nvidia.com/t/nvfp4-on-dgx-spark-gb10-is-broken/367082) for current status.

### MLX — Apple Silicon

`MLX` is Apple's array framework, built around a unified-memory model where arrays live in shared memory across CPU and GPU. `MLX`-quantized models — the format LM Studio and other Mac-first tools offer alongside `GGUF` — target Apple Silicon on macOS 14 or later. The project's own install docs list Apple Silicon + macOS as the primary supported path (with an experimental Linux/CUDA backend for the framework itself, separate from the model files you'll find distributed in `MLX` format). If you're not on a Mac, `MLX` isn't an option — use `GGUF` instead. See [MLX on GitHub](https://github.com/ml-explore/mlx) and the [MLX install docs](https://ml-explore.github.io/mlx/build/html/install.html).

> [!TIP]
>
> **MLX distributed inference (October 2026):** Run trillion-parameter models across multiple Macs using RDMA over Thunderbolt. A 4× M5 Ultra cluster achieves 36.73 tokens/s on Muse Spark Open (816B parameters). MLX supports pipeline parallelization (split model depth) and tensor parallelization (split layer width) across nodes. This is the Apple Silicon equivalent of running large MoE models on consumer NVIDIA GPUs with FreeToken — see [FreeToken MoE inference](https://docs.tokios.com/guides/freetoken-moe-inference.md) for the NVIDIA path.

## Which should you pick?

Match the format to the hardware you already have, not the other way around:

* **NVIDIA, consumer card (RTX 40/30 series or older Blackwell-less cards)** — `GGUF` for the easiest path (Ollama, LM Studio), or `AWQ`/`GPTQ` if you're running vLLM and want the extra throughput of a GPU-native kernel.
* **NVIDIA Blackwell (RTX 50 series or Blackwell datacenter GPUs)** — everything above still works, but look for an `NVFP4` build if the model publishes one; it's the format built for your Tensor Cores.
* **Apple Silicon** — `GGUF` (via Metal) is the simplest and most common path; `MLX` is a Mac-native alternative that some tools (like LM Studio) offer side by side with `GGUF`.
* **AMD GPU** — `GGUF` through ROCm or Vulkan. AWQ/GPTQ support on AMD is inconsistent across runtimes, so don't plan around it.
* **CPU only** — `GGUF`. It's the only format in this matrix with real CPU inference support, and it's what llama.cpp, Ollama, and LM Studio are built on.

For the quant *name* to pick within whichever format you land on (`Q4_K_M` vs `Q8_0`, for example), see [Quantization explained](https://docs.tokios.com/concepts/quantization.md). For sizing a specific model against your VRAM, see [GPU-to-model fit](https://docs.tokios.com/guides/model-fit.md).

> [card] [Turn it into a stable endpoint](https://docs.tokios.com/quickstart.md)
>
> Once a quantized model runs locally — whatever format it's in — Tokios doesn't change how it runs. It gives you a stable, authenticated API endpoint to reach that same local deployment from anywhere.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.