> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Serving and scaling a model

> The words in an advanced run command — inference framework, precision, tensor/pipeline/expert parallelism, and speculative decoding (MTP) — explained plainly.

When you run `ollama run qwen3.6`, a lot is hidden from you. The moment you open a model page's advanced **Run it** tab and see a line like `vllm serve … --tensor-parallel-size 8 --enable-expert-parallel`, you meet a new vocabulary. This page explains those words — what each one does and when you'd actually need it.

> [!NOTE]
>
> If you run one model on one GPU (or a Mac) through Ollama or LM Studio, you can skip most of this. Parallelism and kernels matter when a model is too big for one GPU, or when you're serving many users at once. Read this when a model page's advanced tab sends you here.

## The inference framework

An **inference framework** is the engine that loads the model, runs the math, manages batching and the [KV cache](https://docs.tokios.com/concepts/kv-cache-and-context.md), and exposes an API. They trade ease of use for control:

| Framework | Best for | Notes |
| - | - | - |
| **Ollama** | Easiest local start | One command, manages `GGUF` downloads and an OpenAI-compatible `/v1` |
| **LM Studio** | Desktop GUI | Point-and-click model catalog, local server |
| **llama.cpp** | Portable, low-level | The engine under Ollama/LM Studio; runs `GGUF` on almost anything |
| **vLLM** | High-throughput GPU serving | Rich parallelism and quantization flags |
| **SGLang** | High-throughput GPU serving | Similar role to vLLM, strong structured-output support |
| **TensorRT-LLM** | Max NVIDIA performance | Compiles model-specific engines for NVIDIA GPUs |
| **MLX** | Apple Silicon | Mac-native, unified-memory design |

The easy ones (Ollama, LM Studio) are the beginner path on every model page. The rest expose the controls below. Tokios connects to any of them — it forwards to whatever OpenAI-compatible `/v1` your framework exposes.

## Precision

**Precision** is how many bits the runtime uses to store and compute each number in the model — `FP32` (32-bit), `FP16`/`bf16` (16-bit), `FP8` (8-bit), or 4-bit formats. Fewer bits means less memory and faster data movement, at some risk to quality. This is the *same* idea as [quantization](https://docs.tokios.com/concepts/quantization.md) — precision is the knob, quantization is turning it down. Which low-precision *format* your hardware can actually run is a separate question, answered in [Quantization formats and the hardware they need](https://docs.tokios.com/models/quantization-and-hardware.md).

## Kernels

A **kernel** is the small, highly optimized GPU (or CPU) routine that performs one hot operation — a matrix multiply, an attention step, dequantizing a 4-bit weight, sampling the next token. You rarely name kernels directly, but they explain two things you *do* see:

* **Why a quant format needs specific hardware.** A format is only fast if the runtime ships a kernel that understands it on your chip. `NVFP4` needs Blackwell Tensor Core kernels; `GGUF` runs broadly because llama.cpp ships kernels for CPU, Metal, CUDA, ROCm, and Vulkan. See [Quantization formats and the hardware they need](https://docs.tokios.com/models/quantization-and-hardware.md).
* **Why "FlashAttention" comes up.** FlashAttention is a faster attention kernel that saves memory on long context. When a runtime says it's enabled, that's a kernel choice working for you.

## Running across multiple GPUs — parallelism

When a model is too big for one GPU, or one GPU can't serve enough traffic, the runtime spreads the work across several. There are four common strategies, and they answer different problems:

| Strategy | What it splits | Use it when | Typical flag |
| - | - | - | - |
| **Tensor parallelism (TP)** | Each layer's weights, across GPUs | The model won't fit on one GPU | `--tensor-parallel-size N` (vLLM), `--tp N` (SGLang) |
| **Pipeline parallelism (PP)** | Whole layers, into sequential stages | Model spans many GPUs/nodes | `--pipeline-parallel-size N` (vLLM) |
| **Data parallelism (DP)** | Nothing — full copies per GPU | The model fits, but you need more throughput | `--data-parallel-size N` (vLLM) |
| **Expert parallelism (EP)** | A mixture-of-experts model's experts, across GPUs | Serving a large MoE model | `--enable-expert-parallel` (vLLM) |

* **Tensor parallelism** cuts each layer's matrices into slices, one per GPU, and the GPUs combine results every forward pass. It lowers per-GPU memory, but the GPUs must talk constantly — fast interconnects (NVLink) matter.
* **Pipeline parallelism** puts different layers on different GPUs, like a factory line. It scales across many devices but can leave GPUs idle waiting for the stage before them.
* **Data parallelism** just runs independent copies. It adds no help for a model that already doesn't fit — it's purely for handling more simultaneous requests.
* **Expert parallelism** applies only to [MoE](https://docs.tokios.com/concepts/read-a-model-card.md) models: instead of copying every expert to every GPU, each GPU holds some experts and tokens are routed to the GPU that owns the chosen expert. Great for big MoE models, at the cost of routing overhead.

Sources: [vLLM parallelism and scaling](https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html), [vLLM data-parallel deployment](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment/), [vLLM expert-parallel deployment](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/), [TensorRT-LLM parallelism strategy](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/parallel-strategy.md).

## Speculative decoding and MTP

[Decode](https://docs.tokios.com/concepts/how-inference-works.md) is slow because the model reads its weights from memory for every single token. **Speculative decoding** speeds it up without changing the output: a fast "draft" proposes several tokens at once, then the full model *verifies* them in one pass and keeps the ones it agrees with. Because verification is cheaper than generating from scratch, you get more accepted tokens per memory read.

**Multi-token prediction (MTP)** is one way to produce those drafts. The model ships extra heads trained to predict the next few tokens, so it can draft and verify itself. The current `DeepSeek-V4-Flash-0731` release bundles a **DSpark** module in the same checkpoint. Its official model card documents `method: dspark` for vLLM and `DSPARK` for SGLang, so you do not configure a separate draft-model path.

Sources: [DeepSeek-V4-Flash-0731 model card](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731), [Qwen3.6-27B model card](https://huggingface.co/Qwen/Qwen3.6-27B), [vLLM speculative decoding](https://docs.vllm.ai/en/latest/features/spec_decode.html).

> [!TIP]
>
> On a single consumer GPU, use an `-mtp` tag only when your exact model and runtime document it. For `DeepSeek-V4-Flash-0731`, use the bundled DSpark path on a supported multi-device vLLM or SGLang deployment.

## Where this fits

> [card] [How inference works](https://docs.tokios.com/concepts/how-inference-works.md)
>
> The prefill/decode basics these controls optimize.

> [card] [Quantization & hardware](https://docs.tokios.com/models/quantization-and-hardware.md)
>
> Which precision format runs on your chip.

> [card] [What size model fits your GPU?](https://docs.tokios.com/guides/model-fit.md)
>
> Size a model before you reach for multiple GPUs.

> [card] [Give it a Tokios endpoint](https://docs.tokios.com/quickstart.md)
>
> However you serve it, Tokios turns your local `/v1` into a stable, authenticated endpoint you can reach from anywhere.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.