> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Glossary

> Short definitions for the terms you'll hit running local models, each linking to a fuller explanation.

Quick definitions for local-model terms used across these docs. Each entry links to the page that covers it in depth.

## Tokens

The small chunks of text — roughly a few characters or a word-piece each — that a model actually reads and writes. Your prompt is split into tokens before the model sees it, and the reply is generated one token at a time. See [How inference works](https://docs.tokios.com/concepts/how-inference-works.md).

## Prefill vs. decode

**Prefill** is the model reading your entire prompt at once, before writing anything back — it's compute-bound and is what "time to first token" measures. **Decode** is the model generating the reply one token at a time — it's memory-bandwidth-bound and is what tokens/sec measures. See [How inference works](https://docs.tokios.com/concepts/how-inference-works.md).

## Dense vs. mixture-of-experts (MoE)

A **dense** model runs every parameter on every token. A **mixture-of-experts** model — named like `30B-A3B`, 30B total parameters with 3B active — only runs a subset of "expert" parameters per token, so it decodes faster than a dense model of the same total size, but still needs memory for the full parameter set. See [Read a model card](https://docs.tokios.com/concepts/read-a-model-card.md).

## Precision

**Precision** is how many bits the runtime uses to store or compute with each number in the model, such as `bf16`, `FP16`, `FP8`, or 4-bit formats. Lower precision usually means less memory and faster data movement, but it can reduce quality if the quantization is too aggressive. See [Quantization](https://docs.tokios.com/concepts/quantization.md) and [Quantization formats and the hardware they need](https://docs.tokios.com/models/quantization-and-hardware.md).

## Inference framework

An **inference framework** is the serving engine that loads the model, runs the math, manages batching and the KV cache, and exposes an API. Ollama and LM Studio optimize for easy local use; vLLM, SGLang, llama.cpp, TensorRT-LLM, and MLX expose lower-level controls for throughput, parallelism, and hardware-specific kernels. See [Serving and scaling](https://docs.tokios.com/concepts/model-serving-and-scaling.md).

## Kernel

A **kernel** is the low-level GPU or CPU routine that performs a hot operation such as matrix multiplication, attention, quantized dequantization, or sampling. Quantization formats are only useful when your runtime has kernels that understand them; for example, `NVFP4` needs Blackwell Tensor Core support, while `GGUF` works broadly through llama.cpp backends. See [Quantization formats and the hardware they need](https://docs.tokios.com/models/quantization-and-hardware.md).

## Tensor parallelism (TP)

**Tensor parallelism** splits one model layer's weights and computation across multiple GPUs. Use it when the model is too large for one GPU or when one GPU cannot serve enough traffic. It reduces per-GPU memory pressure, but the GPUs must synchronize during each forward pass, so fast interconnects matter. vLLM exposes this as `--tensor-parallel-size`; TensorRT-LLM documents the same strategy as TP. See [Serving and scaling](https://docs.tokios.com/concepts/model-serving-and-scaling.md).

## Data parallelism (DP)

**Data parallelism** runs separate copies of the model on separate GPUs or instances so each copy can process different requests. It increases request throughput when the model already fits per replica. vLLM's data-parallel deployment docs describe this as replicating model weights across instances or GPUs to process independent batches. See [Serving and scaling](https://docs.tokios.com/concepts/model-serving-and-scaling.md).

## Expert parallelism (EP)

**Expert parallelism** applies to mixture-of-experts models. Instead of giving every GPU a full copy of every expert, the runtime places different experts on different GPUs and routes tokens to the GPU that owns the selected expert. This can improve locality and throughput for large MoE models, but it adds routing and synchronization overhead. See [Serving and scaling](https://docs.tokios.com/concepts/model-serving-and-scaling.md).

## Multi-token prediction (MTP)

**Multi-token prediction** trains or adds auxiliary heads that predict more than one future token. At inference time, runtimes can use those predictions for speculative decoding: draft several tokens, verify them with the target model, and keep the accepted tokens. The current `DeepSeek-V4-Flash-0731` checkpoint bundles a related DSpark speculative-decoding module that vLLM enables with `method: dspark` and SGLang enables with `DSPARK`. See [Serving and scaling](https://docs.tokios.com/concepts/model-serving-and-scaling.md).

## Tool-calling

The ability for a model to return a structured request to call an external function or tool, rather than just plain text, so agentic clients can take actions. Reliable tool-calling is a major selection criterion for coding agents and other automated clients. See [Which local model for which task?](https://docs.tokios.com/guides/model-by-task.md).

## Temperature and top-p

Sampling settings that control how predictable versus varied a model's output is. Lower **temperature** makes output more deterministic and repetitive; higher temperature makes it more varied and occasionally less coherent. **Top-p** (nucleus sampling) restricts token choices to the smallest set whose combined probability reaches `p`, trimming away unlikely options before sampling. Both are request-level parameters your client sends alongside the prompt.

## Embeddings

Numeric vector representations of text, used for search, retrieval, and similarity comparison rather than for generating conversational replies. Embedding models are a different tool from chat models — see [Which local model for which task?](https://docs.tokios.com/guides/model-by-task.md) for how they fit into model selection.

## Context window

The maximum number of tokens a model can hold in a single conversation at once, spanning both the prompt and the reply. A larger context window lets you send more text at once, but costs more memory through the KV cache. See [KV cache and context](https://docs.tokios.com/concepts/kv-cache-and-context.md).

## Quantization

Storing a model's weights at lower numeric precision to shrink its size and often speed up inference, at some cost to output quality. GGUF quant names like `Q4_K_M` encode the bit-width and quant family used. See [Quantization](https://docs.tokios.com/concepts/quantization.md).

## KV cache

The running record of key/value representations a model builds while processing a conversation, so it doesn't have to reprocess earlier tokens from scratch on every new token. It grows with context length and batch size, and sits in memory alongside the model weights. See [KV cache and context](https://docs.tokios.com/concepts/kv-cache-and-context.md).

## GGUF

Short for **GGML Universal File** — the file format used by `llama.cpp`-based backends (including Ollama and LM Studio) to store model weights, typically distributed as several quantized variants of the same model. It succeeds the project's older GGML/GGJT weight formats, which is why you'll still see "GGML" in older repos and tooling. See [Read a model card](https://docs.tokios.com/concepts/read-a-model-card.md) and the [GGUF format spec](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md).

## Now that the vocabulary is covered

These terms come up constantly once you start running models locally — bookmark this page and come back to it. Once a model is running and answering requests on `localhost`, the natural next step is reaching it from somewhere other than that one machine.

> [card] [Give it a stable endpoint](https://docs.tokios.com/quickstart.md)
>
> Tokios doesn't pick, host, or fine-tune the model — once it runs locally, Tokios turns it into a stable, authenticated endpoint you can call from anywhere.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.