> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# How inference works

> Follow a prompt from text to tokens to streamed reply, and see what that round trip does to your GPU's memory.

Every reply from a local model follows the same round trip: your text becomes tokens, the model reads the whole prompt at once, then it writes the reply one token at a time. Understanding those steps explains two things every local-model user runs into — why a long prompt takes a moment to "think" before the first word appears, and why generation speed depends on memory, not just raw compute.

This page is the 101 for the rest of the **Concepts** track. If you only read one page here, read this one.

## Step 1: tokenization

Before the model sees anything, your prompt is split into **tokens** — small chunks of text, roughly a few characters or a word-piece each. Tokenization is deterministic and near-instant; it's just a lookup against the model's vocabulary. The model never sees raw text, only a sequence of token ids.

## Step 2: prefill (compute-bound)

**Prefill** is the model reading your entire prompt — every token you sent, in one pass — before it writes a single word back. This step is **compute-bound**: the GPU is doing heavy matrix math across the whole prompt at once, so it scales with how much processing power you have.

Prefill has two effects worth knowing:

* It's what you're waiting on when a reply is slow to start. The delay between sending a prompt and seeing the first token is called **time to first token (TTFT)**, and it's almost entirely prefill time.
* It builds the **KV cache** — a running record of the prompt so the model doesn't have to reprocess it on every subsequent token. See [KV cache and context](https://docs.tokios.com/concepts/kv-cache-and-context.md) for what that cache costs in memory.

Longer prompts mean more prefill work, which means longer TTFT.

## Step 3: decode (memory-bandwidth-bound)

Once prefill finishes, the model starts **decode**: generating the reply one token at a time. Each new token is appended to the sequence, and the next token is predicted from everything so far.

Decode behaves very differently from prefill. On every single step, the model has to read its weights (or, for a mixture-of-experts model, just the active experts) plus the whole KV cache out of memory. Because so little new compute happens per token relative to the amount of data that has to move, decode is **memory-bandwidth-bound** — it's limited by how fast data can travel from memory to the compute units, not by how much raw compute you have.

This is what **tokens per second** measures, and it's why:

* A GPU with less compute but faster memory can out-decode a "faster" chip with slower memory.
* Longer context slows decode down, because there's more KV cache to read on every single step, in addition to costing more memory (see [KV cache and context](https://docs.tokios.com/concepts/kv-cache-and-context.md)).

## Step 4: tokens stream back

As the model produces each token, it's decoded back into text and streamed to the client immediately — this is why you see replies appear word by word instead of all at once. If you're using Tokios, this stream travels from your local backend through the connector and gateway back to whatever app or agent sent the request; see [How Tokios routes requests](https://docs.tokios.com/how-it-works.md) for that leg of the journey.

## What goes where in memory

Everything above happens against three things sitting in VRAM (or unified memory):

| What | Behavior |
| - | - |
| **Model weights** | Fixed size, set the moment the model loads. Doesn't change with prompt length. |
| **KV cache** | Grows with context length and batch size. Built during prefill, extended during decode. |
| **Activations & overhead** | Smaller working memory the framework needs during a forward pass. |

Weights are the floor — you can't run the model without them fully loaded (or offloaded in part; see [Memory and offloading](https://docs.tokios.com/concepts/memory-and-offloading.md)). The KV cache is the part that grows as your conversation or document gets longer, which is why "it worked with a short prompt but ran out of memory on a long one" is such a common failure mode.

> [!NOTE]
>
> This page covers the mechanics. For the size math — how many gigabytes a given model and context actually need — see [Memory and offloading](https://docs.tokios.com/concepts/memory-and-offloading.md) and the [GPU model-fit guide](https://docs.tokios.com/guides/model-fit.md).

## Go deeper

> [card] [KV cache and context](https://docs.tokios.com/concepts/kv-cache-and-context.md)
>
> Why longer context costs more memory and slows decode.

> [card] [Memory and offloading](https://docs.tokios.com/concepts/memory-and-offloading.md)
>
> Sizing formulas and what happens when a model doesn't fully fit.

> [card] [Quantization](https://docs.tokios.com/concepts/quantization.md)
>
> How shrinking weight precision changes both memory and speed.

> [card] [Read a model card](https://docs.tokios.com/concepts/read-a-model-card.md)
>
> Where to find a model's architecture and context length before you run it.

## Once you understand how it runs

Everything above happens on your machine, against a model running on `localhost`. Once you're comfortable with that, the next question is how to reach it from somewhere other than your own machine.

> [card] [Give your local model a stable endpoint](https://docs.tokios.com/quickstart.md)
>
> Tokios doesn't pick, host, or fine-tune your model — once it's running locally, Tokios turns it into a stable, authenticated endpoint you can call from anywhere.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.