> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Decode, prefill, and power across precision levels

> Understand decode throughput vs prefill latency, how precision affects each, and what it means for power draw. Benchmark your own setup.

Every local LLM deployment has 2 speeds that matter — and they respond differently to the precision knob. This page explains why dropping from FP16 to INT4 might triple your generation speed but barely move your prompt-processing time, and how to measure the trade-off on your own hardware.

## Two speeds that matter

**Decode throughput** is how fast the model generates tokens after it starts answering, measured in tokens per second (tok/s). It determines how quickly a chat reply or code completion appears on screen.

**Prefill latency** is how long the model takes to process your input before it generates the 1st token, measured in milliseconds per token (ms/token). It determines the pause between hitting Enter and seeing output start to stream.

Different workloads care about different speeds:

| Workload | What matters most | Why |
| - | - | - |
| Interactive chat | Decode throughput | Short prompts, long replies — generation dominates |
| Document analysis | Prefill latency | Long prompts (10K+ tokens), short replies — input processing dominates |
| Coding agents | Both | Moderate prompts, long multi-turn output — you feel both |
| Batch processing | Total throughput | Decode and prefill both contribute to wall-clock time |

> [!NOTE]
>
> Time-to-first-token (TTFT) measures prefill latency — the pause before output starts streaming. Its counterpart is **TPOT** (Time Per Output Token), which measures decode speed: how long the model takes to generate each token after the first one appears. TTFT and TPOT are the 2 standard metrics for inference performance. For a single user on local hardware, TTFT is roughly equal to prefill time. Under concurrent load, queuing adds to TTFT independently.

## What precision means for speed

Precision controls how many bits represent each weight in the model. Fewer bits means smaller weights, which changes 2 things: how much memory the model occupies and how much data the GPU reads per token.

| Precision | Memory multiplier | Typical quality impact | Supported hardware |
| - | :-: | - | - |
| FP16 / BF16 | 1.0x (baseline) | None — full fidelity | All GPUs |
| FP8 (E4M3) | 0.5x | \<2% on modern models | Ada/Hopper/Blackwell (SM89+) |
| INT8 (W8A8) | 0.5x | 1-3% loss | Most GPUs with INT8 tensor cores |
| INT4 (W4A16, GPTQ, AWQ) | 0.25x | 2-5% loss | All GPUs (dequantized at runtime) |
| NVFP4 | 0.25x | 1-3% loss | Blackwell / DGX Spark (SM121) |

> [!TIP]
>
> The quality column is approximate. Newer models trained with quantization-aware training (QAT) lose less quality at lower precisions. See [QAT concepts](https://docs.tokios.com/concepts/qat.md) for details.

## How precision affects decode vs prefill differently

This is the key insight: **decode and prefill hit different hardware bottlenecks**, so the same precision change affects them unequally.

### Decode is memory-bandwidth-bound

During decode, the model generates 1 token at a time. For each token, it reads the entire weight matrix from memory and performs a tiny amount of compute per weight. The GPU spends most of its time waiting for data to arrive from memory, not doing math.

Smaller weights = less data to read = faster decode. The speedup is roughly proportional to the memory reduction:

* FP16 → FP8: \~1.8-2.0x decode speedup
* FP16 → INT4: \~2.5-3.5x decode speedup
* FP16 → NVFP4: \~2.5-3.0x decode speedup

### Prefill is compute-bound

During prefill, the model processes all input tokens in parallel. It reads the weights once but performs a large matrix multiplication across all prompt tokens. The GPU spends most of its time doing math, not waiting for memory.

Quantization helps prefill less because the math is the bottleneck, not the weight reads:

* FP16 → FP8: \~1.3-1.5x prefill speedup
* FP16 → INT4: \~1.2-1.5x prefill speedup
* FP16 → NVFP4: \~1.3-1.5x prefill speedup

### Batching shifts the bottleneck

At batch size 1, decode is memory-bandwidth-bound — the model reads the weights once per token and does minimal compute. But as you increase batch size (serving multiple concurrent requests), the picture changes: the model still reads the weights once, but now performs the tiny per-token compute across many tokens simultaneously. At some point, the compute adds up enough that it becomes the bottleneck instead of memory bandwidth.

This phase transition matters for multi-user deployments. A server handling 10 concurrent users might see decode become compute-bound, where FP8 or INT4 quantization helps prefill more than the single-user speedup tables suggest. For single-user local setups, batching doesn't apply — you stay memory-bound throughout decode.

### Practical implication

As precision decreases, decode improves faster than prefill. At INT4, a model that felt sluggish at FP16 generates text rapidly but still pauses noticeably on long prompts. This asymmetry drives the workload-specific recommendations below.

## Power draw across precision levels

Lower precision reduces power consumption for 2 reasons: fewer memory reads (memory access dominates energy cost) and simpler arithmetic (INT4 multiply costs less than FP16 multiply).

### DGX Spark power envelope

DGX Spark's GB10 chip has a \~40W TDP shared between Grace CPU and Blackwell GPU. Here are observed power ranges during inference:

| Precision | Idle power | Decode load | Prefill load |
| - | :-: | :-: | :-: |
| FP16 | 8-12W | 30-38W | 35-40W |
| FP8 | 8-12W | 25-33W | 30-38W |
| NVFP4 | 8-12W | 22-30W | 28-35W |

### Consumer GPU power expectations

| GPU | FP16 decode load | INT4/FP8 decode load | Savings |
| - | :-: | :-: | :-: |
| RTX 4090 (450W TDP) | 300-400W | 200-300W | \~25% less at INT4 |
| Strix Halo (120W TDP) | 80-110W | 60-85W | \~20% less at INT4 |

### Cost of 24/7 serving

For a DGX Spark running continuously at decode load:

| Precision | Avg power | Monthly kWh (approx) |
| - | :-: | :-: |
| FP16 | 34W | \~24 kWh |
| FP8 | 29W | \~21 kWh |
| NVFP4 | 26W | \~19 kWh |

On a consumer GPU like the RTX 4090 at 10 hours/day:

| Precision | Avg power | Monthly kWh (approx) |
| - | :-: | :-: |
| FP16 | 350W | \~105 kWh |
| FP8 | 250W | \~75 kWh |
| INT4 | 200W | \~60 kWh |

## Benchmark methodology

Run these tests on your own setup to see how precision affects your specific model and hardware.

### Measure decode throughput

Generate a fixed number of output tokens from a short prompt and measure tokens per second.

```bash vLLM benchmark theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Start the server first
vllm serve <model> --quantization <precision> --max-model-len 8192

# In another terminal
python -m vllm.benchmarks.benchmark_serving \
  --model <model> \
  --num-prompts 5 \
  --request-rate 1 \
  --max-input-len 128 \
  --max-output-len 256
```

```bash llama.cpp bench theme={"theme":{"light":"github-light","dark":"css-variables"}}
./llama-bench \
  -m <model>.gguf \
  -p 512 \
  -n 128 \
  -ngl 99 \
  --repetitions 3
```

```bash Ollama eval theme={"theme":{"light":"github-light","dark":"css-variables"}}
ollama run <model> --verbose "Write a short paragraph about inference speed."
# The --verbose flag prints eval rate in tok/s
```

### Measure prefill latency

Time how long the model takes to produce its 1st token at varying prompt lengths.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# vLLM — vary --max-input-len to test prefill at different prompt sizes
python -m vllm.benchmarks.benchmark_serving \
  --model <model> \
  --num-prompts 3 \
  --request-rate 1 \
  --max-input-len 2048 \
  --max-output-len 1
```

Record time-to-first-token (TTFT) from the output. Repeat at 512, 1024, 4096, and 8192 input tokens.

### Measure power draw

```bash nvidia-smi (discrete GPUs) theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Sample power every second during inference
watch -n 1 nvidia-smi --query-gpu=power.draw,power.limit --format=csv
```

```bash tegrastats (DGX Spark) theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Reports SoC power on Grace/Blackwell
tegrastats --interval 1000
```

### What to report

Fill in this table for each precision level you test:

| Metric | FP16 | FP8 | INT4 / NVFP4 |
| - | :-: | :-: | :-: |
| Decode (tok/s) | | | |
| Prefill (ms/token, 4K prompt) | | | |
| Prefill (ms/token, 16K prompt) | | | |
| TTFT (s, 8K prompt) | | | |
| Power at decode load (W) | | | |
| Power idle (W) | | | |

## Real-world decision table

Pick the row that matches your primary workload:

| Your use case | Prioritize | Recommended precision | Why |
| - | - | - | - |
| Interactive chatbot | Decode speed | Q4\_K\_M or FP8 | Short prompts, long replies — decode dominates your experience |
| Document analysis | Prefill speed | FP16 or FP8 | Long prompts (10K+ tokens) — prefill dominates latency |
| Laptop / battery operation | Power efficiency | INT4 (Q4\_K\_M, AWQ) | Smallest memory footprint, lowest power draw |
| Multi-user serving | Both | FP8 with batching | Balance quality and throughput; FP8 KV cache doubles concurrency |
| Maximum output quality | Quality only | FP16 / BF16 | Accept higher memory and power cost for zero quantization loss |
| DGX Spark deployment | Decode + memory fit | NVFP4 | Native SM121 support, 75% size reduction, fits large models in 128GB |

> [accordion] Why not always use INT4?
>
> INT4 quantization (GPTQ, AWQ, Q4\_K\_M) reduces quality noticeably on smaller models and instruction-following tasks. On models under 13B parameters, the quality gap between FP16 and INT4 can degrade output. On 70B+ models, the gap narrows because the model has more capacity to absorb quantization noise. Test your specific model before committing.

> [accordion] Does KV cache precision affect these numbers?
>
> Yes. FP8 KV cache (set with `--kv-cache-dtype fp8` in vLLM) halves the KV cache memory independently of weight quantization. It has minimal impact on decode speed but lets you fit longer contexts or more concurrent users. See [DGX Spark memory tuning](https://docs.tokios.com/guides/dgx-spark-memory-tuning.md) for sizing tables.

> [accordion] How does context length change decode speed?
>
> Decode reads the entire KV cache on every token. Longer conversations build larger caches, so decode slows down as context grows. A model at 4K context may generate at 40 tok/s but drop to 25 tok/s at 32K context on the same hardware. See [KV cache and context](https://docs.tokios.com/concepts/kv-cache-and-context.md) for the mechanics.

## Quick benchmark commands

Use these as starting points to compare precisions on a single model.

### vLLM benchmark suite

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Launch with a specific quantization
vllm serve <model> --quantization fp8 --max-model-len 8192

# Run the built-in benchmark
python -m vllm.benchmarks.benchmark_serving \
  --model <model> \
  --num-prompts 10 \
  --request-rate 2 \
  --max-input-len 1024 \
  --max-output-len 128

# Repeat with --quantization nvfp4, then compare results
```

### llama.cpp benchmark

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# FP16 baseline
./llama-bench -m model-f16.gguf -p 512 -n 128 -ngl 99

# Q4_K_M quantized
./llama-bench -m model-q4_k_m.gguf -p 512 -n 128 -ngl 99

# Q8_0 for comparison
./llama-bench -m model-q8_0.gguf -p 512 -n 128 -ngl 99
```

llama.cpp prints a comparison table showing prompt processing (pp) and text generation (tg) speeds side by side.

### Ollama built-in evaluation

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Run with verbose output to see eval speed
ollama run <model> --verbose "Summarize the theory of general relativity in 3 sentences."

# The output includes:
# eval count:   42 generation(s)
# eval duration: 1.8s
# eval rate:    23.3 tokens/s
```

### Compare across precisions

Run the same prompt at each precision and record results in a table:

| Precision | pp (ms/token) | tg (tok/s) | Memory (GB) |
| - | :-: | :-: | :-: |
| FP16 | | | |
| FP8 | | | |
| Q4\_K\_M | | | |

> [!TIP]
>
> For the most reproducible results, warm up the model with 1 request before measuring, close other GPU workloads, and run at least 3 repetitions. Report the median value.

## Next steps

> [card] [DGX Spark memory tuning](https://docs.tokios.com/guides/dgx-spark-memory-tuning.md)
>
> KV cache sizing, batch trade-offs, and tuning parameters for 128GB unified memory.

> [card] [vLLM on DGX Spark](https://docs.tokios.com/guides/vllm-dgx-spark.md)
>
> Installation, NVFP4 configuration, and optimal launch commands.

> [card] [Test your deployment](https://docs.tokios.com/guides/benchmark-your-deployment.md)
>
> End-to-end testing through Tokios — latency, streaming, and client comparison.

> [card] [Quantization concepts](https://docs.tokios.com/concepts/quantization.md)
>
> Deeper explanation of GPTQ, AWQ, GGUF, and quantization trade-offs.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.