> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Local LLM recipes

> Copy-paste configurations for common local LLM setups. Each recipe includes the exact commands, settings, and hardware requirements.

Pick a recipe that matches your hardware and use case. Each one gives you the exact commands to get a working local model and expose it through Tokios in minutes.

> [!NOTE]
>
> `DeepSeek-V4-Flash-0731` needs a multi-device configuration and has different support levels on 2× RTX PRO 6000, 2× DGX Spark, and 2× Strix Halo. Its commands live in the [DeepSeek V4 Flash local hardware guide](https://docs.tokios.com/blog/deepseek-v4-flash-local-hardware.md), not in these general recipes.

> [accordion] Recipe 1: Personal coding assistant (8 GB VRAM, 10 min)
>
> **Hardware:** 8 GB VRAM GPU (RTX 3060, 4060, or similar)
> **Use case:** Code completion and chat inside your editor
> **Backend:** Ollama + Continue or Cline
>
> **1. Install Ollama and pull a 7B coder model**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> # Install Ollama (Linux)
> curl -fsSL https://ollama.com/install.sh | sh
>
> # Pull a coding-optimized 7B model
> ollama pull qwen2.5-coder:7b
> ```
>
> **2. Verify the model responds**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> curl http://127.0.0.1:11434/v1/chat/completions \
>   -H "Content-Type: application/json" \
>   -d '{
>     "model": "qwen2.5-coder:7b",
>     "messages": [{"role": "user", "content": "Write a Python hello world"}],
>     "max_tokens": 100
>   }'
> ```
>
> **3. Install the Tokios connector and register a deployment**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
> ```
>
> In the [dashboard](https://tokios.com/console), register a deployment named `my-coder` pointing at upstream `qwen2.5-coder:7b`. Create an API key.
>
> **4. Configure Continue or Cline**
>
> ```json theme={"theme":{"light":"github-light","dark":"css-variables"}}
> {
>   "baseURL": "https://api.tokios.com/v1",
>   "apiKey": "sk-tok-YOUR_KEY",
>   "model": "my-coder"
> }
> ```
>
> **Expected performance:** \~40 tok/s on an RTX 4060. Responsive tab completion and inline edits with 4K context.

> [accordion] Recipe 2: Family chatbot server (16 GB VRAM, 15 min)
>
> **Hardware:** 16 GB VRAM GPU (RTX 4080, 4070 Ti Super, or similar)
> **Use case:** Self-hosted ChatGPT-style interface for household use
> **Backend:** Ollama + Open WebUI + Tokios
>
> **1. Start a 14B chat model with Ollama**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> ollama pull qwen2.5:14b
> ```
>
> **2. Run Open WebUI with Docker**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> docker run -d \
>   --name open-webui \
>   -p 3000:8080 \
>   -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
>   -v open-webui:/app/backend/data \
>   --restart always \
>   ghcr.io/open-webui/open-webui:main
> ```
>
> Open `http://localhost:3000` and create an admin account.
>
> **3. Expose through Tokios for remote access**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
> ```
>
> Register a deployment named `family-chat` pointing at `qwen2.5:14b`. Create an API key for each family member — scoped keys let you revoke access individually.
>
> **Expected performance:** \~25 tok/s on an RTX 4080 with 8K context. Comfortable for multi-turn conversations.
>
> > [!TIP]
> >
> > Create separate API keys per family member so you can revoke one without affecting others.

> [accordion] Recipe 3: Team API endpoint (48 GB+ VRAM, 30 min)
>
> **Hardware:** 48 GB+ VRAM (A6000, 2x RTX 4090, or A100)
> **Use case:** Shared model endpoint for a development team
> **Backend:** vLLM + Tokios
>
> **1. Install vLLM and launch a 70B model**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> pip install vllm
> ```
>
> ```bash Single A6000 (48 GB) theme={"theme":{"light":"github-light","dark":"css-variables"}}
> vllm serve Qwen/Qwen2.5-72B-Instruct-AWQ \
>   --host 127.0.0.1 \
>   --port 8000 \
>   --quantization awq \
>   --gpu-memory-utilization 0.90 \
>   --max-model-len 8192
> ```
>
> ```bash 2x RTX 4090 (tensor parallel) theme={"theme":{"light":"github-light","dark":"css-variables"}}
> vllm serve Qwen/Qwen2.5-72B-Instruct-AWQ \
>   --host 127.0.0.1 \
>   --port 8000 \
>   --quantization awq \
>   --tensor-parallel-size 2 \
>   --gpu-memory-utilization 0.90 \
>   --max-model-len 8192
> ```
>
> **2. Verify the server responds**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> curl http://127.0.0.1:8000/v1/models
> ```
>
> **3. Install the Tokios connector**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
> ```
>
> Point the connector at `http://127.0.0.1:8000/v1`.
>
> **4. Register a deployment and create scoped keys**
>
> Register a deployment named `team-70b`. In the **Keys** tab, create 1 key per teammate. Scope each key to the `team-*` pattern so teammates can only reach shared deployments.
>
> **Expected performance:** \~30 tok/s aggregate throughput with AWQ quantization. Handles 3–5 concurrent requests with acceptable latency.
>
> > [!NOTE]
> >
> > With 2 GPUs, tensor parallelism gives better decode throughput than pipeline parallelism on a single node. See [Tensor vs pipeline parallelism](https://docs.tokios.com/guides/tensor-vs-pipeline-parallelism.md).

> [accordion] Recipe 4: Document Q\&A pipeline (24 GB VRAM, 45 min)
>
> **Hardware:** 24 GB VRAM GPU (RTX 3090, 4090, or similar)
> **Use case:** Ask questions about your PDFs and documents without sending them to the cloud
> **Backend:** Ollama (embedding + chat) + custom RAG script + Tokios
>
> **1. Pull an embedding model and a 32B chat model**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> ollama pull nomic-embed-text
> ollama pull qwen2.5:32b
> ```
>
> The embedding model (\~270 MB) creates vector representations of your documents. The 32B model answers questions grounded in those embeddings.
>
> **2. Build a simple RAG pipeline**
>
> ```python theme={"theme":{"light":"github-light","dark":"css-variables"}}
> import requests, json
>
> EMBED_MODEL = "nomic-embed-text"
> CHAT_MODEL  = "qwen2.5:32b"
> OLLAMA_URL  = "http://127.0.0.1:11434"
>
> def embed(text: str) -> list[float]:
>     r = requests.post(f"{OLLAMA_URL}/api/embeddings",
>                        json={"model": EMBED_MODEL, "prompt": text})
>     return r.json()["embedding"]
>
> def ask(question: str, context: str) -> str:
>     prompt = f"Answer based on this context:\n\n{context}\n\nQuestion: {question}"
>     r = requests.post(f"{OLLAMA_URL}/v1/chat/completions",
>                        json={"model": CHAT_MODEL,
>                              "messages": [{"role": "user", "content": prompt}],
>                              "max_tokens": 512})
>     return r.json()["choices"][0]["message"]["content"]
> ```
>
> **3. Expose the chat model through Tokios**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
> ```
>
> Register a deployment named `doc-qa` pointing at `qwen2.5:32b`. Your RAG script can now call the model through `https://api.tokios.com/v1` from any machine, while embeddings stay local.
>
> **Expected performance:** \~15 tok/s for the 32B model on an RTX 4090. Embedding is near-instant for short passages.
>
> > [!TIP]
> >
> > Keep the embedding model local (it is fast and small) and route only the chat model through Tokios. This avoids sending document text over the network for embedding.

> [accordion] Recipe 5: DGX Spark max throughput (128 GB, 20 min)
>
> **Hardware:** NVIDIA DGX Spark (128 GB unified memory)
> **Use case:** Maximum throughput for large MoE models
> **Backend:** vLLM + Tokios
>
> **1. Launch a 120B MoE model at FP8**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> vllm serve Qwen/Qwen3-235B-A22B-FP8 \
>   --host 127.0.0.1 \
>   --port 8000 \
>   --quantization fp8 \
>   --gpu-memory-utilization 0.85 \
>   --max-model-len 16384 \
>   --trust-remote-code
> ```
>
> The DGX Spark's 128 GB unified memory fits models that need 2–3 consumer GPUs. MoE models like Qwen3-235B-A22B only activate \~22B parameters per token, so inference is fast despite the large total parameter count.
>
> **2. Verify and benchmark**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> curl http://127.0.0.1:8000/v1/chat/completions \
>   -H "Content-Type: application/json" \
>   -d '{
>     "model": "Qwen/Qwen3-235B-A22B-FP8",
>     "messages": [{"role": "user", "content": "Explain quantum entanglement"}],
>     "max_tokens": 256
>   }'
> ```
>
> **3. Expose through Tokios**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
> ```
>
> Register a deployment named `spark-moe`. Create keys for each client that needs access.
>
> **Expected performance:** \~35 tok/s on DGX Spark for the active parameter set. 16K context fits comfortably in unified memory.
>
> > [!NOTE]
> >
> > See [vLLM on DGX Spark](https://docs.tokios.com/guides/vllm-dgx-spark.md) and [DGX Spark memory tuning](https://docs.tokios.com/guides/dgx-spark-memory-tuning.md) for detailed configuration options.

> [accordion] Recipe 6: Laptop-friendly offline AI (16 GB RAM, 5 min)
>
> **Hardware:** Any laptop with 16 GB RAM (no dedicated GPU required)
> **Use case:** Private AI assistant that works without internet
> **Backend:** Ollama (CPU mode)
>
> **1. Install Ollama and pull a quantized 7B model**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> # macOS
> curl -fsSL https://ollama.com/install.sh | sh
>
> # Or download from https://ollama.com for Windows/Linux
> ollama pull qwen2.5:7b-instruct-q4_K_M
> ```
>
> Q4\_K\_M quantization keeps the model under 5 GB, so it runs on CPU with 16 GB RAM.
>
> **2. Chat locally — no internet required**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> ollama run qwen2.5:7b-instruct-q4_K_M
> ```
>
> That is it. The model runs entirely on your laptop. No network calls, no telemetry, no API keys.
>
> **3. Optionally expose through Tokios when online**
>
> When you have internet and want to reach the model from another device:
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
> ```
>
> Register a deployment named `laptop-offline`. The connector only dials out when the network is available — offline, your model keeps working locally.
>
> **Expected performance:** \~5 tok/s on CPU (modern laptop). Usable for chat and drafting, not for real-time coding assistance.
>
> > [!TIP]
> >
> > If your laptop has an Apple silicon chip, Ollama uses the Neural Engine and Metal automatically — expect 20–30 tok/s on M-series chips.

> [accordion] Recipe 7: Multi-model routing (24 GB+ VRAM, 30 min)
>
> **Hardware:** 24 GB+ VRAM GPU
> **Use case:** Serve 3 models of different sizes through 1 Tokios endpoint
> **Backend:** Ollama or vLLM + Tokios with config file
>
> **1. Pull 3 models at different sizes**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> ollama pull qwen2.5:3b     # fast-small: quick classification
> ollama pull qwen2.5:14b    # medium: general chat
> ollama pull qwen2.5:32b    # large: complex reasoning
> ```
>
> **2. Create a multi-route connector config**
>
> Create `tokios-connector.json`:
>
> ```json theme={"theme":{"light":"github-light","dark":"css-variables"}}
> {
>   "Routes": [
>     { "Deployment": "fast-small", "Upstream": "local-ollama" },
>     { "Deployment": "medium",     "Upstream": "local-ollama" },
>     { "Deployment": "large",      "Upstream": "local-ollama" }
>   ],
>   "Upstreams": {
>     "local-ollama": {
>       "BaseUrl": "http://127.0.0.1:11434/v1"
>     }
>   }
> }
> ```
>
> **3. Register all 3 deployments in the dashboard**
>
> In the **Models** tab, register:
>
> | Deployment name | Upstream model id |
> | - | - |
> | `fast-small` | `qwen2.5:3b` |
> | `medium` | `qwen2.5:14b` |
> | `large` | `qwen2.5:32b` |
>
> **4. Route by sending the model field**
>
> ```python theme={"theme":{"light":"github-light","dark":"css-variables"}}
> from openai import OpenAI
>
> client = OpenAI(
>     base_url="https://api.tokios.com/v1",
>     api_key="sk-tok-YOUR_KEY",
> )
>
> # Fast classification
> client.chat.completions.create(
>     model="fast-small",
>     messages=[{"role": "user", "content": "Is this email spam? ..."}],
> )
>
> # Complex reasoning
> client.chat.completions.create(
>     model="large",
>     messages=[{"role": "user", "content": "Analyze this contract clause..."}],
> )
> ```
>
> **Expected performance:** The 3B model responds in under 1 second. The 32B model takes 2–3 seconds for the first token but produces higher-quality output. Same endpoint, same key, different `model` field.
>
> > [!NOTE]
> >
> > Ollama keeps only 1 model in VRAM at a time by default and swaps on request. If you need all 3 loaded simultaneously, run separate backends or use vLLM with LoRA adapters. See [Model routing](https://docs.tokios.com/guides/model-routing.md).

> [accordion] Recipe 8: Benchmarking your setup (any hardware, 10 min)
>
> **Hardware:** Any GPU or CPU running a local model
> **Use case:** Compare throughput and latency across models, quantizations, or hardware
> **Backend:** Any + Tokios
>
> **1. Start your model server**
>
> Use any backend — Ollama, vLLM, llama.cpp, or LM Studio. Confirm it responds on its `/v1` endpoint.
>
> **2. Run a throughput benchmark**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> # Simple throughput test with curl and timing
> for i in 1 2 3 4 5; do
>   time curl -s http://127.0.0.1:11434/v1/chat/completions \
>     -H "Content-Type: application/json" \
>     -d '{
>       "model": "qwen2.5:7b",
>       "messages": [{"role": "user", "content": "Write a 200-word essay about recursion"}],
>       "max_tokens": 256,
>       "stream": false
>     }' | python3 -c "
> import sys, json
> r = json.load(sys.stdin)
> u = r['usage']
> print(f\"Tokens: {u['completion_tokens']}, Time: check wall clock\")
> "
> done
> ```
>
> **3. Benchmark through Tokios to measure end-to-end latency**
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> # Same test through the Tokios endpoint
> time curl -s https://api.tokios.com/v1/chat/completions \
>   -H "Authorization: Bearer sk-tok-YOUR_KEY" \
>   -H "Content-Type: application/json" \
>   -d '{
>     "model": "my-deployment",
>     "messages": [{"role": "user", "content": "Write a 200-word essay about recursion"}],
>     "max_tokens": 256,
>     "stream": false
>   }' | python3 -c "
> import sys, json
> r = json.load(sys.stdin)
> u = r['usage']
> print(f\"Tokens: {u['completion_tokens']}\")
> "
> ```
>
> **4. Compare quantizations**
>
> Pull the same model at different quantizations and benchmark each:
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> ollama pull qwen2.5:7b-instruct-q4_K_M
> ollama pull qwen2.5:7b-instruct-q5_K_M
> ollama pull qwen2.5:7b-instruct-q8_0
> ```
>
> Run the same prompt against each and compare tokens per second. Q5\_K\_M typically gives the best quality-to-speed trade-off at 7B.
>
> **Expected results:** The tunnel adds 50–150 ms of latency per request (1 round trip to the Tokios gateway). Streaming throughput is nearly identical to local — the tunnel does not buffer SSE.
>
> > [!TIP]
> >
> > For a more structured benchmark, see [Test your deployment](https://docs.tokios.com/guides/benchmark-your-deployment.md).

## Next steps

> [card] [Case studies](https://docs.tokios.com/guides/local-llm-case-studies.md)
>
> Real deployment patterns from teams and individuals running local models.

> [card] [Choose a local model](https://docs.tokios.com/guides/choose-a-local-model.md)
>
> Pick the right model and quantization for your hardware and workflow.

> [card] [Model routing](https://docs.tokios.com/guides/model-routing.md)
>
> Register multiple deployments and switch models by changing the model field.

> [card] [Quickstart](https://docs.tokios.com/quickstart.md)
>
> Go from zero to a working connector, deployment, and API key.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.