> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Model routing strategies for local models

> Register multiple local models behind 1 Tokios endpoint and route requests by task type, cost, or resilience — from simple switching to fallback chains.

Tokios lets you expose multiple models — each under its own public name — and route between them by changing the `model` field in your request. The endpoint and API key stay the same; only the `model` value changes.

## How routing works

Each deployment has a unique public name. When Tokios receives a request, it reads the `model` field, finds the deployment with that name, rewrites it to that deployment's upstream model id, and forwards it down the connector's tunnel. Your client config never changes — only the `model` value.

## One connector, many models

A single connector can serve several models. List each upstream model id in `Routes` and map it to a named local upstream in `Upstreams`:

```json tokios-connector.json theme={"theme":{"light":"github-light","dark":"css-variables"}}
{
  "Connector": {
    "ConnectorId": "con_...",
    "Token": "ct-...",
    "Routes": [
      { "Model": "gemma4:e4b",    "UpstreamId": "ollama" },
      { "Model": "qwen2.5-coder", "UpstreamId": "lmstudio" }
    ],
    "Upstreams": {
      "ollama":   { "BaseUrl": "http://127.0.0.1:11434/v1" },
      "lmstudio": { "BaseUrl": "http://127.0.0.1:1234/v1" }
    }
  }
}
```

Then register a **deployment** for each model in the console, giving each a public name (e.g. `gemma-tunnel` → `gemma4:e4b`, `qwen-coder` → `qwen2.5-coder`), all pointing at this connector.

> [!TIP]
>
> Prefer to keep them separate? You can also run one connector per model, each with its own `ConnectorId`/`Token` and `Upstream`. Either approach works — the server only ever sends the model id down the tunnel.

## Switching models in requests

With both deployments registered, switch models by changing only the `model` field:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Use the Gemma model
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "gemma-tunnel", "messages": [{"role": "user", "content": "Hello"}]}'

# Use the Qwen model
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen-coder", "messages": [{"role": "user", "content": "Hello"}]}'
```

The endpoint, the auth header, and every other parameter stay identical. Tokios handles the dispatch.

## Swap the backend without touching clients

Because clients only reference the public name, you can move a model to different hardware or a different backend without changing any client config: update the deployment (or the connector's upstream) to point at the new server, keep the **same public name**, and existing requests keep working.

## Routing strategies

When you run multiple local models, sending every request to the same model wastes resources. A coding request doesn't need a 120B reasoning model, and a simple classification task doesn't need a 70B dense model. Routing lets you direct each request to the model best suited for the task — optimizing for speed, quality, or cost depending on the workload.

Running 1 model for everything forces a trade-off. A large model handles every task well but responds slowly and consumes more VRAM. A small model responds fast but struggles with complex reasoning. Routing lets you have both:

* **Speed**: simple queries go to a small, fast model; complex tasks go to a large model
* **Quality**: code tasks go to a code-specialized model; general chat goes to a generalist
* **Cost**: smaller models consume less power and VRAM, leaving headroom for demanding tasks
* **Resilience**: if your primary model is busy or down, requests fall through to a backup

> [accordion] Single model per deployment
>
> The simplest approach — register each model as its own Tokios deployment and let the client choose which to call by changing the `model` field.
>
> ```json tokios-connector.json theme={"theme":{"light":"github-light","dark":"css-variables"}}
> {
>   "Connector": {
>     "Routes": [
>       { "Model": "qwen3-coder", "UpstreamId": "ollama" },
>       { "Model": "gemma-tunnel", "UpstreamId": "ollama" }
>     ],
>     "Upstreams": {
>       "ollama": { "BaseUrl": "http://127.0.0.1:11434/v1" }
>     }
>   }
> }
> ```
>
> Your client sends requests with the appropriate `model` value:
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> # Code task → code model
> curl https://api.tokios.com/v1/chat/completions \
>   -H "Authorization: Bearer sk-tok-YOUR_KEY" \
>   -H "Content-Type: application/json" \
>   -d '{"model": "qwen3-coder", "messages": [{"role": "user", "content": "Write a Python function to sort a list"}]}'
>
> # General chat → general model
> curl https://api.tokios.com/v1/chat/completions \
>   -H "Authorization: Bearer sk-tok-YOUR_KEY" \
>   -H "Content-Type: application/json" \
>   -d '{"model": "gemma-tunnel", "messages": [{"role": "user", "content": "Summarize this article"}]}'
> ```
>
> **When to use**: you run 2–3 models and your application controls which model to call. This is the most common starting point.

> [accordion] Task-based routing
>
> Route by task type — send coding requests to a code-specialized model, chat to a generalist, and reasoning to a dense model. The client or a middleware layer inspects the request and picks the deployment.
>
> | Task type | Recommended model class | Example deployment | Why |
> | - | - | - | - |
> | Code generation, refactoring | Code-specialized | Qwen3-Coder-30B on vLLM | Trained specifically for code; better at syntax and logic |
> | General chat, summarization | MoE (35B–122B) | Qwen3.5-35B-A3B on Atlas | Fast throughput; 3B active params per token |
> | Complex reasoning, math | Dense 70B+ | Qwen3-32B or 70B on vLLM | Higher quality on multi-step reasoning |
> | Tool calling, agents | Tool-capable model | GLM-4.5-Air on vLLM | Strong function-calling support |
> | Vision, multimodal | Multimodal model | Qwen3-VL on vLLM | Handles image input natively |
>
> The simplest implementation is in your application code — inspect the request type and select the deployment name:
>
> ```python theme={"theme":{"light":"github-light","dark":"css-variables"}}
> def select_model(task_type: str) -> str:
>     routing = {
>         "code": "qwen3-coder",
>         "chat": "gemma-tunnel",
>         "reasoning": "glm-4-chat",
>         "vision": "qwen3-vl",
>     }
>     return routing.get(task_type, "gemma-tunnel")
> ```
>
> For more sophisticated routing, use a gateway like LiteLLM or OpenRouter that inspects requests and forwards to the correct backend based on configurable rules.

> [accordion] Gateway-level routing
>
> Place a routing proxy between your clients and Tokios. The proxy decides which Tokios deployment to use based on request attributes, load, or custom rules.
>
> LiteLLM can sit in front of multiple Tokios deployments and route based on model aliases:
>
> ```yaml litellm_config.yaml theme={"theme":{"light":"github-light","dark":"css-variables"}}
> model_list:
>   - model_name: gpt-4  # alias
>     litellm_params:
>       model: openai/qwen3-coder  # Tokios deployment
>       api_base: https://api.tokios.com/v1
>       api_key: sk-tok-YOUR_KEY
>   - model_name: gpt-4
>     litellm_params:
>       model: openai/gemma-tunnel  # fallback
>       api_base: https://api.tokios.com/v1
>       api_key: sk-tok-YOUR_KEY
>
> router_settings:
>   routing_strategy: simple-shuffle
>   num_retries: 2
> ```
>
> Clients send requests to the LiteLLM proxy using the alias `gpt-4`, and the proxy routes to whichever Tokios deployment you configured.
>
> OpenRouter supports model fallback and load balancing across multiple providers. Add your Tokios endpoint as a custom provider and configure routing rules in the OpenRouter dashboard.

> [accordion] Fallback chains
>
> Configure a primary model with 1 or more fallback models. If the primary is busy, offline, or returns an error, the request falls through to the next model in the chain.
>
> ```python theme={"theme":{"light":"github-light","dark":"css-variables"}}
> import requests
>
> FALLBACK_MODELS = ["qwen3-coder", "gemma-tunnel", "qwen3-32b"]
>
> def chat_with_fallback(prompt: str) -> str:
>     for model in FALLBACK_MODELS:
>         try:
>             response = requests.post(
>                 "https://api.tokios.com/v1/chat/completions",
>                 headers={
>                     "Authorization": "Bearer sk-tok-YOUR_KEY",
>                     "Content-Type": "application/json"
>                 },
>                 json={"model": model, "messages": [{"role": "user", "content": prompt}]},
>                 timeout=30
>             )
>             response.raise_for_status()
>             return response.json()["choices"][0]["message"]["content"]
>         except (requests.RequestException, KeyError):
>             continue
>     raise RuntimeError("All models failed")
> ```
>
> Tokios connectors report their status to the gateway. If a connector goes offline, requests to deployments on that connector return 503. Your application can catch this and retry with a different deployment, or you can configure a gateway like LiteLLM to handle the fallback automatically.

> [accordion] Cost-based routing
>
> Route simple, high-volume requests to a smaller, cheaper model and reserve the large model for tasks that need it.
>
> | Request type | Route to | Reason |
> | - | - | - |
> | Classification, extraction | MoE with 3–4B active params | Fast; low compute per token |
> | Summarization (short) | 7B–14B dense or MoE | Adequate quality; fast response |
> | Complex reasoning | 32B+ dense or 122B MoE | Higher quality on multi-step tasks |
> | Code generation | Code-specialized model | Better output for the token budget |
>
> This strategy reduces total compute and power consumption — important when running on consumer hardware where VRAM and wattage are limited.

## Decision matrix

| Strategy | Complexity | Best for | When to use |
| - | - | - | - |
| Single model per deployment | Low | Getting started, 2–3 models | You control which model to call in application code |
| Task-based routing | Medium | Coding + chat + reasoning | You need different models for different task types |
| Gateway-level routing | High | Multiple clients, shared infrastructure | You want centralized routing rules without changing client code |
| Fallback chains | Medium | High availability | Your primary model may go offline or hit concurrency limits |
| Cost-based routing | Medium | High-volume, mixed workloads | You want to minimize compute cost by sending simple tasks to smaller models |

> [!NOTE]
>
> These strategies are not mutually exclusive. A common production pattern combines task-based routing (code → code model) with fallback chains (primary → backup) behind a gateway proxy.

## Next steps

> [card] [Choose a local model](https://docs.tokios.com/guides/choose-a-local-model.md)
>
> Compare models and backends to decide which deployments to register.

> [card] [Test your deployment](https://docs.tokios.com/guides/benchmark-your-deployment.md)
>
> Measure throughput and latency to validate your routing decisions under real workloads.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.