> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Supported Local Model Backends — Tokios Connector

> Point the Tokios Connector at Ollama, llama.cpp, vLLM, LM Studio, MLX Studio, MLX-LM Server, or Lemonade Server by setting BaseUrl (/v1) in tokios-connector.json — must speak OpenAI Chat Completions.

The Tokios Connector forwards to any model server that exposes an **OpenAI-compatible Chat Completions API** (`/v1/chat/completions`) on your machine. Ollama, llama.cpp, vLLM, LM Studio, MLX Studio, and MLX-LM Server all do. The gateway translates each incoming request to Chat Completions and sends it down the tunnel; the connector forwards it to the upstream `BaseUrl` you set in `tokios-connector.json`. Point the connector at your server by setting that `BaseUrl` — **including the `/v1` segment**.

> [!NOTE]
>
> Always include `/v1` in `BaseUrl`, and use your backend's **OpenAI-compatible** endpoint. For Ollama that's the `/v1` API (not the native `/api`).

> [!TIP]
>
> **New to local models?** Start with the easiest to run: **[Ollama](https://docs.tokios.com/guides/ollama-remote-access.md)** (one command) or **[LM Studio](https://docs.tokios.com/guides/lm-studio-remote-access.md)** (a point-and-click desktop app). **vLLM** and **SGLang** are powerful GPU servers but need more setup — reach for them once you're comfortable and want higher throughput.

## Ease of setup

| Backend | Setup style | Best for |
| - | - | - |
| **Ollama** | One command (`ollama run`) | Beginners; quickest start |
| **LM Studio** | Desktop app, point-and-click | Beginners who prefer a GUI |
| **MLX Studio** | Native macOS app, point-and-click | Mac users; GUI with bundled runtime, no terminal |
| **Jan** | Desktop app, point-and-click | Beginners who prefer a GUI |
| **Unsloth Studio** | No-code web UI | Beginners; also trains and exports models |
| **llama.cpp** | Build/run `llama-server` with a GGUF | Hands-on users running GGUF directly |
| **Docker Model Runner** | Container CLI (`docker model run`) | Users already running Docker |
| **vLLM** | Python GPU server, flags to tune | Advanced; high-throughput GPU serving |
| **SGLang** | Python GPU server, flags to tune | Advanced; high-throughput GPU serving |
| **ExecuTorch** | PyTorch export + build toolchain | Advanced; on-device and edge deployment |
| **Lemonade Server** | Cross-platform installer (MSI, APT, AUR) | AMD NPU support; auto-detects Vulkan/ROCm/CUDA/Metal |
| **MLX-LM Server** | Python package (`pip install`) | Apple Silicon; native unified-memory inference |
| **FreeToken** | Python package (`uv pip install`) | MoE models on consumer NVIDIA GPUs; dynamic CPU-GPU co-scheduling |

Whichever you pick, Tokios connects to it the same way — set the upstream `BaseUrl` (with `/v1`) and you get one endpoint.

> [!NOTE]
>
> This list isn't exclusive. The connector forwards to **any** server that exposes an OpenAI-compatible Chat Completions API at a `/v1` base URL — so **SGLang**, **Xinference**, **LocalAI**, and similar all work too. Ollama, llama.cpp, vLLM, and LM Studio are simply the ones documented most closely here.

> [!TIP]
>
> Jan can also run as a **client** — instead of serving a model locally, it can call a bigger model running elsewhere through Tokios. See [Use Jan with a model on your own hardware](https://docs.tokios.com/guides/jan.md).

## Default ports

Drop the matching `Upstream` into your `tokios-connector.json` (see [Connector Config](https://docs.tokios.com/reference/connector-config.md) for the full file):

```json Ollama theme={"theme":{"light":"github-light","dark":"css-variables"}}
"Upstream": { "BaseUrl": "http://127.0.0.1:11434/v1" }
```

```json llama.cpp theme={"theme":{"light":"github-light","dark":"css-variables"}}
"Upstream": { "BaseUrl": "http://127.0.0.1:8080/v1" }
```

```json vLLM theme={"theme":{"light":"github-light","dark":"css-variables"}}
"Upstream": { "BaseUrl": "http://127.0.0.1:8000/v1" }
```

```json LM Studio theme={"theme":{"light":"github-light","dark":"css-variables"}}
"Upstream": { "BaseUrl": "http://127.0.0.1:1234/v1" }
```

```json MLX Studio theme={"theme":{"light":"github-light","dark":"css-variables"}}
"Upstream": { "BaseUrl": "http://127.0.0.1:8000/v1" }
```

```json Lemonade Server theme={"theme":{"light":"github-light","dark":"css-variables"}}
"Upstream": { "BaseUrl": "http://127.0.0.1:13305/v1" }
```

```json MLX-LM Server theme={"theme":{"light":"github-light","dark":"css-variables"}}
"Upstream": { "BaseUrl": "http://127.0.0.1:8080/v1" }
```

```json FreeToken theme={"theme":{"light":"github-light","dark":"css-variables"}}
"Upstream": { "BaseUrl": "http://127.0.0.1:8000/v1" }
```

> [!NOTE]
>
> The model id your server uses (for Ollama, whatever you `ollama run`) is what you list in the connector's `Models` / `Routes`. You give it a public API name when you [register a deployment](https://docs.tokios.com/setup/register-model.md).

## Backend details

### Ollama

**Upstream:** `http://127.0.0.1:11434/v1`

Ollama serves an OpenAI-compatible API at `/v1` on port `11434`. Pull and run your model first — the id you give Ollama is the upstream model id the connector serves:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
ollama run gemma4:e4b
```

### llama.cpp

**Upstream:** `http://127.0.0.1:8080/v1`

Run the bundled OpenAI-compatible server, `llama-server`. It listens on port `8080` by default and serves `/v1`:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
llama-server -m ./models/my-model.gguf --port 8080
```

### vLLM

**Upstream:** `http://127.0.0.1:8000/v1`

vLLM's OpenAI-compatible server listens on port `8000`. Start it with:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve mistralai/Mistral-7B-Instruct-v0.2
```

### LM Studio

**Upstream:** `http://127.0.0.1:1234/v1`

LM Studio includes a built-in OpenAI-compatible server:

1. Open LM Studio and go to the **Developer** (Local Server) tab.
2. Load a model and start the server. It listens on port `1234`.

### MLX Studio

**Upstream:** `http://127.0.0.1:8000/v1`

MLX Studio is a native macOS app (macOS 14.0+ Sonoma, all Apple Silicon M1–M4) that wraps the [vMLX](https://pypi.org/project/vmlx/) inference server. It ships with a bundled Python 3.12 environment — no terminal, no `pip`, no `venv` required. Download the app, pick a model from any [mlx-community](https://huggingface.co/mlx-community) repository on Hugging Face, and start the built-in API server.

MLX Studio includes chat, vision, image generation, text-to-speech, speech-to-text, and tool calling. Its inference engine supports JANG mixed-precision quantization, continuous batching with PagedAttention, speculative decoding, KV cache quantization, and prefix caching. An interactive API documentation and testing playground is built in, and a menu-bar mode gives quick access without a full window.

1. Open MLX Studio and download a model from the **Model Manager**.
2. Start the API server from the **Server** tab. It listens on port `8000`.

> [!TIP]
>
> MLX Studio is the easiest GUI path for Mac users who want a native Apple Silicon experience without touching the terminal. For a headless or scriptable setup on the same hardware, use [MLX-LM Server](#mlx-lm-server) instead.

### Lemonade Server

**Upstream:** `http://127.0.0.1:13305/v1`

Lemonade Server is an open-source inference server backed by AMD. It auto-detects your hardware and selects the best backend — Vulkan, ROCm, CUDA, or Metal — so you don't pick a backend yourself. It is the only backend here that supports AMD Ryzen AI NPUs. Install via MSI on Windows, APT PPA on Ubuntu, or AUR on Arch. See [lemonade-server.ai](https://lemonade-server.ai/) for details.

Lemonade Server exposes an OpenAI-compatible API at `/v1` on port `13305` by default. It also supports Ollama-compatible and Anthropic-compatible endpoints.

> [!TIP]
>
> Lemonade Server simplifies setup on AMD Strix Halo and DGX Spark boxes — it picks Vulkan or CUDA automatically, no manual backend flags needed.

### MLX-LM Server

**Upstream:** `http://127.0.0.1:8080/v1`

MLX-LM Server is part of the [`mlx-lm`](https://github.com/ml-explore/mlx-lm) Python package and runs natively on Apple Silicon. It uses the MLX framework's unified-memory architecture with zero-copy data paths, so model weights live in the same memory pool as the CPU and GPU — no separate VRAM to manage. Install the package and start the server with any [mlx-community](https://huggingface.co/mlx-community) model from Hugging Face:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
pip install mlx-lm
mlx_lm.server --model mlx-community/Qwen-3.5-4B-8bit
```

The server listens on port `8080` by default and exposes an OpenAI-compatible Chat Completions API at `/v1`. It supports continuous batching, tool calling, structured output, and prompt caching. Available quantizations include 4-bit (default), 8-bit, and BF16.

> [!TIP]
>
> MLX-LM Server is the native Apple Silicon path — it competes with or slightly outperforms llama.cpp on M-series chips and takes full advantage of M5 Neural Accelerators (4x faster prompt processing). See [mlx-framework.org](https://mlx-framework.org) for details.

> [!TIP]
>
> Running on a different port? Just change the port in `BaseUrl`. Any server that speaks OpenAI Chat Completions works.

### FreeToken

**Upstream:** `http://127.0.0.1:8000/v1`

FreeToken is an edge-native inference engine from UC Berkeley, purpose-built for running large Mixture-of-Experts (MoE) models on consumer NVIDIA GPUs. It uses dynamic CPU-GPU co-scheduling instead of static offloading, enabling 35B-753B parameter MoE models to run on 8GB GPUs at \~39 tokens/s — 3-4x faster decode than Ollama.

Install with:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
uv pip install "freetoken[accel]"
```

FreeToken currently supports DeepSeek-V4-Flash (284B), Qwen3.6-35B, and GLM-5.2 (753B) on NVIDIA RTX 30/40/50 series GPUs. Apple Silicon is not yet supported. See [FreeToken MoE inference](https://docs.tokios.com/guides/freetoken-moe-inference.md) for setup and performance tuning.

> [!TIP]
>
> FreeToken is the fastest path for running frontier-scale MoE models on consumer hardware. If you're running Qwen3.6-35B or DeepSeek-V4-Flash on an 8-24GB NVIDIA GPU, start here.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.