> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Run vLLM locally and access it from anywhere through Tokios

> Install vLLM, launch a model, add the Tokios connector, register a deployment, and call your vLLM server from any OpenAI- or Anthropic-compatible client.

This page walks you through the full journey: from a fresh vLLM install on your GPU machine to a remotely accessible, authenticated API endpoint — in about 5 minutes. Your model stays on your hardware. Your prompts never pass through a third-party inference server.

> [!NOTE]
>
> **Already have vLLM running?** Skip to [Step 3: Install the Tokios connector](#step-3-install-the-tokios-connector). If you just want the short version, see [vLLM remote access](https://docs.tokios.com/guides/vllm-remote-access.md).

## Prerequisites

Before you start, make sure you have:

* **1 NVIDIA GPU** with CUDA support (RTX 3090, 4090, A100, or DGX Spark).
* **CUDA 12.1+** installed (run `nvidia-smi` to check).
* **Python 3.9–3.12** with `pip` available.
* A [Tokios account](https://tokios.com/console) (free to sign up).

For multi-GPU setups, see [tensor parallelism](#multi-gpu-tensor-parallelism) below.

## Step 1: Install and launch vLLM

Install vLLM with pip:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
pip install vllm
```

Launch the OpenAI-compatible server. Keep it bound to `127.0.0.1` — don't bind wider:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve mistralai/Mistral-7B-Instruct-v0.2 \
  --host 127.0.0.1 \
  --port 8000
```

vLLM downloads the model weights on first run, loads them into GPU memory, and starts an OpenAI-compatible HTTP server on `http://127.0.0.1:8000`.

### Verify it's running

In another terminal:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl http://127.0.0.1:8000/v1/models
```

You should see your model listed.

### Common launch configurations

Pick the flags that match your hardware:

```bash Single GPU (RTX 3090/4090) theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve mistralai/Mistral-7B-Instruct-v0.2 \
  --host 127.0.0.1 \
  --port 8000 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192
```

```bash Large model with quantization theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve Qwen/Qwen2.5-72B-Instruct-AWQ \
  --host 127.0.0.1 \
  --port 8000 \
  --quantization awq \
  --gpu-memory-utilization 0.90 \
  --max-model-len 4096
```

```bash DGX Spark (NVFP4) theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve nvidia/Qwen3.5-35B-A3B-NVFP4 \
  --host 127.0.0.1 \
  --port 8000 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 8192
```

```bash FP8 KV cache (saves ~50% memory) theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3-8B-Instruct \
  --host 127.0.0.1 \
  --port 8000 \
  --kv-cache-dtype fp8 \
  --max-model-len 16384
```

> [!TIP]
>
> `--max-model-len` controls how much GPU memory vLLM reserves for the KV cache. Start conservative (4096 or 8192) and increase once you confirm the model loads without OOM. See [KV cache and context](https://docs.tokios.com/concepts/kv-cache-and-context.md) for the memory math.

### Multi-GPU: tensor parallelism

For models that don't fit in 1 GPU, use tensor parallelism across multiple GPUs on the same machine:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# 2 GPUs, tensor parallel
vllm serve meta-llama/Llama-3-70B-Instruct \
  --host 127.0.0.1 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90
```

Tensor parallelism is preferred over pipeline parallelism for single-node multi-GPU — it gives better bandwidth utilization during the decode phase.

### Windows (WSL2)

vLLM doesn't run natively on Windows. Use Docker inside WSL2 with GPU passthrough:

```yaml theme={"theme":{"light":"github-light","dark":"css-variables"}}
# docker-compose.yml
services:
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    ports:
      - "127.0.0.1:8000:8000"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    command: >
      --model mistralai/Mistral-7B-Instruct-v0.2
      --host 0.0.0.0
      --port 8000
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
```

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
docker compose up
```

## Step 2: Test locally

Before setting up remote access, confirm vLLM responds correctly:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.2",
    "messages": [{"role": "user", "content": "Write hello world in Python"}],
    "max_tokens": 100
  }'
```

You should get a valid chat completion response.

## Step 3: Install the Tokios connector

The connector dials **out** from your machine to `https://api.tokios.com`. No inbound port opens. No port forwarding. No DNS record.

Sign in at [tokios.com/console](https://tokios.com/console), open the **Setup** tab, and click **Start pairing**. Tokios shows a one-time **claim code** like `TKS-XXXX-XXXX`, valid for about 15 minutes.

Run the install command on the machine running vLLM:

```powershell Windows theme={"theme":{"light":"github-light","dark":"css-variables"}}
irm https://tokios.com/downloads/install/windows | iex; tokios-install -Code 'TKS-XXXX-XXXX'
```

```bash macOS theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX
```

```bash Linux theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
```

Already have `tokios-connector` installed? Pass the code directly:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
tokios-connector TKS-XXXX-XXXX
```

Back in the console, click **Approve** on the **Setup** or **Connectors** tab. The connector's token (`ct-tok-…`) is delivered over the tunnel once approved — you never see it on screen or paste it into a file.

## Step 4: Point the connector at vLLM

vLLM's OpenAI-compatible server speaks the exact surface Tokios expects. For a single model, the console's Setup wizard defaults to `http://127.0.0.1:8000/v1` automatically.

For multiple models or a custom config, create a `tokios-connector.json`:

```json theme={"theme":{"light":"github-light","dark":"css-variables"}}
{
  "Upstream": {
    "BaseUrl": "http://127.0.0.1:8000/v1"
  }
}
```

See [Connector config](https://docs.tokios.com/reference/connector-config.md) for the full multi-model file format.

## Step 5: Register a deployment

In the [Models](https://tokios.com/console/models) tab, register your vLLM model and choose a **public name** — for example, `mistral-tunnel`.

That public name is what clients send in the `model` field. It's decoupled from vLLM's local model id, so you can change the upstream model without breaking client configs.

> [!TIP]
>
> Running LoRA adapters? Start vLLM with `--enable-lora --lora-modules adapter1=/path/to/adapter`, then register each adapter as a separate Tokios deployment. See [Model routing](https://docs.tokios.com/guides/model-routing.md).

## Step 6: Create an API key

In the [Keys](https://tokios.com/console/keys) tab, click **Create key**.

* Optionally scope it to specific deployments with model-name patterns (comma-separated globs, `*` for all).
* Copy the `sk-tok-…` key now — it's shown only once.

You can create multiple keys — one per teammate, agent, or environment — and revoke any of them independently.

## Step 7: Connect from any client

Your vLLM model is now reachable at `https://api.tokios.com`. Here's how to call it:

```bash curl theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-tunnel",
    "stream": true,
    "messages": [{"role": "user", "content": "Write hello world in Rust"}]
  }'
```

```python Python (OpenAI SDK) theme={"theme":{"light":"github-light","dark":"css-variables"}}
from openai import OpenAI

client = OpenAI(
    base_url="https://api.tokios.com/v1",
    api_key="sk-tok-YOUR_KEY",
)

response = client.chat.completions.create(
    model="mistral-tunnel",
    messages=[{"role": "user", "content": "Write hello world in Rust"}],
    stream=True,
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")
```

```bash Claude Code theme={"theme":{"light":"github-light","dark":"css-variables"}}
export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model mistral-tunnel
```

```bash Aider theme={"theme":{"light":"github-light","dark":"css-variables"}}
export OPENAI_API_KEY=sk-tok-YOUR_KEY
aider --openai-api-base https://api.tokios.com/v1 \
      --model openai/mistral-tunnel
```

```json Cline / Continue theme={"theme":{"light":"github-light","dark":"css-variables"}}
{
  "baseURL": "https://api.tokios.com/v1",
  "apiKey": "sk-tok-YOUR_KEY",
  "model": "mistral-tunnel"
}
```

> [!TIP]
>
> The `model` field is always your registered **deployment** name (`mistral-tunnel`), never vLLM's local model id.

## Security model: outbound-only tunnel

Here's what makes this different from SSH tunnels, port forwarding, or Tailscale:

```
Your GPU machine                  api.tokios.com               Remote client
──────────────                    ──────────────               ─────────────
vLLM ─┐                                                        sk-tok-… key
  (localhost:8000)                                               │
       ├─ connector ──► outbound TLS tunnel ──► gateway ──────► HTTPS
       │  (no inbound     (initiates             (auth +       request
       │   port open)      connection)            routing)
       └─ nothing listens
          on any port
```

| Property | Your setup |
| - | - |
| Inbound ports open | 0 |
| Public IP required | No |
| Works behind CGNAT | Yes |
| Auth method | `sk-tok-…` bearer token per request |
| Key scoping | Per deployment, per user |
| Encryption | TLS end-to-end |
| SSE streaming | Native, no buffering |

Your vLLM process never changes. It stays on `127.0.0.1:8000`, unreachable from the outside. The connector is the only thing that can reach it, and the connector only dials out.

> [!NOTE]
>
> You can keep vLLM's `--api-key` flag as defense in depth on the loopback connection between the connector and vLLM. Client requests through Tokios are authenticated with your `sk-tok-…` key — the connector is what's allowed to reach `127.0.0.1:8000` at all.

## FAQ

> [accordion] Does streaming work?
>
> Yes. Set `"stream": true` and Tokios returns standard server-sent events (SSE) in OpenAI or Anthropic format, depending on which API surface you called.

> [accordion] Can I serve more than one model?
>
> Yes. Register additional deployments in the [Models](https://tokios.com/console/models) tab, or use a `tokios-connector.json` config to define multiple upstreams and routes. See [Connector config](https://docs.tokios.com/reference/connector-config.md).

> [accordion] Can I share access with teammates?
>
> Yes. Create a separate `sk-tok-…` key per teammate in the [Keys](https://tokios.com/console/keys) tab. Scope each key to only the deployments they need. Revoking a key cuts access immediately.

> [accordion] What happens if vLLM restarts?
>
> The connector automatically reconnects to vLLM when it comes back up. No manual intervention needed — just restart vLLM and the endpoint recovers.

> [accordion] Does this work on Windows?
>
> Yes. Run vLLM inside WSL2 with Docker (see the WSL2 config above), then install the Tokios connector on the Windows host or inside WSL2 — either works.

## Your vLLM, accessible from anywhere

vLLM keeps running exactly as it does today — on localhost, on your GPU, with no inbound ports. The connector only dials out. Everything past that — authentication, routing, streaming, key management — is one endpoint you call from any machine, teammate, or agent you give a key to.

> [card] [Quickstart](https://docs.tokios.com/quickstart.md)
>
> Pair a connector and register your first model end to end.

> [card] [API keys](https://docs.tokios.com/setup/api-keys.md)
>
> Create, scope, and rotate sk-tok-… keys for teammates and agents.

> [card] [Compatible clients](https://docs.tokios.com/guides/compatible-clients.md)
>
> 39 clients confirmed working — coding agents, chat apps, SDKs.

> [card] [Remote access comparison](https://docs.tokios.com/guides/remote-access-comparison.md)
>
> How Tokios compares to Tailscale, Cloudflare Tunnel, and SSH tunnels.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.