> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Run vLLM on Windows with WSL2 and NVIDIA GPUs

> Install and run vLLM on a Windows PC with NVIDIA GPUs using WSL2 and Docker — configure GPU passthrough, launch inference, and expose it with Tokios.

vLLM does not run natively on Windows. The supported path is WSL2 with Docker and the NVIDIA Container Toolkit, which gives you full GPU passthrough and access to the entire Linux inference stack without dual-booting. This guide walks through the complete setup on Windows 11 with NVIDIA GPUs.

> [!NOTE]
>
> Last tested: July 2026. A [community PR](https://www.reddit.com/r/LocalLLaMA/comments/1jct1lk/pr_for_native_windows_support_was_just_submitted/) for native Windows support was submitted but has not been merged into a stable release. Until then, WSL2 + Docker remains the recommended path.

## Prerequisites

* Windows 11 (22H2 or later) with WSL2 enabled
* 1 or more NVIDIA GPUs with driver version ≥535
* Docker Desktop for Windows with the WSL2 backend enabled
* At least 16 GB of system RAM (32 GB recommended for models ≥13B)

## Step 1: install WSL2 with GPU support

Open PowerShell as Administrator and install WSL2:

```powershell theme={"theme":{"light":"github-light","dark":"css-variables"}}
wsl --install
```

This installs Ubuntu by default. After reboot, confirm the WSL kernel version:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
uname -r
```

You need kernel ≥5.15 for GPU passthrough. Update if necessary:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
sudo apt update && sudo apt upgrade -y
```

Verify that your GPU is visible inside WSL2:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
nvidia-smi
```

You should see your GPU listed with the correct driver version. If `nvidia-smi` reports no devices, update your Windows NVIDIA driver to ≥535 and restart WSL:

```powershell theme={"theme":{"light":"github-light","dark":"css-variables"}}
wsl --shutdown
```

## Step 2: install Docker with NVIDIA Container Toolkit

1. Install [Docker Desktop](https://www.docker.com/products/docker-desktop/) on Windows.
2. Open Docker Desktop settings and confirm **Use the WSL 2 based engine** is checked.
3. Under **Resources → WSL Integration**, enable your Ubuntu distribution.

Inside WSL2, install the NVIDIA Container Toolkit:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
```

Restart Docker after installation:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
sudo service docker restart
```

## Step 3: launch vLLM with Docker Compose

Create a `docker-compose.yml` file:

```yaml docker-compose.yml theme={"theme":{"light":"github-light","dark":"css-variables"}}
services:
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ports:
      - "8000:8000"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    command: >
      --model Qwen/Qwen2.5-7B-Instruct
      --gpu-memory-utilization 0.9
      --max-model-len 8192
```

Start the container:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
docker compose up -d
```

Watch the logs while the model downloads and loads:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
docker compose logs -f vllm
```

When you see `Uvicorn running on http://0.0.0.0:8000`, the server is ready.

## Step 4: verify it works

Send a test request from inside WSL2:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
```

Expected response: a JSON object with a `choices` array containing the model's reply.

## Multi-GPU configuration (MCDM)

WSL2 supports multi-GPU tensor parallelism through the Microsoft Compute Driver Model (MCDM). This replaces the older TCC mode and requires NVIDIA driver ≥535.

### Switch to MCDM mode

On the Windows host, open an Administrator command prompt:

```powershell theme={"theme":{"light":"github-light","dark":"css-variables"}}
nvidia-smi -g 0 -dm 0
```

Repeat for each GPU (replace `0` with the GPU index), then reboot. After reboot, confirm all GPUs appear in WSL2:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
nvidia-smi
```

### Launch with tensor parallelism

Update your `docker-compose.yml` command to use multiple GPUs:

```yaml theme={"theme":{"light":"github-light","dark":"css-variables"}}
    command: >
      --model Qwen/Qwen2.5-72B-Instruct
      --tensor-parallel-size 2
      --gpu-memory-utilization 0.9
      --max-model-len 4096
```

> [!TIP]
>
> Tensor parallelism splits layers across GPUs and is preferred over pipeline parallelism for single-node multi-GPU setups. It delivers better throughput for the decode phase.

## Connect from the Windows host

WSL2 runs in a lightweight VM with its own network. Docker's port mapping forwards `localhost:8000` from the WSL2 VM to the Windows host automatically, so you can reach vLLM from Windows applications:

```powershell theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\": \"Qwen/Qwen2.5-7B-Instruct\", \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}]}"
```

If `localhost` forwarding doesn't work, find the WSL2 IP address and use it directly:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
hostname -I
```

Then on Windows:

```powershell theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl http://<WSL2_IP>:8000/v1/chat/completions ...
```

> [!WARNING]
>
> WSL2's IP address changes on every restart. If you rely on the IP directly, you need to update your configuration after each reboot. Prefer `localhost` forwarding when possible.

## Expose your endpoint with Tokios

Running vLLM on WSL2 gives you local inference, but it's only reachable from the same machine. Install the Tokios connector inside WSL2 alongside your vLLM container to expose your model through a secure tunnel — accessible from any device, without port forwarding or firewall rules:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Inside WSL2, install and pair the connector
curl -fsSL https://install.tokios.com | bash
tokios-connector pair
```

Once paired, register your vLLM deployment on the [Models](https://tokios.com/console/models) tab and generate an [API key](https://docs.tokios.com/setup/api-keys.md). Your model is now reachable at `https://api.tokios.com/v1` from anywhere.

See [Connector install](https://docs.tokios.com/setup/connector-install.md) for the full walkthrough.

## Troubleshooting

> [accordion] GPU not detected in WSL2
>
> Your NVIDIA driver is too old or WSL2 kernel is outdated. Update the Windows NVIDIA driver to ≥535, then run `wsl --update` from PowerShell and restart WSL with `wsl --shutdown`.

> [accordion] CUDA kernel image unavailable
>
> Specific vLLM versions (v0.22/v0.23) ship CUDA kernels that don't match every WSL2 CUDA toolkit version. Pin your Docker image to a known-good tag, for example `vllm/vllm-openai:v0.21.0`, instead of `latest`.

> [accordion] Out of memory on multi-GPU
>
> vLLM reserves more VRAM for CUDA graphs than Ollama. If a 32B model OOMs on 2x RTX 3090, lower `--gpu-memory-utilization` to `0.85` or reduce `--max-model-len`.

> [accordion] Docker container exits immediately
>
> Check that Docker Desktop's WSL2 integration is enabled for your distribution. Run `docker compose logs vllm` to see the exit reason — it's usually a missing GPU driver or an incorrect model path.

> [accordion] MCDM not enabled
>
> If `nvidia-smi` inside WSL2 only shows 1 GPU on a multi-GPU system, MCDM may not be active. Run `nvidia-smi -g <GPU_ID> -dm 0` on the Windows host for each GPU, then reboot.

## Native Windows support status

A community fork by SystemPanic adds native Windows support to vLLM without WSL2 or Docker. The PR was submitted in early 2025 but has not been merged into a stable release. Track progress on the [vLLM GitHub repository](https://github.com/vllm-project/vllm).

Until native support ships in a stable release, WSL2 + Docker is the production path for Windows.

## Next steps

> [card] [vLLM remote access](https://docs.tokios.com/guides/vllm-remote-access.md)
>
> Expose your vLLM server to any device with Tokios — no port forwarding required.

> [card] [Model fit](https://docs.tokios.com/guides/model-fit.md)
>
> Find which models fit your GPU's VRAM and what throughput to expect.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.