> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# vLLM on Strix Halo: status, failures, and alternatives

> vLLM does not work reliably on AMD Strix Halo (Ryzen AI Max+ 395). Here is why, and what to use instead — llama.cpp with Vulkan or pinned Ollama.

> [!WARNING]
>
> **Not recommended hardware.** Tokios does not recommend AMD Strix Halo (Ryzen AI Max+ 395) as a primary inference platform. The hardware is community-explored, not production-ready, and driver/ROCm support remains inconsistent. For reliable inference, use a [Mac Studio](https://docs.tokios.com/use-cases/mac-studio.md), [DGX Spark](https://docs.tokios.com/use-cases/dgx-spark.md), or [RTX workstation](https://docs.tokios.com/use-cases/rtx-workstation.md) instead. This page is for users who already own Strix Halo hardware and want to understand what works.

## Quick verdict: vLLM does not work reliably on Strix Halo

vLLM combined with PyTorch Flash Attention causes GPU hangs on the Radeon 8060S (gfx1151). The TheRock PyTorch builds that target this GPU are incompatible with vLLM's CUDA-origin code paths, and the eager-mode workaround breaks tool calling. Community reports consistently confirm this across multiple kernel and ROCm versions.

Use one of these alternatives instead:

| Alternative | Backend | Status | Best for |
| - | - | - | - |
| **llama.cpp** | Vulkan | Works | Best performance, most control |
| **Ollama 0.17.7** | Vulkan/ROCm | Works (pinned) | Easiest setup |
| **vLLM eager mode** | ROCm | Partial — no tool calls | Not recommended |

## Why vLLM fails

Strix Halo's iGPU is gfx1151 (RDNA 3.5). vLLM was designed for NVIDIA CUDA GPUs and later gained ROCm support for datacenter AMD GPUs (MI250, MI300X). The consumer iGPU in Strix Halo sits outside that supported matrix.

The failure chain:

1. **PyTorch TheRock builds** — AMD ships TheRock builds for gfx1151, but these builds interact poorly with vLLM's Flash Attention implementation.
2. **Flash Attention + vLLM = GPU hang** — when vLLM enables Flash Attention on gfx1151, the GPU hangs and the system becomes unresponsive, requiring a hard power cycle.
3. **Eager mode workaround** — disabling Flash Attention avoids the hang, but throughput drops significantly and tool calling breaks entirely.
4. **Tool calls broken on AMD** — even in eager mode, vLLM's tool calling parser fails on AMD ROCm, which rules out coding agent use.

## Recommended alternative: llama.cpp with Vulkan

`llama.cpp` compiled with the Vulkan backend is the most reliable and fastest option on Strix Halo. Vulkan delivers **21% faster token generation** than ROCm on this hardware and works on both Linux and Windows.

### Prerequisites

* A Strix Halo system with 128GB unified memory
* Vulkan drivers installed (included in Mesa on Linux, Adrenalin on Windows)
* `cmake`, `gcc`/`clang`, and `git`

### Build llama.cpp with Vulkan

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -j$(nproc)
```

On Windows, the same commands work in PowerShell (with Visual Studio or MinGW toolchain).

### Launch the server

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
./build/bin/llama-server \
  -m /path/to/model-Q4_K_M.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  -ngl 999
```

The `-ngl 999` flag offloads all layers to the GPU. Verify the server responds:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl http://127.0.0.1:8080/v1/models
```

> [!TIP]
>
> On Windows, set the **AMD Adrenalin VRAM slider to 96GB** before launching. Without this, the iGPU only sees \~15.5GB and large models will fail to load. See [Strix Halo model compatibility](https://docs.tokios.com/guides/strix-halo-model-compatibility.md) for the full memory allocation guide.

## Alternative: Ollama (pin to 0.17.7)

Ollama is the easiest path if you want `ollama run` simplicity. **You must pin version 0.17.7** — all 0.18.x versions have a GPU detection regression on Strix Halo that causes Ollama to fall back to CPU-only inference.

### Install and pin Ollama 0.17.7

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Download the specific release
curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.17.7 sh

# Verify the version
ollama --version
# Expected: ollama version is 0.17.7

# Prevent auto-updates
sudo systemctl disable ollama-update.timer 2>/dev/null || true
```

### Run a model

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
ollama run qwen3-coder:30b
```

> [!WARNING]
>
> Do **not** run `ollama update` or let a package manager upgrade Ollama past 0.17.7. The 0.18.x GPU detection regression is unresolved. If you accidentally upgrade, reinstall 0.17.7 from the [GitHub releases page](https://github.com/ollama/ollama/releases/tag/v0.17.7).

## Workaround: vLLM eager mode (partial, not recommended)

If you have a specific reason to need vLLM's API shape exactly, you can try eager mode. Expect degraded throughput and broken tool calling.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
export HSA_OVERRIDE_GFX_VERSION=11.5.1
export VLLM_ATTENTION_BACKEND=FLASH_ATTN
export VLLM_USE_FLASH_ATTN_TRITON=0

vllm serve your-model \
  --dtype float16 \
  --enforce-eager \
  --max-model-len 4096 \
  --trust-remote-code
```

Known limitations:

* **No Flash Attention** — throughput is significantly lower than working setups
* **Tool calls broken** — the structured output parser fails on AMD
* **Context length limited** — memory pressure builds fast without paged attention
* **GPU hangs still possible** — long contexts can still trigger hangs

## Backend comparison summary

| Factor | llama.cpp + Vulkan | Ollama 0.17.7 | vLLM eager |
| - | - | - | - |
| Token generation speed | Fastest (21% over ROCm) | Good | Slowest |
| Setup difficulty | Build from source | 1 command + pin | Complex env vars |
| Tool calling | Works | Works | Broken |
| Streaming | Works | Works | Works |
| OpenAI-compatible API | `/v1` on port 8080 | `/v1` on port 11434 | `/v1` on port 8000 |
| Windows support | Yes (Vulkan) | Yes (Vulkan) | No (Linux only) |
| Stability | Good | Good (pinned) | Poor |

## Expose your model via Tokios

Once you have a working runtime (`llama.cpp` or Ollama), run the Tokios connector next to it. The connector dials out to `https://api.tokios.com` — no inbound ports on your box — and gives you a single authenticated endpoint.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Start llama.cpp
./build/bin/llama-server -m model.gguf --port 8080

# Install and pair the Tokios connector
curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
```

See the [Strix Halo use-case page](https://docs.tokios.com/use-cases/strix-halo.md) for the full connector pairing walkthrough.

## If you are buying hardware today

Strix Halo's 128GB unified memory sounds compelling on paper, but the software ecosystem is fragile. For reliable local LLM inference:

| Platform | Memory | Bandwidth | Ecosystem |
| - | - | - | - |
| **[DGX Spark](https://docs.tokios.com/use-cases/dgx-spark.md)** | 128GB unified | 273 GB/s | NVIDIA CUDA — mature, well-supported |
| **[Mac Studio M4 Ultra](https://docs.tokios.com/use-cases/mac-studio.md)** | 192GB unified | 819 GB/s | Metal — excellent, polished |
| **[RTX 4090 workstation](https://docs.tokios.com/use-cases/rtx-workstation.md)** | 24GB VRAM | 1008 GB/s | CUDA — fastest per-token, limited memory |

Any of these will give you a smoother experience than Strix Halo for the same or less effort.

## Next steps

> [card] [Strix Halo model compatibility](https://docs.tokios.com/guides/strix-halo-model-compatibility.md)
>
> Which models fit in 128GB and at what quantization.

> [card] [Strix Halo use case](https://docs.tokios.com/use-cases/strix-halo.md)
>
> Full connector pairing walkthrough for Strix Halo.

> [card] [DGX Spark use case](https://docs.tokios.com/use-cases/dgx-spark.md)
>
> The recommended alternative with 128GB unified memory.

> [card] [vLLM remote access](https://docs.tokios.com/guides/vllm-remote-access.md)
>
> Expose vLLM on supported NVIDIA hardware securely.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.