> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Install and configure vLLM on NVIDIA DGX Spark

> vLLM installation on DGX Spark with NVFP4 quantization, optimal settings for 35B, 70B, and 120B MoE models, and troubleshooting for ARM64 and SM121 issues.

vLLM with NVFP4 quantization is the best-performing general-purpose serving path on DGX Spark. It gives you an OpenAI-compatible `/v1` endpoint, native SM121 support, and the throughput numbers the community reports — 47-50 tok/s on Qwen3.5-35B-A3B, 15-20 tok/s on 70B dense, and \~30 tok/s on 120B MoE. This page walks you through installation, first launch, and the settings that matter.

> [!NOTE]
>
> **Alternative engines.** Some community members report better results with [SGLang](https://docs.tokios.com/guides/sglang-remote-access.md) on specific model configurations, and the Atlas engine claims 2.8x over stock vLLM for Qwen3.6-35B-A3B. This page covers stock vLLM because it has the broadest model support and the most documented configuration surface.

## Prerequisites

Before you start, confirm your DGX Spark meets these requirements:

* **OS:** DGX OS (Ubuntu-based, Linux aarch64)
* **CUDA:** 13.0 or later — CUDA 12 libraries **fail** on SM121
* **Python:** 3.10 or later (for pip install) or Docker with ARM64 support
* **Memory:** 128 GB unified (default on all DGX Spark / GB10 units)

Verify your CUDA version:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
nvcc --version
```

Verify the GPU is visible:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
nvidia-smi
```

You should see the GB10 Blackwell GPU with 128 GB reported.

> [!WARNING]
>
> DGX Spark requires **CUDA 13.0 exclusively**. The SM121 architecture does not support CUDA 12 libraries. If `nvcc --version` shows CUDA 12.x, update your DGX OS packages before proceeding.

## Install vLLM

**1. Install vLLM via pip**

The simplest path on DGX OS is a direct pip install. vLLM ships ARM64 wheels that work on the Grace CPU.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
pip install vllm
```

If you need the nightly build for the latest NVFP4 or SM121 fixes:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
pip install --pre vllm --extra-index-url https://wheels.vllm.ai/nightly
```

**2. Or pull the Docker image**

If you prefer containerized deployment, pull the ARM64-compatible image:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
docker pull vllm/vllm-openai:latest
```

Run with GPU access:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
docker run --gpus all -it --rm -p 8000:8000 vllm/vllm-openai:latest
```

**3. Verify the installation**

Confirm vLLM starts and detects the GPU:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
python -c "import vllm; print(vllm.__version__)"
```

If this prints a version number without CUDA errors, you are ready to serve.

## Launch your first model

Start with a 35B MoE model — it is fast, well-tested, and leaves plenty of memory headroom.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve nvidia/Qwen3.5-35B-A3B-NVFP4 \
  --quantization nvfp4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768
```

This launches an OpenAI-compatible server on `http://127.0.0.1:8000`. Test it:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Qwen3.5-35B-A3B-NVFP4",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 64
  }'
```

You should see a response within a few seconds, with time-to-first-token around 0.12 seconds.

## Optimal settings by model size

The 3 settings that matter most on DGX Spark are `--gpu-memory-utilization`, `--max-model-len`, and `--quantization`. Here are tested starting points:

| Model size | `gpu_memory_utilization` | `max_model_len` | `quantization` | Expected tok/s |
| - | :-: | :-: | :-: | :-: |
| 35B MoE | 0.90 | 32768 | `nvfp4` | 47-50 |
| 70B dense | 0.85 | 8192 | `nvfp4` | 15-20 |
| 120B MoE | 0.85 | 8192 | `nvfp4` | 25-30 |

### 35B MoE (Qwen3.5-35B-A3B, Qwen3.6-35B-A3B)

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve nvidia/Qwen3.5-35B-A3B-NVFP4 \
  --quantization nvfp4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768
```

35B MoE models use only \~18 GB for weights, leaving substantial room for KV cache. You can push `max-model-len` higher if you need longer context.

### 70B dense

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve <your-70b-model> \
  --quantization nvfp4 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 8192
```

70B dense models use \~35 GB for weights at NVFP4. Keep `max-model-len` at 8192 to leave room for KV cache across concurrent requests. Community reports show multi-user concurrency is limited to 5-10 users at this size.

### 120B MoE (Qwen3.5-122B-A10B, Nemotron-3-Super-120B)

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve <your-120b-moe-model> \
  --quantization nvfp4 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 8192
```

120B MoE models use \~60 GB for weights. The 8192 context limit keeps KV cache manageable. Expect \~30 tok/s for single-user inference.

> [!TIP]
>
> For detailed memory tuning — KV cache sizing, batch size trade-offs, and `--kv-cache-dtype fp8` — see [DGX Spark memory tuning](https://docs.tokios.com/guides/dgx-spark-memory-tuning.md).

## NVFP4 configuration recipes

NVFP4 is the recommended quantization for DGX Spark. It runs natively on SM121 and gives a 75% size reduction with low quality impact.

### Use a pre-quantized model

NVIDIA publishes NVFP4 checkpoints on Hugging Face. Use the `nvidia/` prefix:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve nvidia/Qwen3.5-35B-A3B-NVFP4 --quantization nvfp4
```

### Online quantization at load time

If no pre-quantized checkpoint exists, vLLM can quantize on the fly from a BF16 model:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve <model-path> --quantization nvfp4
```

This takes longer to load (the model must be quantized in memory first) but works for any model architecture vLLM supports.

## SGLang alternative

Some community members recommend [SGLang](https://docs.tokios.com/guides/sglang-remote-access.md) over stock vLLM for specific DGX Spark workloads. SGLang runs on the same CUDA 13.0 stack and uses the same model formats.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
python -m sglang.launch_server --model-path <model> --port 8000
```

Consider SGLang if vLLM's default configuration does not meet your throughput requirements for a specific model. The configuration surface is different — check SGLang's own documentation for DGX Spark-specific flags.

## ARM64 and SM121 gotchas

DGX Spark's Grace CPU (ARM64) and Blackwell GPU (SM121) have a few platform-specific issues worth knowing before you hit them.

| Issue | Symptom | Fix |
| - | - | - |
| CUDA 12 libraries | Import errors, kernel launch failures | Use CUDA 13.0 exclusively; update DGX OS |
| NIM containers fail | Container exits or errors on ARM64 | Use vLLM directly instead of NIM containers |
| OOM without clean error | System becomes unresponsive instead of throwing CUDA OOM | Cap `gpu_memory_utilization` at 0.85-0.90; see [memory tuning](https://docs.tokios.com/guides/dgx-spark-memory-tuning.md) |
| Gemma-4 `gpu_memory_utilization` | Parameter ignored during startup profiling, system hangs | Use NVFP4 quantization explicitly; track [vLLM #46307](https://github.com/vllm-project/vllm/issues/46307) |

> [!WARNING]
>
> **DGX Spark becomes unresponsive instead of throwing a CUDA OOM error.** On unified memory systems, when the model plus KV cache exceeds available memory, the system hangs rather than failing gracefully. Always start with conservative `gpu_memory_utilization` values (0.85 for large models) and increase only after confirming stability.

## Benchmark your setup

After launching, verify throughput with a quick benchmark:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
python -m vllm.entrypoints.openai.api_server \
  --model nvidia/Qwen3.5-35B-A3B-NVFP4 \
  --quantization nvfp4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 &

# Wait for the server to start, then:
python -m vllm.benchmarks.benchmark_serving \
  --model nvidia/Qwen3.5-35B-A3B-NVFP4 \
  --num-prompts 10 \
  --request-rate 1
```

Compare your results against the expected tok/s in the settings table above. If throughput is significantly lower, check that CUDA 13.0 is active and that no other processes are consuming GPU memory.

Monitor memory usage while serving:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
watch -n 1 nvidia-smi
```

## Expose your endpoint with Tokios

vLLM on the Spark gives you a local `/v1` endpoint. To reach it from your laptop, coding agents, or teammates without SSH tunnels or open ports, run the Tokios connector alongside vLLM.

**1. Start vLLM on the Spark**

Launch your model as shown above. Confirm it answers on `http://127.0.0.1:8000/v1`.

**2. Install and pair the connector**

In the [Tokios dashboard](https://tokios.com/console), open **Setup** and select **Start pairing** to get a claim code. On the Spark:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
```

**3. Register a deployment**

On the **Models** tab, map your vLLM model id to a deployment name. Clients use the deployment name, not the upstream model id.

See the full [DGX Spark use-case guide](https://docs.tokios.com/use-cases/dgx-spark.md) for client configuration examples.

> [card] [Model compatibility](https://docs.tokios.com/guides/dgx-spark-model-compatibility.md)
>
> Check which models fit in 128GB and their expected throughput.

> [card] [Memory tuning](https://docs.tokios.com/guides/dgx-spark-memory-tuning.md)
>
> KV cache sizing, FP8 KV cache, and tuning parameters for production workloads.

> [card] [vLLM remote access](https://docs.tokios.com/guides/vllm-remote-access.md)
>
> General vLLM remote access patterns beyond DGX Spark.

> [card] [One vs two Sparks](https://docs.tokios.com/guides/one-vs-two-dgx-sparks.md)
>
> Scale to 405B models with a second Spark and tensor parallelism.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.