> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# DGX Spark memory tuning and benchmarks

> How unified memory works on DGX Spark, KV cache sizing for different context lengths, batch trade-offs, and tuning parameters for stable inference on 128GB.

DGX Spark's 128GB unified memory pool is both its biggest advantage and its biggest tuning challenge. Unlike discrete GPUs where VRAM is fixed and separate from system RAM, the Grace CPU and Blackwell GPU share one pool — and when you run out, the system becomes unresponsive instead of throwing a clean out-of-memory error. This page explains where the memory goes, how to size the KV cache, and which vLLM parameters to tune for your workload.

## How unified memory works on GB10

DGX Spark uses NVIDIA's GB10 chip, which combines a Grace ARM CPU and a Blackwell GPU on one package. They share 128GB of LPDDR5X memory over a 273 GB/s interconnect.

There is no separate VRAM. When vLLM allocates "GPU memory," it is carving a region out of the same 128GB pool that DGX OS, the CUDA runtime, and your applications use. This means:

* **`gpu_memory_utilization` controls a share of the total pool**, not a dedicated GPU memory budget.
* **System processes compete for the same bytes.** DGX OS, background services, and CUDA context overhead all reduce what the model can use.
* **There is no OOM killer safety net.** When the pool is exhausted, DGX Spark hangs rather than terminating the offending process.

A realistic memory budget looks like this:

| Allocation | Typical size |
| - | - |
| DGX OS and system services | 4-6 GB |
| CUDA runtime and context | 2-3 GB |
| vLLM CUDA graphs | 3-5 GB |
| **Available for model + KV cache** | **\~115 GB** |

## KV cache overhead on 128GB

The KV cache stores the key-value tensors from every token in every active sequence. Its size grows with 3 factors: context length, number of concurrent sequences, and the model's architecture (number of layers and KV heads).

### The formula

For a single sequence at a given context length:

```
KV cache per sequence = 2 × layers × KV_heads × head_dim × context_length × bytes_per_element
```

The factor of 2 accounts for both keys and values.

### Example: 70B dense model at different context lengths

Assuming a typical 70B model with 80 layers, 8 KV heads, head dimension 128:

| Context length | KV cache (FP16) | KV cache (FP8) | Notes |
| - | - | - | - |
| 4K tokens | \~5.2 GB | \~2.6 GB | Comfortable for single user |
| 8K tokens | \~10.5 GB | \~5.2 GB | Recommended limit for multi-user |
| 16K tokens | \~21 GB | \~10.5 GB | Tight — only 1-2 concurrent users |
| 32K tokens | \~42 GB | \~21 GB | Single user only on 128GB |

> [!TIP]
>
> **Use FP8 KV cache to cut memory in half.** Set `--kv-cache-dtype fp8` in your vLLM launch command. The quality impact is less than 2% on most benchmarks, and it effectively doubles the number of concurrent sequences or context length you can support.

### Example: 35B MoE model

MoE models have the same KV cache math as dense models for their total layer count. A 35B MoE model with \~64 layers at the same KV head configuration:

| Context length | KV cache (FP16) | KV cache (FP8) |
| - | - | - |
| 8K tokens | \~4.2 GB | \~2.1 GB |
| 32K tokens | \~16.8 GB | \~8.4 GB |
| 128K tokens | \~67.2 GB | \~33.6 GB |

At 35B MoE with NVFP4 weights (\~18 GB), you can fit 128K context with FP8 KV cache and still have \~65 GB of headroom.

## Context length vs batch size trade-off

Every token of context in every active sequence consumes KV cache memory. The total KV cache budget is:

```
Total KV cache = KV per sequence × max_num_seqs × max_model_len / sequence_length
```

In practice, the trade-off on DGX Spark looks like this:

| Priority | You can have | But you sacrifice |
| - | - | - |
| Long context (32K+) | 1-2 concurrent users | Throughput under load |
| High concurrency (10+ users) | Short context (4K-8K) | Document-length tasks |
| Balanced (5 users, 8K context) | Reasonable for both | Neither extreme |

For a 70B dense model at NVFP4 (\~35 GB weights, \~75 GB remaining for KV cache):

| `max_num_seqs` | `max_model_len` | FP16 KV fits? | FP8 KV fits? |
| :-: | :-: | :-: | :-: |
| 5 | 4096 | Yes (\~26 GB) | Yes (\~13 GB) |
| 5 | 8192 | Tight (\~52 GB) | Yes (\~26 GB) |
| 10 | 4096 | Tight (\~52 GB) | Yes (\~26 GB) |
| 10 | 8192 | No (\~105 GB) | Tight (\~52 GB) |

## Real-world benchmarks vs NVIDIA claims

NVIDIA's published figure of **82,739 tok/s** for DGX Spark refers to **aggregate fine-tuning throughput**, not single-user inference. The real numbers for interactive serving are:

| Model | Quantization | tok/s (single user) | tok/s (5 concurrent) | Notes |
| - | - | :-: | :-: | - |
| Qwen3.5-35B-A3B | NVFP4 (vLLM) | 47-50 | Not measured | Sustained; TTFT 0.12s |
| Qwen3.6-35B-A3B | NVFP4 (Atlas) | \~120 | Not measured | 2.8x stock vLLM; optimized engine |
| Qwen3.5-122B-A10B | NVFP4 (vLLM) | \~30 | Not measured | Breaks under 100K concurrent load |
| 70B dense | NVFP4 (vLLM) | 15-20 | \~5-10 users max | Concurrency-limited by KV cache |
| Qwen3-32B | Q4\_K\_M (Ollama) | 9-44 | Not measured | Wide variance by config |

> [!NOTE]
>
> The 82,739 tok/s figure measures aggregate throughput across many parallel fine-tuning streams. Single-user interactive inference is a different workload — expect 15-120 tok/s depending on model size and engine, which is still fast enough for real-time chat and coding agent use.

## Memory guard: prevent system hangs

The most important tuning parameter on DGX Spark is `--gpu-memory-utilization`. Set it too high and the system hangs without a recoverable error. Set it conservatively and you lose some KV cache headroom.

### Recommended starting values

| Model size | `gpu_memory_utilization` | Rationale |
| - | :-: | - |
| ≤35B (MoE or dense) | 0.90 | Plenty of headroom; weights are small |
| 70B dense | 0.85 | Leave \~15 GB for system and KV cache overflow |
| 120B MoE | 0.85 | Weights alone use \~60 GB; keep reserve |
| Any model + long context | 0.80 | Extra safety margin for large KV caches |

> [!WARNING]
>
> **Never set `gpu_memory_utilization` above 0.90 on DGX Spark.** Unlike discrete GPUs, unified memory exhaustion makes the entire system unresponsive. You will need a hard power cycle to recover. Start conservative and increase only after confirming stability under your actual workload.

### Monitor memory in real time

Open a second terminal and watch memory usage while serving:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
watch -n 1 "nvidia-smi --query-gpu=memory.used,memory.free,memory.total --format=csv"
```

If `memory.free` drops below 2-3 GB, your system is at risk of hanging. Reduce `max_model_len`, `max_num_seqs`, or `gpu_memory_utilization`.

## Tuning parameters reference

These are the vLLM flags that matter most on DGX Spark, in order of impact:

| Parameter | What it controls | DGX Spark guidance |
| - | - | - |
| `--gpu-memory-utilization` | Fraction of memory vLLM can use | 0.80-0.90; never above 0.90 |
| `--max-model-len` | Maximum sequence length | 8192 for 70B+; 32768 for 35B MoE |
| `--max-num-seqs` | Maximum concurrent sequences | 5-10 for 70B; higher for 35B MoE |
| `--kv-cache-dtype` | KV cache precision | `fp8` to halve KV cache memory |
| `--swap-space` | CPU swap for evicted sequences (GB) | 0 on DGX Spark — swapping to shared memory hurts |
| `--quantization` | Weight quantization | `nvfp4` is recommended for SM121 |
| `--enable-prefix-caching` | Reuse KV cache across prompts | Enable for repeated system prompts |

### Decision table: common configurations

| Scenario | `gpu_memory_utilization` | `max_model_len` | `max_num_seqs` | `kv_cache_dtype` |
| - | :-: | :-: | :-: | :-: |
| 35B MoE, single user, long context | 0.90 | 32768 | 1 | `auto` |
| 35B MoE, 5 users, 8K context | 0.90 | 8192 | 5 | `fp8` |
| 70B dense, single user | 0.85 | 8192 | 1 | `fp8` |
| 70B dense, 5 users | 0.85 | 4096 | 5 | `fp8` |
| 120B MoE, single user | 0.85 | 8192 | 1 | `fp8` |

## Tuning recipes

### Maximize throughput for a single user

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve nvidia/Qwen3.5-35B-A3B-NVFP4 \
  --quantization nvfp4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --max-num-seqs 1 \
  --enable-prefix-caching
```

### Maximize concurrent users for 70B

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve <your-70b-model> \
  --quantization nvfp4 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 4096 \
  --max-num-seqs 10 \
  --kv-cache-dtype fp8
```

### Safest configuration for large models

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve <your-120b-moe-model> \
  --quantization nvfp4 \
  --gpu-memory-utilization 0.80 \
  --max-model-len 8192 \
  --max-num-seqs 1 \
  --kv-cache-dtype fp8 \
  --swap-space 0
```

## Next steps

> [card] [vLLM on DGX Spark](https://docs.tokios.com/guides/vllm-dgx-spark.md)
>
> Installation and launch commands for every model size.

> [card] [Model compatibility](https://docs.tokios.com/guides/dgx-spark-model-compatibility.md)
>
> Which models fit in 128GB and their quantization options.

> [card] [One vs two Sparks](https://docs.tokios.com/guides/one-vs-two-dgx-sparks.md)
>
> When memory tuning isn't enough and you need a second Spark.

> [card] [KV cache concepts](https://docs.tokios.com/concepts/kv-cache-and-context.md)
>
> Deeper explanation of KV cache mechanics across model architectures.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.