> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# One vs two DGX Sparks — when to scale and how to set it up

> Decision guide for one vs two NVIDIA DGX Sparks: when one is enough, tensor parallelism over the 200Gb/s link, and vLLM distributed configuration.

One DGX Spark handles models that fit within its `128 GB` unified-memory pool. Two Sparks connected over a 200Gb/s QSFP link can run selected distributed models that do not fit on 1 node. The added network hop also adds coordination overhead, so measure throughput and latency on your exact runtime instead of assuming a 2× gain.

## Quick verdict

| Model size | 1 Spark | 2 Sparks | Notes |
| - | :-: | :-: | - |
| ≤35B MoE | Yes | Overkill | Runs comfortably at 47-120 tok/s |
| 70B dense | Yes (tight) | Better | Single Spark caps at 5-10 concurrent users |
| 120B MoE | Yes | Better | Fits at NVFP4 but memory is tight |
| DeepSeek-V4-Flash-0731 | No | Community-tested | \~162–167 GB weights before KV cache; revalidate 0731 |
| 405B dense at FP8 | No | No | \~405 GB weights before runtime overhead |
| 460B dense-equivalent weight count at 4-bit | No | Tight | \~230 GB raw weights before format, runtime, or KV-cache overhead |
| 700B at 4-bit | No | No | \~350 GB raw weights before overhead |

## When 1 Spark is enough

A single DGX Spark at 128GB is the right choice when:

* **Your model fits at a reasonable quantization.** Dense models up to 70B at NVFP4 (\~35 GB) or MoE models up to 120B at NVFP4 (\~60 GB) load on a single node with room for KV cache.
* **You serve 1-5 concurrent users.** Single-Spark inference handles small team workloads at 35B MoE. For 70B dense, plan for 5-10 users maximum.
* **Latency matters more than raw model size.** A single Spark avoids the inter-node communication overhead that tensor parallelism introduces. For a 35B MoE model, single-node serving delivers 47-50 tok/s — fast enough for real-time coding agents and chat.
* **You want simple operations.** 1 machine, 1 runtime, 1 point of failure. No network configuration, no distributed launch, no NCCL debugging.

See the [DGX Spark model compatibility matrix](https://docs.tokios.com/guides/dgx-spark-model-compatibility.md) for a full table of what fits.

## When you need 2 Sparks

Add a second Spark when any of these apply:

* **Your quantized model exceeds 128 GB but fits comfortably below 256 GB after runtime and KV-cache headroom.** `DeepSeek-V4-Flash-0731` is an example, with a published `162–167 GB` weight floor.
* **You need more KV-cache capacity or concurrency on a model that is already validated for distributed serving.** Tensor parallelism can distribute memory pressure, but it does not guarantee proportional throughput.
* **You are prepared to benchmark the networked runtime.** Communication overhead and the model architecture determine the result.

> [!NOTE]
>
> A 405B dense model at FP8 needs roughly `405 GB` for weights alone. It does not fit in 2 Sparks' `256 GB` combined memory. For a tested 2-Spark DeepSeek route and context guidance, see [DeepSeek V4 Flash local hardware](https://docs.tokios.com/blog/deepseek-v4-flash-local-hardware.md).

## 200 Gbps QSFP link configuration

Two DGX Sparks connect directly via a **200Gb/s QSFP cable** — no switch required. NVIDIA publishes an official "Connect Two Sparks Playbook" for this configuration.

**1. Connect the QSFP cable**

Plug the 200Gb/s QSFP cable into the QSFP port on each DGX Spark. The link is a direct point-to-point connection — it does not go through your network switch.

**2. Verify the link**

On both Sparks, confirm the link is up and running at full speed:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
ip link show
```

Look for the QSFP interface showing `UP` state. Verify the speed:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
ethtool <qsfp-interface> | grep Speed
```

You should see `200000Mb/s` (200 Gb/s).

**3. Configure IP addresses**

Assign static IPs on the QSFP link subnet. On Spark A:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
sudo ip addr add 10.0.0.1/24 dev <qsfp-interface>
```

On Spark B:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
sudo ip addr add 10.0.0.2/24 dev <qsfp-interface>
```

Test connectivity:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
ping 10.0.0.2   # from Spark A
ping 10.0.0.1   # from Spark B
```

**4. Set NCCL environment variables**

NCCL (NVIDIA Collective Communications Library) handles inter-node GPU communication. Set these on both Sparks before launching vLLM:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
export NCCL_SOCKET_IFNAME=<qsfp-interface>
export NCCL_IB_DISABLE=0
export NCCL_NET_GDR_LEVEL=2
```

These variables tell NCCL to use the QSFP link for communication and enable GPU Direct RDMA for the lowest latency.

## Tensor parallelism across 2 Sparks

vLLM supports distributed tensor parallelism, where each Spark holds a shard of the model weights and they communicate over the QSFP link during inference.

### Launch a distributed model

On Spark A (the head node), start vLLM in distributed mode:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve <your-distributed-model> \
  --tensor-parallel-size 2 \
  --quantization fp8 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 8192 \
  --distributed-executor-backend ray
```

vLLM uses Ray to coordinate the two Sparks. Spark A loads half the model shards, Spark B loads the other half, and they synchronize through NCCL over the QSFP link.

> [!NOTE]
>
> **Tensor parallelism is the right parallelism strategy for two Sparks.** Pipeline parallelism (splitting layers sequentially) introduces pipeline bubbles that waste compute on small clusters. Tensor parallelism splits each layer's computation across both GPUs, giving better utilization for 2-node setups.

### Model examples for 2 Sparks

| Model | Quantization | Weight floor | Configuration | What to verify |
| - | - | - | - | - |
| DeepSeek-V4-Flash-0731 | Published fused/GGUF weights | \~162–167 GB | Community-tested route; revalidate 0731 | KV cache, tool calls, and end-to-end throughput |
| 70B dense | NVFP4 | \~35 GB | `--tensor-parallel-size 2` | Compare throughput and concurrency with 1 Spark |
| 120B MoE | NVFP4 | \~60 GB | `--tensor-parallel-size 2` | KV-cache headroom and network overhead |

### Pipeline parallelism (when to consider)

Pipeline parallelism assigns different layers to each Spark. It is simpler to configure but less efficient for 2 nodes because the second Spark sits idle while the first processes each token's early layers.

Use pipeline parallelism only when:

* The model architecture does not support tensor parallelism.
* You are chaining more than 2 Sparks and want to reduce communication overhead per hop.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve <model> \
  --pipeline-parallel-size 2 \
  --tensor-parallel-size 1
```

## Network requirements between Sparks

The QSFP link is purpose-built for this configuration, but the network path matters for performance:

| Requirement | Specification | Why |
| - | - | - |
| Link speed | 200 Gb/s minimum | Tensor parallelism exchanges activation tensors every layer — bandwidth is the bottleneck |
| Latency | Sub-microsecond (direct QSFP) | Each token requires multiple all-reduce operations; latency compounds |
| Topology | Direct point-to-point | Avoid switches between Sparks; each hop adds latency |
| Protocol | RoCE or native QSFP | NCCL uses RDMA when available for zero-copy GPU-to-GPU transfers |

> [!WARNING]
>
> **Do not use a standard Ethernet switch between two Sparks for tensor parallelism.** A 10GbE or even 100GbE switch adds latency that bottlenecks the decode phase. The 200Gb/s direct QSFP link is the supported and tested configuration.

## Cost-benefit analysis

| Factor | 1 Spark | 2 Sparks |
| - | - | - |
| Hardware cost | 1x DGX Spark | 2x DGX Spark + QSFP cable |
| Power | \~240W | \~480W |
| Setup complexity | Low (single host) | Medium (network + distributed vLLM) |
| Max model size | Limited by 128 GB memory | Limited by 256 GB combined memory, runtime overhead, and interconnect |
| Max concurrency (70B) | 5-10 users | 10-20 users |
| Throughput | Measure on your runtime | Measure against a 1-Spark baseline |
| Failure domain | Single point of failure | 2 points; need both for distributed models |

### When the second Spark pays off

* **You need a distributed model with a measured weight and KV-cache plan.** For example, `DeepSeek-V4-Flash-0731` needs 2 nodes and starts as a community-tested configuration.
* **You serve a team on a model whose distributed runtime increases your measured throughput or concurrency enough to justify the second unit.**
* **You run latency-sensitive workloads on 120B models.** The extra KV cache headroom from two Sparks lets you use longer context or serve more users without hitting the memory wall.

### When it doesn't

* **Your model is ≤70B and your team is small.** One Spark at 47-50 tok/s on a 35B MoE model is already faster than most people type. A second Spark adds cost and complexity for no user-visible improvement.
* **You run MoE models that already fit.** GLM-4.5-Air, Qwen3.5-122B, and Nemotron-3-Super-120B all fit on one Spark at NVFP4. Splitting them across two nodes introduces communication overhead that may not recover the parallelism gain.

## Set up a 2-Spark cluster

**1. Connect and verify the QSFP link**

Follow the [200Gb/s link configuration](#200-gbps-qsfp-link-configuration) steps above. Confirm 200Gb/s link speed and bidirectional ping on both Sparks.

**2. Install vLLM on both Sparks**

Both nodes need identical vLLM installations. Follow the [vLLM on DGX Spark](https://docs.tokios.com/guides/vllm-dgx-spark.md) guide on each unit.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
pip install vllm
```

**3. Install Ray on both Sparks**

vLLM uses Ray for distributed coordination:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
pip install ray
```

Start the Ray head node on Spark A:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
ray start --head --port=6379
```

Join Spark B to the Ray cluster:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
ray start --address=10.0.0.1:6379
```

Verify both nodes appear:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
ray status
```

**4. Set NCCL environment variables on both Sparks**

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
export NCCL_SOCKET_IFNAME=<qsfp-interface>
export NCCL_IB_DISABLE=0
export NCCL_NET_GDR_LEVEL=2
```

**5. Launch vLLM in distributed mode on Spark A**

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve <your-model> \
  --tensor-parallel-size 2 \
  --quantization fp8 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 8192 \
  --distributed-executor-backend ray
```

vLLM distributes the model shards across both Sparks automatically.

**6. Verify distributed serving**

Test the endpoint from either Spark:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<your-model>",
    "messages": [{"role": "user", "content": "Hello from two Sparks"}],
    "max_tokens": 64
  }'
```

## Serve both Sparks through Tokios

You can aggregate multiple Sparks behind a single Tokios endpoint. Install the connector on each Spark, register each as a separate deployment, and use [model routing](https://docs.tokios.com/guides/model-routing.md) to direct traffic.

For a 2-Spark tensor-parallel setup where vLLM runs on Spark A and coordinates with Spark B internally, you only need the connector on Spark A — vLLM's `/v1` endpoint already represents the combined cluster.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# On Spark A (the vLLM head node)
curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
```

Register the distributed deployment in the Tokios dashboard, and clients reach both Sparks through `https://api.tokios.com` without any awareness of the underlying topology.

> [card] [vLLM on DGX Spark](https://docs.tokios.com/guides/vllm-dgx-spark.md)
>
> Installation and launch commands for single-node vLLM.

> [card] [Model compatibility](https://docs.tokios.com/guides/dgx-spark-model-compatibility.md)
>
> Check which models fit on 1 vs 2 Sparks.

> [card] [Memory tuning](https://docs.tokios.com/guides/dgx-spark-memory-tuning.md)
>
> KV cache math and tuning parameters for either configuration.

> [card] [DGX Spark use case](https://docs.tokios.com/use-cases/dgx-spark.md)
>
> Full Tokios connector setup for DGX Spark.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.