> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Run DeepSeek V4.1 Flash on 2 DGX Sparks

> Step-by-step guide to running DeepSeek V4.1 Flash (763B MoE) on 2 NVIDIA DGX Sparks with tensor parallelism. Community benchmarks show 38-73 tok/s single-stream, up to 270 tok/s aggregate.

DeepSeek V4.1 Flash is a 763B-parameter mixture-of-experts model with native vision, 1M-token context, and MIT licensing. On 2 NVIDIA DGX Sparks connected over a ConnectX-7 cluster link, you get 38-73 tok/s single-stream decode — or up to 270 tok/s aggregate at 16 concurrent users.

The model does not fit at full precision on 256 GB of aggregate unified memory. It fits when you use mixed-precision serving: FP8 for dense layers, MXFP4 for expert weights. This guide walks you through the setup.

Last verified: 2026-10-24

## What you'll get

* **Model:** DeepSeek V4.1 Flash (763B total, 8B/16B active per token)
* **Hardware:** 2 NVIDIA DGX Sparks (256 GB aggregate unified memory)
* **Throughput:** 38-73 tok/s single-stream, up to 270 tok/s at 16 concurrent users
* **Context:** 1M tokens verified
* **Memory usage:** \~79-105 GiB per node for weights
* **Use case:** Large-scale reasoning, vision tasks, long-context analysis, multi-user serving

## Prerequisites

* 2 NVIDIA DGX Sparks with DGX OS installed
* ConnectX-7 cluster link between the Sparks (RoCE networking)
* Internet connection for downloading model weights (\~400 GiB total)
* Tokios account (free tier works)
* 45 minutes

## Step 1: Verify the cluster link

Before downloading the model, verify the RoCE cluster link between the 2 Sparks is working:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# On Spark 1
ibstat

# Expected: Port active, LinkUp, rate 200 Gb/s
```

Test RDMA bandwidth:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# On Spark 1 (server)
ib_write_bw -d mlx5_0

# On Spark 2 (client)
ib_write_bw -d mlx5_0 &lt;spark1-ip&gt;
```

Expected: \~183-185 Gb/s effective throughput. If you see significantly lower numbers, troubleshoot the cluster link before proceeding.

## Step 2: Download the model

The NVFP4 variant stores expert weights at NVFP4 (W4A4), Engram lookup tables at MXFP4 (E2M1), and preserves source precision for attention, shared experts, vision, and MTP/DSpark layers.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# On both Sparks
huggingface-cli download nvidia/DeepSeek-V4.1-Flash-NVFP4
```

The download is \~400 GiB total. On DGX Spark's 273 GB/s memory bandwidth, this takes about 25 minutes.

> [!NOTE]
>
> The model does not fit at full FP8 precision on 256 GB. The NVFP4+MXFP4 mixed-precision layout is required. Community teams have independently verified this configuration works.

## Step 3: Serve the model with vLLM (TP=2)

vLLM v0.31.0 or later is required. Start the server with distributed tensor parallelism across 2 nodes:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# On Spark 1 (head node)
vllm serve nvidia/DeepSeek-V4.1-Flash-NVFP4 \
  --tensor-parallel-size 2 \
  --kv-cache-dtype fp8_ds_mla \
  --host 127.0.0.1 \
  --port 8000 \
  --max-model-len 131072 \
  --distributed-executor-backend ray \
  --ray-address auto
```

On Spark 2, start the Ray worker:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# On Spark 2
ray start --address='&lt;spark1-ip&gt;:6379'
```

The server starts in \~2-3 minutes. You'll see:

```
INFO:     Started server process [12345]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000
```

Test the endpoint:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/DeepSeek-V4.1-Flash-NVFP4",
    "messages": [{"role": "user", "content": "Explain quantum computing in 3 sentences"}],
    "max_tokens": 200
  }'
```

## Step 4: Benchmark performance

### Single-stream decode

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
python -m vllm.entrypoints.benchmark_serving \
  --backend vllm \
  --model nvidia/DeepSeek-V4.1-Flash-NVFP4 \
  --dataset-name sonnet \
  --dataset-path sonnet.txt \
  --num-prompts 10 \
  --request-rate 1
```

Expected results:

| Metric | Value |
| - | - |
| Throughput (1 user) | 38-73 tok/s |
| Time to first token (64K context) | \~31s |
| Time to first token (8K context) | \~5s |
| Memory usage per node | \~79-105 GiB weights + \~15 GiB KV cache |

The wide range (38 vs 73 tok/s) reflects different serving engines and quantization strategies. vLLM with FP8+MXFP4 gives \~38-41 tok/s. TensorFold with EXL3 quantization gives \~73 tok/s.

### Multi-user aggregate throughput

| Concurrency | tok/s (aggregate) |
| - | - |
| 2 users | 104-110 |
| 4 users | 135-169 |
| 6 users | 219-256 |
| 16 users | 135-270 |

The model scales well with concurrency because only 8B/16B parameters activate per token. The bottleneck shifts from compute to the inter-Spark interconnect at high concurrency.

## Step 5: Connect with Tokios

Your model is running on `127.0.0.1:8000` on Spark 1. To give your team remote access, install the Tokios connector on Spark 1.

### Install the connector

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
```

### Claim the connector

In the [dashboard](https://tokios.com/console), **approve** the device. Configure the connector to point at vLLM:

```json theme={"theme":{"light":"github-light","dark":"css-variables"}}
{
  "upstream": "http://127.0.0.1:8000/v1"
}
```

### Register the deployment

On the **Models** tab:

* **Upstream model id:** `nvidia/DeepSeek-V4.1-Flash-NVFP4`
* **Deployment name:** `deepseek-v41-flash`

### Create API keys

On the **Keys** tab, create a key for each teammate. Scope each key to the `deepseek-v41-flash` deployment.

## Step 6: Use it from any client

Your team can now call the 763B model from anywhere:

```python OpenAI SDK theme={"theme":{"light":"github-light","dark":"css-variables"}}
from openai import OpenAI

client = OpenAI(
    base_url="https://api.tokios.com/v1",
    api_key="sk-tok-YOUR_KEY",
)

response = client.chat.completions.create(
    model="deepseek-v41-flash",
    messages=[{"role": "user", "content": "Explain quantum computing"}],
    max_tokens=500,
)
print(response.choices[0].message.content)
```

```bash curl theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v41-flash",
    "messages": [{"role": "user", "content": "Explain quantum computing"}],
    "max_tokens": 500
  }'
```

## Memory math

Each DGX Spark has 128 GB of LPDDR5x unified memory. Two Sparks give 256 GB aggregate.

| Component | Per node (2-node TP=2) |
| - | - |
| Model weights | \~79-105 GiB |
| KV cache | \~15 GiB |
| System and runtime overhead | \~8-12 GB |
| Available headroom | \~10-40 GB |

The NVFP4 accuracy retention is strong. NVIDIA published benchmark comparisons showing less than 1% deviation from the MXFP4 source across GPQA Diamond (91.0 → 90.8), SciCode (54.4 → 54.0), and Terminal-Bench 2.1 (81.6 → 81.9).

## Troubleshooting

### vLLM fails to start with "CUDA out of memory"

Reduce `--max-model-len` from 131072 to 65536 or 32768. The KV cache grows linearly with context length, and at 1M context the memory pressure is significant.

### Slow inter-Spark bandwidth (\<150 Gb/s)

Check the RoCE cluster link. Verify ConnectX-7 drivers are up to date, the cable is seated properly, and flow control is enabled. Expected: \~183-185 Gb/s.

### Ray worker fails to connect

Make sure the Ray port (6379) is accessible from Spark 2 to Spark 1. Check firewall rules on both Sparks. You may need to open the port:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
sudo ufw allow 6379/tcp
```

### Time to first token is very long (>30s at 8K context)

This is expected for a 763B model. The prefill pass over the prompt is compute-intensive. At 8K context, TTFT should be \~5s. At 64K context, it's \~31s. At 512K context, it's \~334s. The inter-Spark interconnect is the bottleneck for long prompts.

## Next steps

* [DGX Spark model compatibility](https://docs.tokios.com/guides/dgx-spark-model-compatibility.md) — what else fits on 1 or 2 Sparks
* [One vs two DGX Sparks](https://docs.tokios.com/guides/one-vs-two-dgx-sparks.md) — when to scale up
* [Team access to local models](https://docs.tokios.com/use-cases/team-access.md) — give your team API access
* [DeepSeek V4 Flash local hardware](https://docs.tokios.com/blog/deepseek-v4-flash-local-hardware.md) — comparison across hardware tiers


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.