Skip to main content
DeepSeek V4.1 Flash is a 763B-parameter mixture-of-experts model with native vision, 1M-token context, and MIT licensing. On 2 NVIDIA DGX Sparks connected over a ConnectX-7 cluster link, you get 38-73 tok/s single-stream decode — or up to 270 tok/s aggregate at 16 concurrent users. The model does not fit at full precision on 256 GB of aggregate unified memory. It fits when you use mixed-precision serving: FP8 for dense layers, MXFP4 for expert weights. This guide walks you through the setup. Last verified: 2026-10-24

What you’ll get

  • Model: DeepSeek V4.1 Flash (763B total, 8B/16B active per token)
  • Hardware: 2 NVIDIA DGX Sparks (256 GB aggregate unified memory)
  • Throughput: 38-73 tok/s single-stream, up to 270 tok/s at 16 concurrent users
  • Context: 1M tokens verified
  • Memory usage: ~79-105 GiB per node for weights
  • Use case: Large-scale reasoning, vision tasks, long-context analysis, multi-user serving

Prerequisites

  • 2 NVIDIA DGX Sparks with DGX OS installed
  • ConnectX-7 cluster link between the Sparks (RoCE networking)
  • Internet connection for downloading model weights (~400 GiB total)
  • Tokios account (free tier works)
  • 45 minutes
Before downloading the model, verify the RoCE cluster link between the 2 Sparks is working:
Test RDMA bandwidth:
Expected: ~183-185 Gb/s effective throughput. If you see significantly lower numbers, troubleshoot the cluster link before proceeding.

Step 2: Download the model

The NVFP4 variant stores expert weights at NVFP4 (W4A4), Engram lookup tables at MXFP4 (E2M1), and preserves source precision for attention, shared experts, vision, and MTP/DSpark layers.
The download is ~400 GiB total. On DGX Spark’s 273 GB/s memory bandwidth, this takes about 25 minutes.
The model does not fit at full FP8 precision on 256 GB. The NVFP4+MXFP4 mixed-precision layout is required. Community teams have independently verified this configuration works.

Step 3: Serve the model with vLLM (TP=2)

vLLM v0.31.0 or later is required. Start the server with distributed tensor parallelism across 2 nodes:
On Spark 2, start the Ray worker:
The server starts in ~2-3 minutes. You’ll see:
Test the endpoint:

Step 4: Benchmark performance

Single-stream decode

Expected results: The wide range (38 vs 73 tok/s) reflects different serving engines and quantization strategies. vLLM with FP8+MXFP4 gives ~38-41 tok/s. TensorFold with EXL3 quantization gives ~73 tok/s.

Multi-user aggregate throughput

The model scales well with concurrency because only 8B/16B parameters activate per token. The bottleneck shifts from compute to the inter-Spark interconnect at high concurrency.

Step 5: Connect with Tokios

Your model is running on 127.0.0.1:8000 on Spark 1. To give your team remote access, install the Tokios connector on Spark 1.

Install the connector

Claim the connector

In the dashboard, approve the device. Configure the connector to point at vLLM:

Register the deployment

On the Models tab:
  • Upstream model id: nvidia/DeepSeek-V4.1-Flash-NVFP4
  • Deployment name: deepseek-v41-flash

Create API keys

On the Keys tab, create a key for each teammate. Scope each key to the deepseek-v41-flash deployment.

Step 6: Use it from any client

Your team can now call the 763B model from anywhere:

Memory math

Each DGX Spark has 128 GB of LPDDR5x unified memory. Two Sparks give 256 GB aggregate. The NVFP4 accuracy retention is strong. NVIDIA published benchmark comparisons showing less than 1% deviation from the MXFP4 source across GPQA Diamond (91.0 → 90.8), SciCode (54.4 → 54.0), and Terminal-Bench 2.1 (81.6 → 81.9).

Troubleshooting

vLLM fails to start with “CUDA out of memory”

Reduce --max-model-len from 131072 to 65536 or 32768. The KV cache grows linearly with context length, and at 1M context the memory pressure is significant.

Slow inter-Spark bandwidth (<150 Gb/s)

Check the RoCE cluster link. Verify ConnectX-7 drivers are up to date, the cable is seated properly, and flow control is enabled. Expected: ~183-185 Gb/s.

Ray worker fails to connect

Make sure the Ray port (6379) is accessible from Spark 2 to Spark 1. Check firewall rules on both Sparks. You may need to open the port:

Time to first token is very long (>30s at 8K context)

This is expected for a 763B model. The prefill pass over the prompt is compute-intensive. At 8K context, TTFT should be ~5s. At 64K context, it’s ~31s. At 512K context, it’s ~334s. The inter-Spark interconnect is the bottleneck for long prompts.

Next steps