What you’ll get
- Model: DeepSeek V4.1 Flash (763B total, 8B/16B active per token)
- Hardware: 2 NVIDIA DGX Sparks (256 GB aggregate unified memory)
- Throughput: 38-73 tok/s single-stream, up to 270 tok/s at 16 concurrent users
- Context: 1M tokens verified
- Memory usage: ~79-105 GiB per node for weights
- Use case: Large-scale reasoning, vision tasks, long-context analysis, multi-user serving
Prerequisites
- 2 NVIDIA DGX Sparks with DGX OS installed
- ConnectX-7 cluster link between the Sparks (RoCE networking)
- Internet connection for downloading model weights (~400 GiB total)
- Tokios account (free tier works)
- 45 minutes
Step 1: Verify the cluster link
Before downloading the model, verify the RoCE cluster link between the 2 Sparks is working:Step 2: Download the model
The NVFP4 variant stores expert weights at NVFP4 (W4A4), Engram lookup tables at MXFP4 (E2M1), and preserves source precision for attention, shared experts, vision, and MTP/DSpark layers.The model does not fit at full FP8 precision on 256 GB. The NVFP4+MXFP4 mixed-precision layout is required. Community teams have independently verified this configuration works.
Step 3: Serve the model with vLLM (TP=2)
vLLM v0.31.0 or later is required. Start the server with distributed tensor parallelism across 2 nodes:Step 4: Benchmark performance
Single-stream decode
The wide range (38 vs 73 tok/s) reflects different serving engines and quantization strategies. vLLM with FP8+MXFP4 gives ~38-41 tok/s. TensorFold with EXL3 quantization gives ~73 tok/s.
Multi-user aggregate throughput
The model scales well with concurrency because only 8B/16B parameters activate per token. The bottleneck shifts from compute to the inter-Spark interconnect at high concurrency.
Step 5: Connect with Tokios
Your model is running on127.0.0.1:8000 on Spark 1. To give your team remote access, install the Tokios connector on Spark 1.
Install the connector
Claim the connector
In the dashboard, approve the device. Configure the connector to point at vLLM:Register the deployment
On the Models tab:- Upstream model id:
nvidia/DeepSeek-V4.1-Flash-NVFP4 - Deployment name:
deepseek-v41-flash
Create API keys
On the Keys tab, create a key for each teammate. Scope each key to thedeepseek-v41-flash deployment.
Step 6: Use it from any client
Your team can now call the 763B model from anywhere:Memory math
Each DGX Spark has 128 GB of LPDDR5x unified memory. Two Sparks give 256 GB aggregate.
The NVFP4 accuracy retention is strong. NVIDIA published benchmark comparisons showing less than 1% deviation from the MXFP4 source across GPQA Diamond (91.0 → 90.8), SciCode (54.4 → 54.0), and Terminal-Bench 2.1 (81.6 → 81.9).
Troubleshooting
vLLM fails to start with “CUDA out of memory”
Reduce--max-model-len from 131072 to 65536 or 32768. The KV cache grows linearly with context length, and at 1M context the memory pressure is significant.
Slow inter-Spark bandwidth (<150 Gb/s)
Check the RoCE cluster link. Verify ConnectX-7 drivers are up to date, the cable is seated properly, and flow control is enabled. Expected: ~183-185 Gb/s.Ray worker fails to connect
Make sure the Ray port (6379) is accessible from Spark 2 to Spark 1. Check firewall rules on both Sparks. You may need to open the port:Time to first token is very long (>30s at 8K context)
This is expected for a 763B model. The prefill pass over the prompt is compute-intensive. At 8K context, TTFT should be ~5s. At 64K context, it’s ~31s. At 512K context, it’s ~334s. The inter-Spark interconnect is the bottleneck for long prompts.Next steps
- DGX Spark model compatibility — what else fits on 1 or 2 Sparks
- One vs two DGX Sparks — when to scale up
- Team access to local models — give your team API access
- DeepSeek V4 Flash local hardware — comparison across hardware tiers