> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# How model sharding works

> Understand tensor parallelism, pipeline parallelism, data parallelism, and expert parallelism. Know which sharding strategy fits your hardware.

A 70B model at FP16 needs \~140 GB of VRAM. No single consumer GPU holds that. **Sharding** is how inference engines split a model across multiple GPUs — or multiple machines — so you can serve models that physically cannot fit on 1 device, or serve more users than 1 device handles.

This page explains the 4 sharding strategies, how each one splits the work, and which one fits your hardware. For a quick TP-vs-PP decision, see [Tensor vs pipeline parallelism](https://docs.tokios.com/guides/tensor-vs-pipeline-parallelism.md).

## Why shard?

You reach for sharding when 1 of 4 things is true:

1. **The model is too large for 1 GPU.** A 70B dense model at FP16 needs \~140 GB. Even at Q4 quantization (\~35 GB), it fills a single RTX 4090 (24 GB). You need at least 2 GPUs to hold the weights.
2. **You need lower latency per token.** When every GPU contributes to each token simultaneously, the wall-clock time per token drops.
3. **You need higher throughput.** More GPUs means more concurrent users served in parallel.
4. **You need redundancy.** Running 2 replicas behind a load balancer means 1 GPU can fail without downtime.

If the model already fits on 1 GPU and you have no throughput problem, stay on 1 GPU. Sharding adds complexity and communication overhead.

## Where your GPU memory actually goes

Before you pick a sharding strategy, know what competes for VRAM. It is not just model weights:

| Component | What it is | Typical size (70B dense, Q4) |
| - | - | - |
| **Model weights** | The quantized parameter tensors | \~35 GB |
| **KV cache** | Per-token key/value state for active sequences | 2–20 GB (scales with context length and batch size) |
| **Activations + CUDA graphs** | Intermediate tensors and pre-compiled kernel graphs | 2–5 GB |
| **OS + driver reserve** | CUDA runtime, display driver, system | \~1–2 GB |

A 24 GB RTX 4090 running a 70B Q4 model has \~35 GB of weights — it does not fit on 1 GPU at all. With TP=2, each GPU holds \~17.5 GB of weights and has \~6.5 GB left for KV cache and overhead. That is enough for short contexts and a few concurrent users, but not much more.

> [!NOTE]
>
> **Usable memory is less than total VRAM.** On a DGX Spark (128 GB unified), expect \~110 GB usable for model + KV after the OS and runtime reserve their share. On discrete GPUs, vLLM's `--gpu-memory-utilization` (default 0.90) controls this. Set it lower if you hit OOM.

For MoE models, the breakdown looks different — the expert planes dominate:

| Component | Qwen3-235B (2-bit experts + NVFP4 dense) |
| - | - |
| Dense weights (attention, embeddings) | \~14 GB at 4-bit |
| Expert planes (256 routed experts) | \~231 GB at 2-bit |
| KV cache + activations | Variable |
| **Total on 1 node** | \~262 GB — far exceeds any single GPU |

This is why expert parallelism exists: instead of loading all 231 GB of experts onto every GPU, each GPU holds only its assigned experts. See [Expert parallelism](#expert-parallelism-for-mixture-of-experts-models) below.

## 2 decisions: make it fit, then make it fast

Sharding answers 2 questions in order:

**Make it fit** — get the weights (and headroom for KV cache) inside your GPU memory budget:

* **Quantize first.** Going from FP16 to Q4 cuts weight memory by \~4×. This is the cheapest win. See the [quantization guide](https://docs.tokios.com/concepts/quantization.md).
* **Shard across GPUs.** When quantization alone is not enough, split the model with TP, PP, or EP.
* **Prune experts (MoE only).** For MoE models with 256 experts where only 6 fire per token, REAP-style expert pruning cuts routed-expert weights roughly in half without touching active compute.

**Make it fast** — once it fits, reduce bytes per token and decode steps:

* **Pick the right parallelism.** TP gives lowest per-token latency. PP tolerates slow interconnects. DP multiplies throughput.
* **Use the model's documented speculative path.** For `DeepSeek-V4-Flash-0731`, enable the bundled DSpark module with the official vLLM or SGLang flags. Do not configure a separate draft checkpoint. See [model serving and scaling](https://docs.tokios.com/concepts/model-serving-and-scaling.md).
* **Tune KV cache and batch size.** More KV cache headroom means more concurrent users. See [DGX Spark memory tuning](https://docs.tokios.com/guides/dgx-spark-memory-tuning.md).

## The 4 sharding strategies at a glance

> [!NOTE]
>
> `DeepSeek-V4-Flash-0731` is a practical sharding case: 13B active parameters lower per-token compute, but all 284B parameters still need resident weight memory. See [DeepSeek V4 Flash local hardware](https://docs.tokios.com/blog/deepseek-v4-flash-local-hardware.md) for the memory floor and the maturity of each 2-device route.

| Strategy | What gets split | Communication pattern | Bandwidth needed |
| - | - | - | - |
| Tensor parallelism (TP) | Each layer's weight matrices | AllReduce every layer | Very high |
| Pipeline parallelism (PP) | Layers into sequential stages | Point-to-point between stages | Low |
| Data parallelism (DP) | Nothing — full copies | None during inference | None |
| Expert parallelism (EP) | MoE experts across GPUs | All-to-all for expert routing | Medium–high |

## Tensor parallelism: splitting layers across GPUs

Tensor parallelism (TP) slices every layer's weight matrices across GPUs. Each GPU holds a fraction of every layer, and all GPUs work on the same token simultaneously.

```
Layer N:
  GPU 0: weights[0:50%] × input  ──┐
                                    ├── AllReduce ──→ output
  GPU 1: weights[50:100%] × input ─┘
```

After every layer, the GPUs run an **AllReduce** operation — they combine their partial results into the full output. This keeps per-token latency low because every GPU contributes, but it demands high bandwidth between GPUs.

**Memory:** each GPU holds `1/N` of the model weights, where N is the TP degree. A 70B model at Q4 (\~35 GB) with TP=2 puts \~17.5 GB of weights per GPU.

**Latency:** lowest per token. All GPUs compute on every token, so the wall-clock time is the compute time plus the AllReduce synchronization.

**Best for:** 2–4 GPUs connected by NVLink or a fast PCIe bus. The 200Gb/s QSFP link between 2 DGX Sparks is ideal.

**vLLM flag:** `--tensor-parallel-size 2`

> [!NOTE]
>
> PCIe delivers 3–5× less bandwidth than NVLink for AllReduce operations. Tensor parallelism over PCIe still works — 2× RTX 4090 at TP=2 is a common setup — but expect 20–30% lower decode throughput than the same model on NVLink-connected GPUs.

### Example: 70B on 2× RTX 4090

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --quantization awq \
  --gpu-memory-utilization 0.90
```

Each GPU holds \~17.5 GB of Q4 weights. The remaining \~6.5 GB per GPU covers KV cache and CUDA graphs.

## Pipeline parallelism: splitting layers sequentially

Pipeline parallelism (PP) assigns contiguous groups of layers to different GPUs. GPU 0 holds layers 0–19, GPU 1 holds layers 20–39. Tokens flow through the pipeline like an assembly line.

```
GPU 0: [Layers 0–19]  ──activations──→  GPU 1: [Layers 20–39]  ──→ output
```

Each GPU processes its assigned layers and passes the activations to the next stage. The data that crosses GPUs is the activation tensor for a single layer boundary — much smaller than the full weight matrices that TP exchanges.

**Memory:** each GPU holds its portion of layers. With PP=2, GPU 0 holds the first half of the model, GPU 1 holds the second half.

**Latency:** additive. A token must pass through every stage sequentially. With PP=2, the latency is roughly the sum of both stages.

**Communication:** point-to-point between adjacent stages. Only neighboring GPUs talk, so the bandwidth requirement is low. This makes PP suitable for slower interconnects — PCIe, Ethernet, or network links between machines.

**The bubble problem:** when the batch size is small, later GPUs sit idle while earlier GPUs process their layers. With PP=2 and batch size 1, GPU 1 waits for GPU 0 to finish every single layer boundary. Larger batches amortize this — vLLM's continuous batching helps fill the pipeline.

**vLLM flag:** `--pipeline-parallel-size 2`

### When PP beats TP

Use pipeline parallelism over tensor parallelism when:

* GPUs are in **different machines** connected by a network (10GbE–100GbE).
* GPUs are connected by **PCIe** and the model is small enough that PP's bubble waste costs less than TP's AllReduce bandwidth.
* You chain **3+ nodes** where TP across all of them would saturate the interconnect.

## Data parallelism: running the same model on multiple GPUs

Data parallelism (DP) runs a full copy of the model on each GPU and distributes incoming requests across them. There is no inter-GPU communication during inference — each GPU independently handles its own batch of requests.

```
Request A ──→ GPU 0 (full model copy)
Request B ──→ GPU 1 (full model copy)
Request C ──→ GPU 2 (full model copy)
```

**Memory:** N copies of the full model. A 7B model at Q4 (\~4 GB) with DP=4 uses 16 GB total across 4 GPUs. This is the expensive part — you trade memory for throughput.

**Latency:** identical to single-GPU inference. Each GPU processes tokens independently.

**Throughput:** scales linearly. 4 replicas serve \~4× the concurrent users.

**Best for:** models that fit on 1 GPU (7B, 14B, sometimes 32B at aggressive quantization) when you need to serve many users simultaneously.

**vLLM setup:** launch multiple vLLM instances and put a load balancer in front.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Instance 1 on GPU 0
CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-14B --port 8000

# Instance 2 on GPU 1
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-14B --port 8001
```

Then use nginx, HAProxy, or any round-robin balancer to distribute `/v1/chat/completions` requests across both ports.

## Expert parallelism: for mixture-of-experts models

Expert parallelism (EP) applies only to [mixture-of-experts (MoE) models](https://docs.tokios.com/concepts/read-a-model-card.md). In an MoE model, each token activates only a small subset of the model's "experts" — for example, Qwen3-235B has 256 routed experts but only 6 fire per token (\~22B active parameters out of 235B total). EP puts different experts on different GPUs so you do not waste memory storing inactive experts everywhere.

```
Token routes to Expert 3 ──→ sent to GPU 1 (holds Expert 3)
Token routes to Expert 7 ──→ sent to GPU 2 (holds Expert 7)
```

**Memory:** each GPU holds only its assigned experts. The dense stack (attention, embeddings, shared layers — \~14 GB at 4-bit for Qwen3-235B) is replicated on every GPU. The expert planes (\~231 GB at 2-bit) are sharded. This is far more memory-efficient than TP, which would replicate all 256 experts on every GPU even though most sit idle for each token.

**Communication:** all-to-all between GPUs. When a token's router selects an expert on a different GPU, the activation must be sent there. This creates variable traffic patterns — some tokens stay local, others cross GPUs.

**vLLM flag:** `--enable-expert-parallel` combined with `--tensor-parallel-size`

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve Qwen/Qwen3-235B-A22B \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --gpu-memory-utilization 0.90
```

> [!TIP]
>
> DGX Spark's 128 GB unified memory is well-suited for expert parallelism. A single Spark with \~110 GB usable can hold the dense stack plus a large share of experts locally, reducing the cross-GPU routing traffic that EP generates. For models like Qwen3-235B where even EP on 1 node is not enough, combine expert pruning with 2-bit expert quantization to bring the per-node footprint under budget. See the [DGX Spark use case](https://docs.tokios.com/use-cases/dgx-spark.md) for hardware details.

## Combining strategies: TP, PP, and DP

When you have many GPUs, you combine strategies hierarchically:

* **TP within a node** (where the interconnect is fast).
* **PP across nodes** (where the interconnect is slower).
* **DP for throughput** (run multiple replicas of the combined TP+PP setup).

### 4 GPUs: TP=2 + PP=2

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 2 \
  --distributed-executor-backend ray
```

This creates 2 pipeline stages, each using 2-way tensor parallelism. GPUs 0 and 1 handle the first half of layers with TP, GPUs 2 and 3 handle the second half with TP.

### 8 GPUs: TP=4 + DP=2

Run 2 replicas of a 4-way tensor-parallel setup. Each replica uses 4 GPUs for TP, and a load balancer distributes requests between the 2 replicas. This gives you the low latency of TP=4 with 2× the throughput.

### The rule of thumb

1. **TP first** — within the fastest interconnect you have (NVLink, QSFP, PCIe).
2. **PP second** — across slower links (Ethernet, network between machines).
3. **DP last** — when you need more throughput and have spare GPUs for replicas.

vLLM requires `TP × PP = total GPUs used for one replica`. If you set `--tensor-parallel-size 4 --pipeline-parallel-size 2`, vLLM uses 8 GPUs for 1 replica.

## Sharding decision matrix

> [card] [70B, 2× RTX 4090 (PCIe)](#pipeline-parallelism-splitting-layers-sequentially)
>
> **PP.** PCIe bandwidth is too low for efficient AllReduce. Pipeline parallelism avoids the bottleneck.

> [card] [70B, 2× A100 (NVLink)](#tensor-parallelism-splitting-layers-across-gpus)
>
> **TP.** NVLink delivers enough bandwidth for AllReduce. Lower latency per token.

> [card] [235B MoE, DGX Spark](#expert-parallelism-for-mixture-of-experts-models)
>
> **EP.** Native MoE handling — each GPU holds only its experts, not the full 235B.

> [card] [7B, high traffic](#data-parallelism-running-the-same-model-on-multiple-gpus)
>
> **DP.** Model fits on 1 GPU. Replicate for throughput.

> [card] [70B, 4× A100 (NVLink)](#combining-strategies-tp-pp-and-dp)
>
> **TP=4** (or TP=2 + DP=2 if you need throughput over latency).

> [card] [70B, 2 machines](#pipeline-parallelism-splitting-layers-sequentially)
>
> **PP=2.** Network between machines is too slow for TP. Pipeline avoids the bandwidth wall.

## Practical commands

```bash vLLM tensor parallel theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90
```

```bash vLLM pipeline parallel theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --pipeline-parallel-size 2 \
  --tensor-parallel-size 1
```

```bash vLLM hybrid TP + PP theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-405B-Instruct \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 2 \
  --distributed-executor-backend ray
```

```bash SGLang distributed theme={"theme":{"light":"github-light","dark":"css-variables"}}
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --tp 2 \
  --port 30000
```

```bash llama.cpp split across GPUs theme={"theme":{"light":"github-light","dark":"css-variables"}}
./llama-server \
  -m llama-3.1-70b-instruct-q4_k_m.gguf \
  -ngl 99 \
  --split-mode layer \
  --tensor-split 0.5,0.5
```

### Verify sharding is working

Run `nvidia-smi` on every GPU. All participating GPUs should show memory allocated for the model:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Check all GPUs at once
nvidia-smi --query-gpu=index,memory.used,memory.total,utilization.gpu \
  --format=csv
```

With TP=2, you should see roughly equal memory usage on both GPUs. With PP=2, the memory split may be uneven if the model's early layers differ in size from later layers.

## Common mistakes

**Using TP over slow PCIe.** AllReduce sends activation data after every layer. On PCIe 3.0, this chokes decode throughput. If your GPUs lack NVLink, test PP and compare.

**Using PP with batch size 1.** The pipeline bubble wastes the most compute when only 1 token flows at a time. vLLM's continuous batching mitigates this — make sure you run with enough concurrent requests to fill the pipeline.

**Forgetting that DP requires N× model memory.** 4 replicas of a 70B Q4 model (\~35 GB each) need 140 GB total. Plan memory before adding replicas.

**Not accounting for KV cache.** Model weights are only part of GPU memory. The [KV cache](https://docs.tokios.com/concepts/kv-cache-and-context.md) for active sequences takes the rest. When you shard weights across 2 GPUs, you also shard the KV cache — effectively doubling your concurrent user capacity. But if you plan memory for weights alone, you will OOM. Budget for weights + KV cache + \~3 GB overhead before you pick a sharding strategy.

**Mixing TP and PP in the wrong order.** vLLM applies TP within each pipeline stage. Set `--tensor-parallel-size` to the GPU count within a single node and `--pipeline-parallel-size` to the node count. Reversing them puts PP within a node (wasting fast interconnect on point-to-point) and TP across nodes (saturating the network with AllReduce).

## Next steps

> [card] [Tensor vs pipeline parallelism](https://docs.tokios.com/guides/tensor-vs-pipeline-parallelism.md)
>
> Detailed TP vs PP comparison with performance benchmarks.

> [card] [One vs two DGX Sparks](https://docs.tokios.com/guides/one-vs-two-dgx-sparks.md)
>
> When to add a second Spark and how to configure the 200Gb/s link.

> [card] [DGX Spark memory tuning](https://docs.tokios.com/guides/dgx-spark-memory-tuning.md)
>
> KV cache math and memory parameters for sharded deployments.

> [card] [Model serving and scaling](https://docs.tokios.com/concepts/model-serving-and-scaling.md)
>
> How inference engines manage batching, precision, and parallelism.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.