> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Tensor vs pipeline parallelism for local LLM inference

> When to use tensor parallelism, pipeline parallelism, or both for multi-GPU LLM inference on DGX Spark, RTX workstations, and multi-node clusters.

When your model does not fit on a single GPU — or you need more throughput than 1 GPU delivers — you split the workload across multiple GPUs. The 2 main strategies are **tensor parallelism** (TP) and **pipeline parallelism** (PP). They solve different problems, and picking the wrong one wastes hardware.

## Quick verdict

| Scenario | Strategy | Why |
| - | - | - |
| Single node, 2+ GPUs (DGX Spark pair, 2× RTX 4090) | **TP** | Lower latency, better GPU-to-GPU bandwidth usage |
| Multi-node (2+ machines over network) | **PP** | Tolerates network latency; each node holds a model shard |
| Large model + high concurrency | **TP + PP hybrid** | TP within a node, PP across nodes |
| Throughput scaling (many users, model fits on 1 GPU) | **Data parallelism (DP)** | Replicate the model, split the requests |

**Use tensor parallelism for single-node multi-GPU. Use pipeline parallelism for multi-node.** This rule covers most setups.

## Tensor parallelism explained

Tensor parallelism splits each layer's weight matrices across GPUs. When a forward pass runs, every GPU computes a slice of the matrix multiplication simultaneously, then they synchronize via a high-bandwidth interconnect (NVLink, PCIe, or InfiniBand).

```
Layer N:
  GPU 0: weights[0:50%] × input  ──┐
                                    ├── all-reduce ──→ output
  GPU 1: weights[50:100%] × input ─┘
```

**Key properties:**

* Every GPU holds a fraction of every layer
* All GPUs are active on every token
* Requires high-bandwidth interconnect (NVLink >> PCIe)
* Adds synchronization overhead per layer, but keeps latency low per token
* vLLM calls this `--tensor-parallel-size`

**Best for:** 2× DGX Spark connected via 200Gb/s QSFP, 2× RTX 4090 over PCIe, or any single-node setup where GPUs share a fast bus.

## Pipeline parallelism explained

Pipeline parallelism assigns different layers to different GPUs. GPU 0 holds layers 0–15, GPU 1 holds layers 16–31. A forward pass flows like an assembly line: GPU 0 processes the input, passes activations to GPU 1, which continues.

```
GPU 0: [Layer 0–15]  ──activations──→  GPU 1: [Layer 16–31]  ──→ output
```

**Key properties:**

* Each GPU holds a contiguous shard of layers
* Only 1 GPU is active per token at a time (the others are idle or processing different tokens)
* Tolerates slower interconnects — activations are smaller than full weight matrices
* Adds pipeline bubbles (idle time) that reduce utilization
* vLLM calls this `--pipeline-parallel-size`

**Best for:** multi-node clusters where nodes are connected over 10GbE–100GbE network, or when the model is too large for TP alone.

## Decision matrix

Answer these questions to pick your strategy:

1. **Does the model fit on 1 GPU?**
   * Yes → no parallelism needed. Use data parallelism if you need more throughput.
   * No → continue.

2. **Are all GPUs in the same machine?**
   * Yes → **tensor parallelism** (lower latency, better decode throughput).
   * No → continue.

3. **Are GPUs across 2+ machines?**
   * Yes → **pipeline parallelism** across nodes. If each node has 2+ GPUs, use **TP within each node + PP across nodes**.

4. **Is it a very large model (405B+)?**
   * Yes → hybrid TP + PP. Example: TP=2 within each DGX Spark, PP=2 across 2 Sparks.

### Hardware-specific recommendations

| Hardware | GPUs | Interconnect | Recommended strategy |
| - | - | - | - |
| 2× DGX Spark (GB10) | 2 unified | 200Gb/s QSFP | TP=2 |
| 2× RTX 4090 | 2 discrete | PCIe 4.0 | TP=2 |
| 2× RTX 3090 | 2 discrete | PCIe 4.0 | TP=2 (watch OOM — vLLM reserves more than Ollama) |
| 2-node DGX Spark cluster | 2 nodes | 200Gb/s QSFP | PP=2 (or TP=2 + PP=2 for 405B) |
| 3+ node cluster | 3+ nodes | 10–100GbE | PP = node count, TP within each node |

## Data parallelism and expert parallelism

Two other strategies matter for specific scenarios:

**Data parallelism (DP)** — replicate the full model on each GPU and split incoming requests across them. Use this when the model already fits on 1 GPU and you want more concurrent users. vLLM supports this natively — just launch multiple instances behind a load balancer.

**Expert parallelism (EP)** — specific to MoE models. Different GPUs hold different expert groups. When a token routes to an expert, the activation is sent to the GPU holding that expert. Useful for very large MoE models where TP alone wastes memory on inactive experts.

| Strategy | Best for | Latency | Throughput | Hardware requirement |
| - | - | - | - | - |
| Tensor parallelism (TP) | Model too big for 1 GPU, single node | Low | High | Fast interconnect (NVLink, PCIe) |
| Pipeline parallelism (PP) | Multi-node, very large models | Higher (bubbles) | Medium | Network (10GbE+) |
| Data parallelism (DP) | Model fits, need more users | Same as single GPU | Scales linearly | Any — just more GPUs |
| Expert parallelism (EP) | Large MoE models | Medium | High | Fast interconnect |

## Configuration examples

### vLLM with tensor parallelism

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90
```

For 2× DGX Spark connected via 200Gb/s QSFP:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --distributed-executor-backend ray
```

### vLLM with pipeline parallelism

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --pipeline-parallel-size 2 \
  --tensor-parallel-size 1
```

### Hybrid TP + PP (large model, multi-node)

For a 405B model across 2× DGX Spark (TP within each node, PP across nodes):

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-405B-Instruct \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 2 \
  --distributed-executor-backend ray
```

### SGLang with tensor parallelism

SGLang is competitive with vLLM on dual-GPU setups for specific workloads:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --tp 2 \
  --port 30000
```

### NCCL environment variables

For multi-GPU communication, set these before launching:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=0          # Enable InfiniBand if available
export NCCL_SOCKET_IFNAME=eth0    # Pin the network interface
export NCCL_NET_GDR_LEVEL=2       # Enable GPUDirect RDMA
```

> [!NOTE]
>
> On DGX Spark clusters connected via 200Gb/s QSFP, NCCL automatically selects the optimal transport. You typically do not need to override `NCCL_IB_DISABLE` or `NCCL_SOCKET_IFNAME` unless you are debugging connectivity issues.

## Performance comparison

Based on community benchmarks for 70B-class dense models:

| Configuration | Decode tok/s (single user) | Concurrent users (5 tok/s each) | Notes |
| - | - | - | - |
| 1× DGX Spark (no parallelism) | \~10 | 2–3 | Baseline |
| 2× DGX Spark, TP=2 (200Gb/s QSFP) | \~18 | 5–8 | Near-linear scaling for decode |
| 2× RTX 4090, TP=2 (PCIe) | \~20 | 5–7 | PCIe bandwidth limits prefill |
| 2× DGX Spark, PP=2 (network) | \~12 | 4–6 | Pipeline bubbles reduce utilization |

> [!TIP]
>
> vLLM delivers **5.9× more tokens/s** than Ollama at 50 concurrent users on the same hardware. If you serve models to a team, the inference engine matters as much as the parallelism strategy.

## When not to use parallelism

Not every setup benefits from splitting across GPUs:

* **Model fits on 1 GPU with room to spare** — parallelism adds overhead. Run on 1 GPU and use data parallelism for throughput.
* **Small models (7B–13B) on consumer GPUs** — TP synchronization overhead can make 2× GPU slower than 1× GPU for small models. SGLang shows measurable degradation here.
* **Single-user inference** — unless the model literally does not fit, 1 GPU is simpler and often faster.
* **Network slower than 10GbE between nodes** — pipeline parallelism starves on slow links. Upgrade networking first.

## Troubleshooting common issues

### OOM on multi-GPU with vLLM

vLLM reserves memory for CUDA graphs and KV cache aggressively. On 2× RTX 3090 (48GB total), a 32B model may OOM even though Ollama handles it fine. Reduce `--gpu-memory-utilization` from the default 0.90:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve your-model \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.80
```

### NCCL connection failures

If GPUs cannot communicate, check:

1. `nvidia-smi topo -m` — verify NVLink or PCIe topology
2. `NCCL_DEBUG=INFO` — confirm which transport NCCL selects
3. Firewall rules on port 29500 (default NCCL rendezvous port)

### Mixed GPU types

vLLM supports heterogeneous GPU setups, but the slowest GPU determines throughput. For best results, use matched GPUs. If mixing, assign more layers to the faster GPU via pipeline parallelism with uneven stage allocation.

## Expose multi-GPU deployments via Tokios

Whether you run TP, PP, or both, your vLLM or SGLang server exposes an OpenAI-compatible `/v1` API on localhost. Run the Tokios connector on the same machine and your deployment becomes reachable from any device, agent, or teammate — without opening inbound ports.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Start vLLM with tensor parallelism
vllm serve your-model --tensor-parallel-size 2

# Pair the Tokios connector
curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
```

See the [vLLM remote access guide](https://docs.tokios.com/guides/vllm-remote-access.md) for the full walkthrough.

## Next steps

> [card] [Model serving and scaling](https://docs.tokios.com/concepts/model-serving-and-scaling.md)
>
> Understand how inference engines manage memory, batching, and KV cache.

> [card] [DGX Spark use case](https://docs.tokios.com/use-cases/dgx-spark.md)
>
> Set up Tokios on a DGX Spark or a 2-Spark cluster.

> [card] [vLLM remote access](https://docs.tokios.com/guides/vllm-remote-access.md)
>
> Expose your vLLM server securely from anywhere.

> [card] [Quantization guide](https://docs.tokios.com/concepts/quantization.md)
>
> Shrink models to fit on fewer GPUs before reaching for parallelism.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.