> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Local LLM case studies

> Real deployment patterns from teams and individuals running local models. What worked, what didn't, and what they'd do differently.

These are composite profiles based on common deployment patterns. Each one captures real trade-offs, numbers, and lessons from running local models in production or personal use.

## Solo developer: coding assistant on a MacBook

**Setup:** Mac Studio M2 Ultra + Ollama + Claude Code via Tokios
**Model:** Qwen3-30B Q5\_K\_M
**Time to first working setup:** 20 minutes

A solo full-stack developer wanted a coding assistant that keeps source code on their machine. They run Ollama on a Mac Studio M2 Ultra with 64 GB unified memory and point Claude Code at it through Tokios.

```bash Model launch theme={"theme":{"light":"github-light","dark":"css-variables"}}
ollama pull qwen3:30b-a3b-q5_K_M
```

```bash Claude Code config theme={"theme":{"light":"github-light","dark":"css-variables"}}
export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model mac-coder
```

| Metric | Value |
| - | - |
| Tokens per second | \~28 tok/s |
| Monthly cost | \$0 (hardware already owned) |
| Cloud alternative | \~\$20/month for hosted API |
| Break-even | Already owned hardware |

**What worked:** The M2 Ultra's memory bandwidth handles the 30B model well. Tokios means the same setup works from a laptop on the road — just point Claude Code at `api.tokios.com`.

**Limitation:** Context length tops out around 8K tokens on this model. For large codebases with many files open, the developer chunks work into smaller sessions rather than feeding entire repositories.

> [!NOTE]
>
> **Key takeaway:** If you already own Apple silicon with 32 GB+ RAM, a local coding assistant costs \$0/month and keeps your code private. Tokios adds remote access without a VPN.

## Small team: shared model endpoint for 5 developers

**Setup:** RTX 4090 workstation + vLLM + Tokios connector
**Model:** Qwen3-72B FP8 (AWQ quantization)
**Time to first working setup:** 45 minutes

A 5-person backend team shares a single RTX 4090 workstation as a team model server. They use vLLM for throughput and Tokios to give each developer a scoped API key.

```bash vLLM launch theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve Qwen/Qwen2.5-72B-Instruct-AWQ \
  --host 127.0.0.1 \
  --port 8000 \
  --quantization awq \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 8
```

```json tokios-connector.json theme={"theme":{"light":"github-light","dark":"css-variables"}}
{
  "Upstream": {
    "BaseUrl": "http://127.0.0.1:8000/v1"
  }
}
```

| Metric | Value |
| - | - |
| Concurrent users | 5 |
| Aggregate throughput | \~30 tok/s |
| Per-user latency (1 user active) | \~800 ms first token |
| Per-user latency (5 users active) | \~2 s first token |
| VRAM used | \~42 GB of 24 GB (overflow to system RAM) |

**What worked:** Consistent model access with no rate limits. Scoped keys mean each developer has their own `sk-tok-…` key — revoking one doesn't affect others.

**Gotcha:** The 72B AWQ model barely fits in 24 GB VRAM. During peak concurrent usage, vLLM spills KV cache to CPU memory, which slows throughput. The team added `--max-num-seqs 8` to cap concurrent requests and prevent OOM crashes.

**What they'd change:** Move to a 48 GB A6000 or add a second GPU for tensor parallelism. See [Tensor vs pipeline parallelism](https://docs.tokios.com/guides/tensor-vs-pipeline-parallelism.md).

> [!NOTE]
>
> **Key takeaway:** A single consumer GPU can serve a small team, but plan for peak concurrency. Cap `--max-num-seqs` to prevent OOM and accept higher latency under load.

## Homelab enthusiast: DGX Spark as always-on inference server

**Setup:** DGX Spark + vLLM + Tokios + Tailscale (for LAN access)
**Models:** Rotating collection — Qwen3-30B for daily use, DeepSeek-R1 for reasoning tasks, Phi-4 for quick classification
**Time to first working setup:** 1 hour (including model downloads)

A homelab hobbyist runs a DGX Spark 24/7 as a personal inference server. The 128 GB unified memory lets them keep a large model loaded and swap others in as needed. Tokios exposes models to their phone, laptop, and home automation scripts.

```bash Daily driver model theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve Qwen/Qwen3-30B-A3B-FP8 \
  --host 127.0.0.1 \
  --port 8000 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 16384
```

```json tokios-connector.json (multi-model) theme={"theme":{"light":"github-light","dark":"css-variables"}}
{
  "Routes": [
    { "Deployment": "daily",      "Upstream": "vllm-main" },
    { "Deployment": "reasoning",  "Upstream": "vllm-reason" },
    { "Deployment": "classifier", "Upstream": "vllm-fast" }
  ],
  "Upstreams": {
    "vllm-main":   { "BaseUrl": "http://127.0.0.1:8000/v1" },
    "vllm-reason": { "BaseUrl": "http://127.0.0.1:8001/v1" },
    "vllm-fast":   { "BaseUrl": "http://127.0.0.1:8002/v1" }
  }
}
```

| Metric | Value |
| - | - |
| Idle power draw | \~65 W |
| Under-load power draw | \~200 W |
| Monthly electricity cost | \~\$8–15 (varies by region) |
| Uptime | 99%+ (systemd auto-restart) |

**What worked:** The DGX Spark's 128 GB unified memory fits models that would need 2–3 consumer GPUs. Running multiple vLLM instances on different ports lets them serve 3 models simultaneously. Tokios model routing means their phone app just changes the `model` field to switch between daily chat and deep reasoning.

**Tip:** Use [model routing](https://docs.tokios.com/guides/model-routing.md) to auto-select a deployment by request type. Point chat clients at `daily` and automation scripts at `classifier`.

> [!NOTE]
>
> **Key takeaway:** A DGX Spark as an always-on server costs \$8–15/month in electricity and gives you unlimited inference on models that won't fit on consumer hardware. The 128 GB unified memory is the real differentiator.

## Startup: local models for customer data processing

**Setup:** 2x A100 80 GB + vLLM + Tokios
**Models:** Llama 3.1 70B for analysis, Phi-4 for classification
**Time to first working setup:** 2 hours (including infrastructure)

A 10-person startup processes customer documents that contain sensitive financial data. Compliance requires that data never leaves their infrastructure. They run 2 separate vLLM instances — 1 for analysis, 1 for classification — and expose both through Tokios.

```bash Analysis model (A100 #1) theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --host 127.0.0.1 \
  --port 8000 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192
```

```bash Classification model (A100 #2) theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve microsoft/phi-4 \
  --host 127.0.0.1 \
  --port 8001 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 4096
```

```json tokios-connector.json theme={"theme":{"light":"github-light","dark":"css-variables"}}
{
  "Routes": [
    { "Deployment": "analyzer",   "Upstream": "analysis" },
    { "Deployment": "classifier", "Upstream": "classify" }
  ],
  "Upstreams": {
    "analysis": { "BaseUrl": "http://127.0.0.1:8000/v1" },
    "classify": { "BaseUrl": "http://127.0.0.1:8001/v1" }
  }
}
```

| Metric | Value |
| - | - |
| Documents processed per day | \~2,000 |
| Cost before (cloud GPU) | \~\$3,000/month |
| Cost after (owned A100s + Tokios) | \~\$500/month (amortized hardware + electricity) |
| Break-even | 3 months |
| Compliance requirement | Data never leaves premises |

**Architecture:** Their processing pipeline sends documents to the `analyzer` deployment for summarization and entity extraction, then routes extracted fields to the `classifier` deployment for categorization. Both models run on separate A100s to avoid VRAM contention.

**What worked:** Separate deployments for different tasks means they can scale, restart, or swap models independently. Tokios scoped keys limit the processing service to only the `analyzer` and `classifier` deployments.

**What they'd change:** Add a 3rd model for quality checking. The current 2-model pipeline occasionally misclassifies edge cases that a larger reasoning model would catch.

> [!NOTE]
>
> **Key takeaway:** When compliance requires on-premises data, local models pay for themselves fast. Separate deployments per task give you independent scaling and failure isolation. This startup broke even in 3 months versus cloud GPU costs.

## Researcher: benchmarking across quantizations

**Setup:** RTX 4090 + llama.cpp + lm-evaluation-harness
**Testing:** Same model (Qwen2.5-7B) at Q4\_K\_M, Q5\_K\_M, Q8\_0, FP16
**Time to full benchmark suite:** 3 hours

An ML researcher needed to measure how quantization affects evaluation scores for a specific task (code generation). They ran the same model at 4 quantization levels and compared perplexity and pass rates.

```bash Download GGUF files theme={"theme":{"light":"github-light","dark":"css-variables"}}
# From HuggingFace — 4 quantization levels of the same model
huggingface-cli download Qwen/Qwen2.5-7B-Instruct-GGUF \
  qwen2.5-7b-instruct-q4_k_m.gguf \
  qwen2.5-7b-instruct-q5_k_m.gguf \
  qwen2.5-7b-instruct-q8_0.gguf \
  --local-dir ./models
```

```bash Run llama.cpp server for each quant theme={"theme":{"light":"github-light","dark":"css-variables"}}
./llama-server \
  -m ./models/qwen2.5-7b-instruct-q5_k_m.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  -ngl 99 \
  -c 4096
```

```bash Benchmark through Tokios theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Point evaluation harness at Tokios endpoint
export OPENAI_API_BASE=https://api.tokios.com/v1
export OPENAI_API_KEY=sk-tok-YOUR_KEY

# Run eval suite
python -m lm_eval --model local-completions \
  --model_args model=bench-q5km,base_url=$OPENAI_API_BASE \
  --tasks humaneval \
  --batch_size 1
```

| Quantization | Model size | Tokens/sec | Eval score | VRAM used |
| - | - | - | - | - |
| Q4\_K\_M | 4.4 GB | \~55 | 72.1% | 5.2 GB |
| Q5\_K\_M | 5.1 GB | \~48 | 76.8% | 6.0 GB |
| Q8\_0 | 7.7 GB | \~38 | 78.2% | 8.5 GB |
| FP16 | 14 GB | \~25 | 79.0% | 15 GB |

**Finding:** Q5\_K\_M scored within 2.2 percentage points of FP16 while running almost twice as fast. For this researcher's code generation tasks, Q5\_K\_M is the sweet spot.

**How Tokios helped:** Registering each quantization as a separate deployment (`bench-q4km`, `bench-q5km`, `bench-q8`, `bench-fp16`) let the evaluation harness switch between them by changing only the `model` field. Results were directly comparable because the harness called the same endpoint shape for each run.

> [!NOTE]
>
> **Key takeaway:** Q5\_K\_M is the sweet spot for most 7B evaluation tasks — within 2–3% of FP16 quality at nearly 2x the speed. Register each quantization as a separate Tokios deployment to benchmark them through a consistent API.

## Common patterns across all case studies

| Pattern | Who uses it | Why |
| - | - | - |
| Scoped API keys per user | Teams, startups | Revoke access without affecting others |
| Separate deployments per task | Teams, researchers | Independent scaling and model swaps |
| Tokios for remote access | Solo devs, homelab | Reach models from laptop or phone without port forwarding |
| Quantized models | Everyone | Fit larger models in less VRAM with minimal quality loss |
| vLLM for throughput | Teams, startups | Better concurrent request handling than Ollama |

## Next steps

> [card] [Recipes](https://docs.tokios.com/guides/local-llm-recipes.md)
>
> Copy-paste configurations for 8 common local LLM setups.

> [card] [Test your deployment](https://docs.tokios.com/guides/benchmark-your-deployment.md)
>
> Structured benchmarks to compare your hardware and model choices.

> [card] [Model routing](https://docs.tokios.com/guides/model-routing.md)
>
> Register multiple deployments and switch between them.

> [card] [Choose a local model](https://docs.tokios.com/guides/choose-a-local-model.md)
>
> Pick the right model and quantization for your hardware.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.