> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Tokios vs building your own local LLM stack

> What it takes to build a DIY local LLM stack (runtime + proxy + tunnel + auth + TLS + DNS + monitoring) and when that's worth it vs using Tokios.

## What a DIY stack looks like

A production-ready local LLM API has 7 layers. Here's what each one does and what people typically use:

| Layer | Purpose | Common tools |
| - | - | - |
| Runtime | Serve the model | Ollama, vLLM, llama.cpp, LM Studio, SGLang |
| Reverse proxy | Route requests, terminate TLS, buffer control | nginx, Caddy, Traefik |
| Tunnel | Make the service reachable without a public IP | Cloudflare Tunnel, Tailscale Funnel, frp, ngrok |
| Auth | Per-user API keys, token validation | Custom middleware, LiteLLM virtual keys, Cloudflare Access |
| TLS | Encrypt traffic in transit | Let's Encrypt + Caddy auto-TLS, or Cloudflare edge TLS |
| DNS | Map a domain name to your endpoint | Cloudflare DNS, Route 53, your registrar |
| Monitoring | Uptime, latency, error rates, usage | Prometheus + Grafana, Uptime Kuma, Datadog |

Each layer is a separate tool to install, configure, update, and debug. They interact in non-obvious ways — nginx buffering affects SSE streaming, tunnel timeouts interact with proxy timeouts, and auth middleware needs to understand the API protocol.

## Setup comparison

| Step | DIY stack | Tokios |
| - | - | - |
| Install runtime | You run Ollama/vLLM/etc. | You run Ollama/vLLM/etc. |
| Configure reverse proxy | Write nginx/Caddy config, tune `proxy_buffering off` for SSE, set timeouts | Not needed — gateway handles it |
| Set up tunnel | Install `cloudflared` or Tailscale Funnel, configure ingress rules | Install connector (1 command) |
| Add authentication | Write or deploy auth middleware, manage key storage and rotation | Create `sk-tok-…` keys in dashboard |
| Provision TLS | Let's Encrypt + Caddy, or Cloudflare edge certificates | Handled by Tokios gateway |
| Configure DNS | Create A/CNAME records, wait for propagation | Not needed — uses `api.tokios.com` |
| Set up monitoring | Deploy Prometheus/Grafana or Uptime Kuma, configure alerts | Built-in dashboard |
| **Total setup time** | **4–8 hours** (first time) | **2–5 minutes** |
| **Ongoing maintenance** | **2–4 hours/month** (updates, cert renewals, config drift) | **\~0** (managed service) |

## Where DIY wins

* **Full control.** You own every layer. You can inspect, modify, and debug anything. No opaque managed service between your client and your model.
* **No third-party data path.** All traffic stays on infrastructure you control. For regulated environments (healthcare, finance, government), this may be a hard requirement — not a preference.
* **No vendor lock-in.** If Tokios shuts down or changes pricing, you're stuck. A DIY stack you built, you can maintain indefinitely.
* **Unlimited customization.** Custom rate limiting, your own logging pipeline, model-specific routing rules, A/B testing — anything is possible when you control the proxy config.
* **Potentially lower cost at scale.** For 100+ users, the per-seat cost of a managed service can exceed the cost of self-hosted infrastructure. At scale, DIY economics improve.

## Where Tokios wins

* **Minutes vs hours.** The entire Tokios setup — install connector, register model, create key — takes 2–5 minutes. A DIY stack takes 4–8 hours for a first-time setup, even for someone who knows the tools.
* **No maintenance burden.** Certificates renew, tunnels reconnect, and the gateway stays updated without your involvement. A DIY stack needs ongoing attention: cert expirations, tunnel crashes, security patches on nginx.
* **Built-in streaming.** SSE streaming works correctly out of the box. In a DIY stack, you need to explicitly disable `proxy_buffering` in nginx, configure `proxy_read_timeout`, and verify that your tunnel doesn't buffer responses.
* **Built-in auth.** Per-user API keys with deployment scoping. In DIY, you write or deploy an auth middleware, handle key storage securely, and build revocation logic.
* **CGNAT handling.** Tokios's outbound-only connection works behind any NAT. A DIY tunnel setup may or may not handle CGNAT depending on the tool — Cloudflare Tunnel does, but nginx + Let's Encrypt alone does not.
* **No security surface to manage.** No exposed ports, no certificate lifecycle, no auth middleware vulnerabilities. The connector dials out; the gateway handles everything else.

## The streaming pitfalls

This is where DIY stacks most commonly break for LLM workloads. 3 separate layers can buffer or time out a streaming response:

1. **nginx buffering.** By default, nginx buffers upstream responses. You need `proxy_buffering off;` and `proxy_cache off;` in the location block for your LLM endpoint. Miss this, and tokens dump in one burst at the end.

2. **Proxy timeouts.** nginx's default `proxy_read_timeout` is 60 seconds. A long generation on consumer hardware can exceed that. You need to set it to 300s or more. If a tunnel sits in front (Cloudflare Tunnel), its \~100–120s timeout becomes the new ceiling.

3. **SSE configuration.** The `Content-Type: text/event-stream` header must propagate from your backend through every proxy layer to the client. If any layer strips or overrides it, downstream buffering may kick in.

Each of these is solvable — but they're 3 separate configs across 2–3 tools, and the failure mode is silent: streaming appears to work but actually delivers all tokens at once, or cuts off mid-generation with a `524` or `504` error.

Tokios's gateway handles all 3 correctly by default because it's designed for this specific traffic pattern.

## When DIY makes sense

* **Regulated environments.** Healthcare, finance, or government work where no third-party data path is permitted. All traffic must stay on infrastructure you control and can audit.
* **Existing DevOps capacity.** You already run Kubernetes, have monitoring in place, and adding one more service is incremental work — not a new skill set.
* **Need full control.** Custom routing logic, a logging pipeline beyond Tokios's built-in opt-in [request logging](https://docs.tokios.com/setup/request-logging.md), or integration with internal systems that a managed service can't accommodate.
* **100+ users where per-seat cost matters.** At high user counts, self-hosted infrastructure can be significantly cheaper than per-seat managed pricing.

## When Tokios makes sense

* **Want the outcome without building the stack.** Your goal is "my model is reachable with an API key" — not "I want to learn nginx config."
* **Small teams.** 2–10 people who need model access. It's free on the community plan, and nobody's time goes to infrastructure maintenance.
* **Behind CGNAT.** Your ISP blocks inbound connections. A DIY tunnel can work, but Tokios handles it natively with no extra configuration.
* **Need fast setup.** You want to be making API calls in 5 minutes, not spending an afternoon configuring proxy layers.

> [card] [Tokios vs LiteLLM](https://docs.tokios.com/guides/tokios-vs-litellm.md)
>
> When an open-source proxy/gateway is the right choice vs a managed service.

> [card] [When to use Tokios](https://docs.tokios.com/guides/when-to-use-tokios.md)
>
> An honest guide to when Tokios is — and isn't — the right choice.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.