> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Run Kimi K3 locally: hardware and self-hosting reality

> Can you run Kimi K3 locally? Moonshot AI's 2.8T-parameter MoE needs ~1.4 TB at its 4-bit floor — no consumer or prosumer hardware fits it today.

> [!NOTE]
>
> Last verified: 2026-10-14

Kimi K3 is Moonshot AI's frontier model: **2.8 trillion total parameters**, **104 billion active per token** through Stable LatentMoE routing (16 active + 2 shared experts across 896 total), native visual input via MoonViT-V2, and a **1-million-token context window**. Moonshot launched K3 on July 17, 2026 for long-horizon coding, knowledge work, and reasoning, and released the downloadable weights on July 27-28, 2026 on [HuggingFace](https://huggingface.co/moonshotai/Kimi-K3) (1.1M+ downloads).

The weights, model card, and license are public. You can self-host K3 today on a sufficiently large cluster and connect it to Tokios.

> [!NOTE]
>
> Kimi K3 weights are available on [HuggingFace](https://huggingface.co/moonshotai/Kimi-K3). The repository has 1.1M+ downloads. You can self-host K3 with vLLM on a cluster with 64+ accelerators and connect it to Tokios.

## What is available?

| Path | Available? | Runs on your hardware? | Where Tokios fits |
| - | - | - | - |
| Kimi web, Kimi Work, Kimi Code | Yes | No | It's a Moonshot-hosted product, not a Tokios deployment |
| Official Kimi API (`kimi-k3`) | Yes | No | Call Moonshot directly; Tokios doesn't turn a third-party hosted API into a local model |
| Official K3 weights | Yes — released July 27, 2026 | Yes, on server-class hardware (64+ accelerators) | Serve the weights yourself and connect that local `/v1` endpoint to Tokios |
| Kimi K2.6 / K2.7-Code weights | Yes | Yes, on server-class hardware | Self-host K2.x today — see the [Kimi model family](https://docs.tokios.com/models/kimi.md) |

Source: [Kimi K3 launch post](https://www.kimi.com/blog/kimi-k3) and [Kimi K3 API guide](https://platform.kimi.com/docs/guide/kimi-k3-quickstart).

## Kimi K3 at a glance

| Property | Kimi K3 |
| - | - |
| Official API model id | `kimi-k3` |
| HuggingFace repository | `moonshotai/Kimi-K3` |
| Total parameters | 2.8T |
| Sparse routing | 16 active + 2 shared experts per token, 896 total via Stable LatentMoE |
| Active parameters per token | 104B |
| Context window | 1,048,576 tokens (1M) |
| Modalities | Text, image, and video understanding (MoonViT-V2 vision encoder, 401M params) |
| Architecture | 93 layers (69 KDA + 24 Gated MLA), hidden dim 7168, 96 attention heads |
| Attention | Kimi Delta Attention (KDA) plus Gated MLA |
| Residual path | Attention Residuals (AttnRes) |
| Native serving precision | MXFP4 weights with MXFP8 activations, trained with QAT from SFT onward |
| Thinking mode | Always on; `reasoning_effort="max"` is the only supported effort |
| Official self-hosting target | Supernode configuration with **64 or more accelerators** |
| Weight file size | \~1.4 TB at 4-bit floor |
| License | Custom "Kimi K3 License" (not MIT, not Apache — read before use) |
| Downloads | 1.1M+ on HuggingFace |

Moonshot published the technical report, checkpoint files, and model card alongside the weight release. Read the [model card](https://huggingface.co/moonshotai/Kimi-K3) for the full specification and license terms.

## Can you run Kimi K3 locally?

Yes — if you have a large enough cluster. The weights are available on [HuggingFace](https://huggingface.co/moonshotai/Kimi-K3), but K3's **\~1.4 TB weight floor** at 4-bit precision rules out every consumer and prosumer setup. Moonshot's own recommendation is a **supernode with 64 or more accelerators**. Eight DGX Sparks pool 1 TB of unified memory and still fall **376 GB short**. See the [DGX Spark memory math](https://docs.tokios.com/blog/kimi-k3-hardware-deep-dive.md) for the full calculation.

vLLM is the officially supported serving backend. You can also use Docker Model Runner or Transformers with `trust_remote_code`.

> [!TIP]
>
> K3 requires 64+ accelerators. See [what to run instead of Kimi K3](https://docs.tokios.com/models/kimi-k3-alternatives.md) for the best coding and general model for your exact hardware, from Strix Halo to a 4-Spark cluster.

## Kimi K3 hardware requirements

No consumer or prosumer setup holds K3. At exactly 4 bits per parameter, 2.8T parameters need about **1.4 TB** for the weights alone — before microscaling metadata, runtime buffers, activations, and context state. For scale: eight DGX Sparks pool 1 TB of unified memory and are still at least **376 GB short of the raw weight floor**, NVIDIA currently documents Spark clusters only to four nodes, and Moonshot recommends serving K3 on a supernode with **64 or more accelerators**.

See the [DGX Spark memory math](https://docs.tokios.com/blog/kimi-k3-hardware-deep-dive.md) for the full calculation, cluster table, and network constraints.

## Serving K3 locally

The officially supported serving path is vLLM. Docker Model Runner and HuggingFace Transformers (with `trust_remote_code`) also work.

```bash vLLM (recommended) theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve "moonshotai/Kimi-K3"
```

```python Transformers theme={"theme":{"light":"github-light","dark":"css-variables"}}
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "moonshotai/Kimi-K3",
    trust_remote_code=True
)
```

> [!WARNING]
>
> K3 needs KDA-aware kernels and cache semantics. A runtime that merely loads the weights is not automatically K3-compatible. Use vLLM with KDA support or one of the other verified backends.

## Why the architecture matters for self-hosting

K3 is not simply a larger K2 checkpoint. Five changes matter to anyone planning to serve it:

* **Kimi Delta Attention (KDA)** — a hybrid linear-attention design that makes million-token prompts cheaper to prefill. It needs matching kernels and cache semantics: Moonshot says conventional prefix caching does not map cleanly to KDA and that its vLLM prefill-cache contribution ships with the weights. A runtime that merely loads the weights is not automatically K3-compatible.
* **Attention Residuals (AttnRes)** — layers selectively retrieve useful representations from earlier depth instead of adding every layer's output with fixed weight. Moonshot's [AttnRes paper](https://arxiv.org/abs/2603.15031) reports more stable information flow in deeper models.
* **Stable LatentMoE** — 16 active + 2 shared experts route per token across 896 total. Sparse activation cuts per-token compute, but it does **not** shrink the stored checkpoint: every expert must stay resident across the serving cluster, which makes fast interconnects a first-order requirement.
* **MoonViT-V2** — a 401M-parameter vision encoder that processes text, image, and video input natively.
* **Native MXFP4** — quantization-aware training ran from the supervised fine-tuning stage onward, targeting MXFP4 weights and MXFP8 activations. This is the intended serving format, not a community quant — so the 4-bit figure above is a floor, not a compromise.

## K3 API behavior to preserve

If you test K3 through Moonshot's API, or validate a self-hosted runtime, these protocol details matter:

* K3 always reasons. Use the top-level `reasoning_effort: "max"`; do not send K2.x's `thinking` parameter.
* Streaming responses separate reasoning into `reasoning_content` and the final answer into `content`.
* For multi-turn conversations and tool loops, replay the **complete assistant message**, including preserved reasoning and tool-call fields. Returning only `content` can destabilize later turns.
* `max_completion_tokens` defaults to 131,072 and can be set as high as 1,048,576.
* The launch API fixes sampling values. Moonshot recommends that you omit `temperature`, `top_p`, `n`, and penalty parameters rather than override them.
* Public image URLs are not accepted. Send base64 data or an `ms://<file-id>` reference.

See the [official K3 API guide](https://platform.kimi.com/docs/guide/kimi-k3-quickstart) for the current request schema.

## Connect K3 with Tokios

Tokios sits after a local runtime. It doesn't download the model or replace vLLM. Serve K3 on a supported cluster and connect it to Tokios:

**1. Verify the official artifacts**

Confirm the repository is [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3). Read the model card, license, checkpoint manifest, supported accelerator list, and recommended parallelism settings.

**2. Serve K3 on the supported cluster**

Run `vllm serve "moonshotai/Kimi-K3"` or follow the Docker Model Runner / Transformers instructions. Confirm the local server can complete a streaming request, preserve `reasoning_content`, and execute a multi-turn tool loop on its own `/v1` API.

**3. Pair the Tokios connector**

Install the connector on a node that can reach the serving endpoint. In the [dashboard](https://tokios.com/console), open **Setup**, select **Start pairing**, and use the one-time `TKS-XXXX-XXXX` claim code.

```powershell Windows theme={"theme":{"light":"github-light","dark":"css-variables"}}
irm https://tokios.com/downloads/install/windows | iex; tokios-install -Code 'TKS-XXXX-XXXX'
```

```bash macOS theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX
```

```bash Linux theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
```

Then **approve** the device on the **Setup** or **Connectors** tab.

**4. Register a deployment**

Point the connector at the cluster's OpenAI-compatible `/v1` base URL. Register the upstream model id under a public deployment name such as `kimi-k3-tunnel`.

**5. Create an API key**

In the [Keys](https://tokios.com/console/keys) tab, click **Create key**. Copy the `sk-tok-…` key now — it's shown only once.

**6. Test before sharing the endpoint**

Test streaming, long turns, vision, structured output, and tool calls through `https://api.tokios.com`. Use a dedicated `sk-tok-…` API key and keep the initial context short while you validate memory use.

### Call it from anywhere

Once that path is verified, clients call your deployment name rather than the upstream model id:

```bash curl (OpenAI-compatible) theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kimi-k3-tunnel",
    "stream": true,
    "messages": [{"role": "user", "content": "Audit this repository and propose a phased refactor."}]
  }'
```

```bash Claude Code (Anthropic-compatible) theme={"theme":{"light":"github-light","dark":"css-variables"}}
export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model kimi-k3-tunnel
```

## Launch caveats

* K3 benchmark results are Moonshot-reported. Independent reproduction requires the evaluation setup.
* Moonshot warns that losing preserved reasoning history, or switching another model's existing session to K3, can make output unstable.
* K3 is trained for long, difficult tasks and can act too proactively when instructions are ambiguous. Put explicit boundaries in your system prompt or `AGENTS.md`.
* A 1M-token limit is a maximum, not a target. Long contexts increase prefill time, cache storage, and inference cost.
* The Kimi K3 License is custom — read the [model card](https://huggingface.co/moonshotai/Kimi-K3) for terms before deploying in production.

## FAQ

> [accordion] How much VRAM does Kimi K3 need?
>
> At its native MXFP4 precision (4 bits per parameter), K3's 2.8T parameters need about **1.4 TB** for weights alone — before block scales, runtime buffers, activations, and KV cache. That is roughly 11 DGX Sparks at their 128 GB each, just for the raw weight floor. In practice you need 14-16 Sparks minimum for a hypothetical distributed serving stack, and NVIDIA only documents Spark clusters up to 4 nodes.

> [accordion] Can Kimi K3 run on a DGX Spark cluster?
>
> Not practically. Eight DGX Sparks give you 1 TB of unified memory — 376 GB short of the weight floor. NVIDIA's documented Spark topology tops out at 4 nodes (512 GB), and Moonshot targets K3 at supernodes with 64 or more accelerators. For Spark-scale models today, see [DGX Spark model compatibility](https://docs.tokios.com/guides/dgx-spark-model-compatibility.md).

> [accordion] Is Kimi K3 open source?
>
> The weights are available on [HuggingFace](https://huggingface.co/moonshotai/Kimi-K3) under a custom "Kimi K3 License" — not MIT or Apache. Read the [model card](https://huggingface.co/moonshotai/Kimi-K3) for the full license terms before deploying. Kimi K2.6 and K2.7-Code are also open-weight — see the [Kimi model family](https://docs.tokios.com/models/kimi.md).

> [accordion] How do I run Kimi K3?
>
> Serve the checkpoint on a supported cluster with 64+ accelerators. The officially supported path is `vllm serve "moonshotai/Kimi-K3"`. Docker Model Runner and Transformers with `trust_remote_code` also work. Tokios can front that local endpoint with scoped API keys — see [Connect K3 with Tokios](#connect-k3-with-tokios) above. You can also call Moonshot's hosted API using the model id `kimi-k3`.

> [accordion] What is Kimi K3's context length?
>
> 1,048,576 tokens (1M). The `max_completion_tokens` parameter defaults to 131,072 and can be set as high as the full context window. Long contexts increase prefill time, cache storage, and inference cost — 1M is a maximum, not a target.

> [accordion] What architecture does Kimi K3 use?
>
> K3 is a Mixture-of-Experts model with 93 layers (69 KDA + 24 Gated MLA), hidden dim 7168, 96 attention heads, and 4 key innovations: **Stable LatentMoE** (16 active + 2 shared experts per token across 896 total, 104B active), **Kimi Delta Attention** (hybrid linear attention for efficient million-token prefill), **Attention Residuals** (depth-wise information routing), **MoonViT-V2** (401M-param vision encoder for text + image + video), and **native MXFP4** (quantization-aware training from SFT onward). These are not drop-in compatible with standard transformer serving — KDA in particular needs matching kernels and cache semantics.

> [card] [Kimi K3 on DGX Spark](https://docs.tokios.com/blog/kimi-k3-hardware-deep-dive.md)
>
> The full memory math, cluster constraints, and launch-evidence table.

> [card] [Kimi models](https://docs.tokios.com/models/kimi.md)
>
> Compare K3 with the downloadable K2.6 and K2.7-Code, and self-host Kimi today.

> [card] [What to run instead of K3](https://docs.tokios.com/models/kimi-k3-alternatives.md)
>
> The best coding and general model for your hardware, from Strix Halo to a 4-Spark cluster.

> [card] [Model serving and scaling](https://docs.tokios.com/concepts/model-serving-and-scaling.md)
>
> Understand tensor, pipeline, and expert parallelism before you plan a frontier-scale cluster.

> [card] [Quickstart](https://docs.tokios.com/quickstart.md)
>
> Pair a connector and register a local model that is available today.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.