> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Run Granite locally and use it from anywhere

> Serve IBM Granite 4 — an enterprise-focused, hybrid Mamba-2/transformer family built for tool-calling and RAG — with Ollama, then reach it through Tokios.

> [!NOTE]
>
> Last verified: 2026-10-14

Granite 4 is IBM's enterprise-focused open-weight family, tuned for instruction following, tool-calling, and Retrieval Augmented Generation (RAG). Most sizes use a hybrid Mamba-2/transformer architecture — only a handful of layers use full attention, and the rest use Mamba-2 — which keeps memory use down at long context. It ships from a 350M edge model up to a 32B mixture-of-experts (MoE) model with just 9B active parameters per token, all under Apache-2.0. Run whichever size fits your hardware locally, then pair the Tokios connector to call it from any machine, teammate, or hosted agent through `https://api.tokios.com` — no inbound ports, no VPN.

## Run it locally with Ollama

Granite 4 is in the Ollama library under the `granite4` slug. Pull and run the default tag:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
ollama run granite4
```

> [!NOTE]
>
> `granite4:latest` currently aliases to `granite4:micro` — the dense 3B fallback model, not one of the hybrid MoE models. If you want the hybrid Mamba-2 architecture, request a `-h` tag explicitly, for example `ollama run granite4:tiny-h`. See the [Ollama library page](https://ollama.com/library/granite4) for the full tag list — this page covers connecting it, not every runtime detail.

## Variants & tags

Every value below is from the Ollama library and the Hugging Face model cards. Tags and download sizes change as new quantizations ship — confirm against [ollama.com/library/granite4](https://ollama.com/library/granite4) before you standardize on one.

| Tag | Aliases | Architecture | Total / active params | Download size |
| - | - | - | - | - |
| `granite4:350m` | — | Dense transformer | 350M | 708 MB (`bf16`) |
| `granite4:350m-h` | — | Hybrid Mamba-2 | 350M | 366 MB |
| `granite4:1b` | — | Dense transformer | 1B | 3.3 GB (`bf16`) |
| `granite4:1b-h` | — | Hybrid Mamba-2 | 1B | 1.6 GB |
| `granite4:3b` | `micro` | Dense transformer | 3B | 2.1 GB |
| `granite4:3b-h` | `micro-h` | Hybrid Mamba-2 | 3B | 1.9 GB |
| `granite4:7b-a1b-h` | `tiny-h` | Hybrid Mamba-2 MoE | 7B total / **1B active** | 4.2 GB |
| `granite4:32b-a9b-h` | `small-h` | Hybrid Mamba-2 MoE | 32B total / **9B active** | 19 GB |

Sources: [ollama.com/library/granite4](https://ollama.com/library/granite4), [ollama.com/library/granite4/tags](https://ollama.com/library/granite4/tags), [ibm-granite/granite-4.0-h-tiny](https://huggingface.co/ibm-granite/granite-4.0-h-tiny), [ibm-granite/granite-4.0-h-small](https://huggingface.co/ibm-granite/granite-4.0-h-small).

> [!TIP]
>
> The non-`-h` sizes (350m, 1b, 3b/micro) are plain dense transformers, kept as fallback options for runtimes where Mamba-2 kernels aren't yet optimized. Everything tagged `-h` — `1b-h` through `32b-a9b-h` — uses the hybrid Mamba-2 architecture that Granite 4 is actually built around. If you have a choice, prefer the `-h` tag for the memory-efficiency benefit at long context. See [how inference works](https://docs.tokios.com/concepts/how-inference-works.md) for background on attention vs. state-space layers.

## Quantization & formats

Ollama's `granite4` tags ship as `GGUF`, mostly `Q4_K_M`-class quants, with `-bf16` and `-q8_0` variants available for the smaller dense and hybrid sizes (for example `1b-bf16`, `350m-h-q8_0`). The MoE hybrid models (`7b-a1b-h`, `32b-a9b-h`) are currently published only as the default quant on Ollama. See [Quantization formats and the hardware they need](https://docs.tokios.com/models/quantization-and-hardware.md) for the full matrix, and [what quantization is](https://docs.tokios.com/concepts/quantization.md) for how to read `Q4_K_M`-style names.

## Hardware fit

Comfort at the Ollama default quant for a modest context length. Treat every entry as an estimate — the ground truth is the actual `GGUF` size plus the KV cache for your context. Granite's Mamba-2 layers keep KV-cache growth lower than a pure-attention model of the same size, so long context costs less memory than the parameter count alone suggests. See [what size model fits your GPU?](https://docs.tokios.com/guides/model-fit.md) for the sizing math.

| Target | `350m`/`350m-h` (\~0.4–0.7 GB) | `1b`/`1b-h` (\~1.6–3.3 GB) | `3b`/`3b-h` (\~1.9–2.1 GB) | `7b-a1b-h` (\~4.2 GB) | `32b-a9b-h` (\~19 GB) |
| - | - | - | - | - | - |
| CPU-only laptop | Comfortable | Comfortable | Comfortable | Usable, slower | Not recommended |
| RTX 3060 / 4060 (8–12 GB) | Comfortable | Comfortable | Comfortable | Comfortable | Won't fit |
| RTX 3090 / 4090 (24 GB) | Comfortable | Comfortable | Comfortable | Comfortable, long context | Tight — reduce context |
| RTX 5090 / RTX 6000 Ada (32–48 GB) | Comfortable | Comfortable | Comfortable | Comfortable | Comfortable |
| Apple Silicon (16 GB+ unified) | Comfortable | Comfortable | Comfortable | Comfortable | Needs 32 GB+ unified |

## Run it — beginner to advanced

> [tab] Ollama (easiest)
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> ollama run granite4:tiny-h
> ```
>
> Leave Ollama listening on the default `http://127.0.0.1:11434`. Swap in `micro`, `small-h`, or any other tag from the table above.

> [tab] LM Studio (GUI)
>
> Search for **Granite 4** in LM Studio's model catalog, download a `GGUF` quant that fits your VRAM, then start LM Studio's local server. It exposes an OpenAI-compatible API on `http://127.0.0.1:1234/v1`.

> [tab] vLLM (advanced)
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> vllm serve ibm-granite/granite-4.0-h-small \
>   --max-model-len 131072 \
>   --enable-auto-tool-choice \
>   --tool-call-parser granite
> ```
>
> Add `--tensor-parallel-size <N>` to shard across GPUs. Requires a `transformers`/`vllm` version with `GraniteMoeHybridForCausalLM` support for the `-h` models.

> [tab] SGLang (advanced)
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> python -m sglang.launch_server \
>   --model-path ibm-granite/granite-4.0-h-small \
>   --context-length 131072 \
>   --tool-call-parser granite
> ```
>
> Add `--tp-size <N>` for tensor parallelism. Confirm hybrid-architecture support in your SGLang version before deploying an `-h` model.

## Capabilities

* **Tool-calling:** Native. Granite 4 instruct models are tuned for function-calling — define tools with an OpenAI-style function schema, and the model returns calls wrapped in `<tool_call>` XML tags. Ollama and vLLM/SGLang's `granite` tool-call parser handle the formatting for you.
* **Retrieval Augmented Generation (RAG):** Listed as a core intended use case on every Granite 4 model card, alongside summarization, text classification, and question-answering.
* **Long context:** 131,072 tokens (128K) natively across every size, dense and hybrid alike.
* **Hybrid architecture:** The `-h` models interleave a small number of full-attention layers (4 out of 40) with Mamba-2 state-space layers. This keeps KV-cache memory from growing linearly with context the way a pure-transformer model's does, so long-context inference is cheaper on the same hardware.
* **Mixture-of-experts:** `7b-a1b-h` (64 experts, \~1B active) and `32b-a9b-h` (72 experts, \~9B active) trade total capacity for lower per-token compute — but you still need to fit the full parameter count in memory. See the note below.

## Connect it with Tokios

Once Granite 4 is running locally, pair the Tokios connector so you can reach it as `https://api.tokios.com` from Claude Code, Cline, or any other compatible client — instead of only from `localhost:11434`.

**1. Pair the Tokios connector**

Sign in at [tokios.com/console](https://tokios.com/console), open the **Setup** tab, and click **Start pairing** to get a one-time claim code (`TKS-XXXX-XXXX`). Install and run the connector on the machine running Ollama:

```powershell Windows theme={"theme":{"light":"github-light","dark":"css-variables"}}
irm https://tokios.com/downloads/install/windows | iex; tokios-install -Code 'TKS-XXXX-XXXX'
```

```bash macOS theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX
```

```bash Linux theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
```

Then **approve** the device on the **Setup** or **Connectors** tab.

**2. Point the connector at Ollama's /v1 API**

Ollama exposes an OpenAI-compatible surface at `/v1`. For a single-model setup, the console's Setup wizard fills this in automatically as `http://127.0.0.1:11434/v1`. For multiple models, set it explicitly in `tokios-connector.json` — see [Connector config](https://docs.tokios.com/reference/connector-config.md).

**3. Register a deployment**

In the [Models](https://tokios.com/console/models) tab, register the model and choose a public deployment name, for example `granite-tunnel`. This name — not Ollama's local model id — is what clients send in the `model` field.

**4. Create an API key**

In the [Keys](https://tokios.com/console/keys) tab, click **Create key**. Copy the `sk-tok-…` key now — it's shown only once.

### Call it from anywhere

```bash curl (OpenAI-compatible) theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "granite-tunnel",
    "stream": true,
    "messages": [{"role": "user", "content": "Summarize this support ticket and suggest next steps."}]
  }'
```

```bash Claude Code (Anthropic-compatible) theme={"theme":{"light":"github-light","dark":"css-variables"}}
export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model granite-tunnel
```

> [!TIP]
>
> The `model` field is always your registered **deployment** name (`granite-tunnel` above), never Ollama's local model id. See [Use Claude Code](https://docs.tokios.com/guides/claude-code.md) and [Use Cline](https://docs.tokios.com/guides/cline.md) for full client setup.

## Caveats

* `granite4:latest` is the dense `micro` (3B) model, not a hybrid Mamba-2 model — pull a `-h` tag explicitly if that's what you want.
* The MoE hybrid models (`7b-a1b-h`, `32b-a9b-h`) need their full total parameter count in memory even though only a fraction is active per token — size your hardware off the download size, not the active-parameter number.
* Hybrid Mamba-2 support is newer in some inference runtimes. If vLLM, SGLang, or another server doesn't yet support `GraniteMoeHybridForCausalLM`, fall back to a dense tag (`3b`, `1b`, `350m`) or update the runtime.
* Tool-call parsing depends on the server applying Granite's chat template and tool-call format. Ollama handles this for you; on vLLM/SGLang, pass the `granite` tool-call parser or calls won't parse correctly.

> [card] [Quickstart](https://docs.tokios.com/quickstart.md)
>
> Pair a connector and register your first model end to end.

> [card] [What size model fits your GPU?](https://docs.tokios.com/guides/model-fit.md)
>
> Match Granite 4 to your available VRAM before you pull it.

> [card] [Quantization & hardware](https://docs.tokios.com/models/quantization-and-hardware.md)
>
> Which of Granite's formats runs on your hardware.

> [card] [Which local model for which task?](https://docs.tokios.com/guides/model-by-task.md)
>
> See how Granite compares to other local models for tool-calling and RAG.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.