> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Run Gemma 4 locally and use it from anywhere

> Serve Google's Gemma — the multimodal, multilingual open-weight family from Gemma 3 to Gemma 4 — with Ollama or LM Studio, then reach it through Tokios.

> [!NOTE]
>
> Last verified: 2026-10-14

Gemma is Google DeepMind's open-weight model family. **Gemma 4** is the current generation — multimodal (text, image, and on the smallest sizes, audio), multilingual across 140+ languages, and available as both dense and mixture-of-experts (MoE) models from edge-sized to desktop-GPU-sized. **Gemma 3** is the prior generation and still widely used, especially at the small end (`270m`, `1b`) for on-device and CPU-only work. Run whichever fits your hardware locally, then pair the Tokios connector to call it from any machine, teammate, or hosted agent through `https://api.tokios.com` — no inbound ports, no VPN.

## Run it locally with Ollama

Both generations are in the Ollama library. For Gemma 4, pull and run the default tag (the `E4B` edge model):

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
ollama run gemma4
```

For the desktop-GPU tier, run `ollama run gemma4:12b`, `ollama run gemma4:26b` (MoE), or `ollama run gemma4:31b`. For the prior generation, `ollama run gemma3` runs the `4b` default — see the [gemma4](https://ollama.com/library/gemma4) and [gemma3](https://ollama.com/library/gemma3) library pages for the full tag lists — this page covers connecting it, not every runtime detail.

> [!NOTE]
>
> Ollama's default context window is small (2048 tokens) — far below Gemma's native 128K–256K. Raise `num_ctx` in your Modelfile or client request to use more of its context.

## Variants & tags

Every value below is from the model cards and the Ollama library. Tags and download sizes change as new quantizations ship — confirm against [ollama.com/library/gemma4](https://ollama.com/library/gemma4) and [ollama.com/library/gemma3](https://ollama.com/library/gemma3) before you standardize on one.

| Property | `E2B` | `E4B` | `12B` | `26B-A4B` (MoE) | `31B` |
| - | - | - | - | - | - |
| Ollama tag | `gemma4:e2b` | `gemma4:e4b` (alias `:latest`) | `gemma4:12b` | `gemma4:26b` | `gemma4:31b` |
| Download size (`Q4_K_M`) | \~7.2 GB | \~9.6 GB | \~7.6 GB | \~18 GB | \~20 GB |
| Parameters | Edge-sized dense | Edge-sized dense | 12B dense | 26B total / **\~4B active** (128 experts, 8 routed + 1 shared) | 31B dense (31.3B) |
| Context window | 131,072 | 131,072 | 262,144 | 262,144 | 262,144 |
| Modality | Text + image + audio (ASR) | Text + image + audio (ASR) | Text + image | Text + image | Text + image |
| License | Apache-2.0 | Apache-2.0 | Apache-2.0 | Apache-2.0 | Apache-2.0 |
| Hugging Face | [`google/gemma-4-E2B-it`](https://huggingface.co/google/gemma-4-E2B-it) | [`google/gemma-4-E4B-it`](https://huggingface.co/google/gemma-4-E4B-it) | [`google/gemma-4-12B-it`](https://huggingface.co/google/gemma-4-12B-it) | [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it) | [`google/gemma-4-31B-it`](https://huggingface.co/google/gemma-4-31B-it) |

| Property | `270m` | `1b` | `4b` | `12b` | `27b` |
| - | - | - | - | - | - |
| Ollama tag | `gemma3:270m` | `gemma3:1b` | `gemma3:4b` | `gemma3:12b` | `gemma3:27b` |
| Download size (`Q4_K_M`) | — (fp16/q8\_0 only) | \~1.0 GB (`-it-qat`) | \~3.3 GB | \~8.1 GB | \~17 GB |
| Modality | Text only | Text only | Text + image | Text + image | Text + image |
| License | Gemma license | Gemma license | Gemma license | Gemma license | Gemma license |

Sources: [ollama.com/library/gemma4](https://ollama.com/library/gemma4), [ollama.com/library/gemma3](https://ollama.com/library/gemma3), [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it), [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it), [google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it), [google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it).

> [!TIP]
>
> The `26B-A4B` is a mixture-of-experts model with only about **4B active parameters** per token (8 of 128 experts routed, plus 1 shared expert), so it decodes much faster than its 26B total suggests — but all 26B parameters still have to fit in memory. Don't size your GPU off the active-parameter count alone. See [how inference works](https://docs.tokios.com/concepts/how-inference-works.md) for why.

## Quantization & formats

Ollama's default tags are `GGUF` `Q4_K_M`. The `gemma4` library also publishes `-it-qat` (quantization-aware-trained int4), `-q8_0`, `-bf16`, `-mlx` (Apple Silicon), `-mxfp8`, and `-nvfp4` (NVIDIA Blackwell) variants for each size. Which format you can run depends on your hardware — see [Quantization formats and the hardware they need](https://docs.tokios.com/models/quantization-and-hardware.md) for the full matrix, and [what quantization is](https://docs.tokios.com/concepts/quantization.md) for how to read the `Q4_K_M`-style names.

> [!TIP]
>
> **QAT (quantization-aware training).** Gemma 4's `-it-qat` tags are int4 checkpoints Google trained to stay accurate at 4-bit, so they hold quality better than a plain post-training `Q4` quant — at a comparable or *smaller* download: `gemma4:12b-it-qat` is \~7.2 GB (vs \~7.6 GB for `Q4_K_M`), and `gemma4:e4b-it-qat` is \~6.1 GB (vs \~9.6 GB). If you're running Gemma 4 at 4-bit, prefer the `-it-qat` tag. See [quantization-aware training](https://docs.tokios.com/concepts/qat.md) for how it works.

> [!TIP]
>
> **Apple Silicon?** MLX builds of Gemma 4 are available from [mlx-community](https://huggingface.co/mlx-community) on Hugging Face. Serve them with `mlx_lm.server` for native unified-memory inference on Mac Studio, Mac mini, and MacBook Pro — see [MLX-LM Server](https://docs.tokios.com/setup/supported-backends.md#mlx-lm-server) in the backends reference.

## Hardware requirements — will Gemma 4 fit your GPU?

Comfort at roughly `Q4` for a modest context length. Treat every entry as an estimate — the ground truth is the actual `GGUF` size plus the KV cache for your context. See [what size model fits your GPU?](https://docs.tokios.com/guides/model-fit.md) for the sizing math.

| Target | `E2B`/`E4B` edge (\~7–10 GB) | `12B` dense (\~7.6 GB) | `26B-A4B` MoE (\~18 GB) | `31B` dense (\~20 GB) |
| - | - | - | - | - |
| Laptop / CPU-only, ≤16 GB RAM | Comfortable (`E2B`) | Tight | Won't fit at `Q4` | Won't fit at `Q4` |
| RTX 3090 / 4090 (24 GB) | Comfortable | Comfortable | Comfortable | Comfortable |
| RTX 5090 (32 GB, Blackwell) | Comfortable; `nvfp4` tag available | Comfortable; `nvfp4` tag available | Comfortable; `nvfp4`-style formats improving | Comfortable |
| RTX 6000 Ada / A6000 (48 GB) | Comfortable, long context | Comfortable, long context | Comfortable, long context | Comfortable, long context |
| RTX 5080 / 4080 (16 GB) | Comfortable | Comfortable | Tight — reduce context | Tight — reduce context |
| Apple Silicon (16 GB+ unified) | Comfortable; `mlx` tag available | Comfortable; `mlx` tag available | Needs 24 GB+ unified | Needs 24 GB+ unified |

## Run it — beginner to advanced

> [tab] Ollama (easiest)
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> ollama run gemma4
> ```
>
> Leave Ollama listening on the default `http://127.0.0.1:11434`. Raise `num_ctx` to use more context, and swap in `gemma4:12b`, `gemma4:26b`, or `gemma4:31b` for a larger tier, or `gemma3` for the prior generation.

> [tab] LM Studio (GUI)
>
> Search for **Gemma 4** (or **Gemma 3**) in LM Studio's model catalog, download a `GGUF` quant that fits your VRAM (start with `Q4_K_M`, or `-it-qat` for a smaller int4-native download), then start LM Studio's local server. It exposes an OpenAI-compatible API on `http://127.0.0.1:1234/v1`.

> [tab] vLLM (advanced)
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> vllm serve google/gemma-4-12B-it \
>   --max-model-len 262144
> ```
>
> Add `--tensor-parallel-size <N>` to shard across GPUs. Swap in `google/gemma-4-26B-A4B-it`, `google/gemma-4-31B-it`, or an `E2B`/`E4B` repo for a different tier — check the model card for the current recommended `transformers`/`vllm` version before serving.

> [tab] SGLang (advanced)
>
> ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
> python -m sglang.launch_server \
>   --model-path google/gemma-4-12B-it \
>   --context-length 262144
> ```
>
> Add `--tp-size <N>` for tensor parallelism. Check the model card for the current recommended `transformers`/`sglang` version before serving.

## Capabilities

* **Reasoning:** Gemma 4 supports a configurable **thinking mode**, off by default. Enable it by including the `<|think|>` control token at the start of the system prompt (or `enable_thinking=True` if you're using the model card's `parse_response` helper); the model then emits step-by-step reasoning before its final answer.
* **Native tool-calling:** Yes, on Gemma 4 — the model card documents native function-calling for structured tool use and agentic workflows, using the standard `system`/`user`/`assistant` roles.
* **Vision:** Multimodal — accepts text and images (with variable aspect ratio and resolution) across the whole Gemma 4 family; Gemma 3 (`4b` and up) is also vision-capable.
* **Audio:** `E2B` and `E4B` only — native automatic speech recognition (ASR) and speech-to-translated-text, up to 30 seconds of audio per request.
* **Long context:** 131,072 tokens natively on `E2B`/`E4B`, 262,144 tokens on `12B`, `26B-A4B`, and `31B`.

## Connect it with Tokios

Once Gemma is running locally, pair the Tokios connector so you can reach it as `https://api.tokios.com` from Claude Code, Cline, or any other compatible client — instead of only from `localhost:11434`.

**1. Pair the Tokios connector**

Sign in at [tokios.com/console](https://tokios.com/console), open the **Setup** tab, and click **Start pairing** to get a one-time claim code (`TKS-XXXX-XXXX`). Install and run the connector on the machine running Ollama:

```powershell Windows theme={"theme":{"light":"github-light","dark":"css-variables"}}
irm https://tokios.com/downloads/install/windows | iex; tokios-install -Code 'TKS-XXXX-XXXX'
```

```bash macOS theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX
```

```bash Linux theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
```

Then **approve** the device on the **Setup** or **Connectors** tab.

**2. Point the connector at Ollama's /v1 API**

Ollama exposes an OpenAI-compatible surface at `/v1`. For a single-model setup, the console's Setup wizard fills this in automatically as `http://127.0.0.1:11434/v1`. For multiple models, set it explicitly in `tokios-connector.json` — see [Connector config](https://docs.tokios.com/reference/connector-config.md).

**3. Register a deployment**

In the [Models](https://tokios.com/console/models) tab, register the model and choose a public deployment name, for example `gemma-tunnel`. This name — not Ollama's local model id — is what clients send in the `model` field.

**4. Create an API key**

In the [Keys](https://tokios.com/console/keys) tab, click **Create key**. Copy the `sk-tok-…` key now — it's shown only once.

### Call it from anywhere

```bash curl (OpenAI-compatible) theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma-tunnel",
    "stream": true,
    "messages": [{"role": "user", "content": "Summarize this document and list open questions."}]
  }'
```

```bash Claude Code (Anthropic-compatible) theme={"theme":{"light":"github-light","dark":"css-variables"}}
export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model gemma-tunnel
```

> [!TIP]
>
> The `model` field is always your registered **deployment** name (`gemma-tunnel` above), never Ollama's local model id. See [Use Claude Code](https://docs.tokios.com/guides/claude-code.md) and [Use Cline](https://docs.tokios.com/guides/cline.md) for full client setup.

## Caveats

* Gemma 4's thinking mode is **off by default** — the opposite of some other reasoning families. If you want step-by-step reasoning, you have to opt in with the `<|think|>` control token or your client's equivalent.
* Audio input (ASR and speech translation) only works on the `E2B` and `E4B` sizes — `12B`, `26B-A4B`, and `31B` accept text and image only.
* The `26B-A4B` MoE needs all 26B parameters in memory even though only about 4B are active per token — size your hardware off the full download, not the active count.
* Licensing differs by generation: **Gemma 4** is **Apache 2.0** and ungated, while **Gemma 3** uses Google's custom **Gemma license** and its Hugging Face checkpoints are **gated** (you accept the terms before downloading; the Ollama tags are redistributed under the same license). Google's Gemma Prohibited Use Policy applies to both — check the [license terms](https://ai.google.dev/gemma/docs/gemma_4_license) before commercial or sensitive deployments.
* Gemma 3 remains useful for very small, CPU-friendly deployments (`270m`, `1b`) where Gemma 4's smallest tier (`E2B`) may still be heavier than you need.

> [!NOTE]
>
> **New same-class alternative.** Meta's [Muse Glimmer 30B](https://docs.tokios.com/models/muse-glimmer.md) lands in the same tier as `gemma4:31b` — dense, multimodal, Apache 2.0, and sized for one consumer GPU. Meta benchmarks them against each other directly and its numbers favour Muse Glimmer, but they are vendor-run with no independent reproduction yet. Gemma 4 still spans a much wider range, from `E2B` on edge hardware up to `31B`; Muse Glimmer ships at one size. Test both on your own workload.

## FAQ

> [accordion] How much VRAM does Gemma 4 need?
>
> At `Q4_K_M`, the edge models (`E2B`, `E4B`) need \~7–10 GB, the `12B` dense needs \~7.6 GB, the `26B-A4B` MoE needs \~18 GB, and the `31B` dense needs \~20 GB. The `-it-qat` int4-native tags are smaller still — for example `gemma4:12b-it-qat` is \~7.2 GB. BF16 builds are much larger: the `31B` alone is \~57 GB. Add KV cache on top for your context length.

> [accordion] Can Gemma 4 run on a 24 GB GPU?
>
> Yes — every Gemma 4 variant fits a 24 GB card (RTX 3090 / 4090) at `Q4_K_M`, including the `31B` dense at \~20 GB and the `26B-A4B` MoE at \~18 GB. The edge models and `12B` leave plenty of headroom for long context; the `31B` is comfortable but tighter at very large context lengths.

> [accordion] Is Gemma 4 open source?
>
> Gemma 4 is released under **Apache 2.0** and is ungated — no acceptable-use policy or sign-up to download. Gemma 3 uses Google's custom **Gemma license** and its Hugging Face checkpoints are gated (you accept the terms before downloading). Google's Gemma Prohibited Use Policy applies to both generations. Note that open weights and open source are not the same thing — the training data and pipeline are not released.

> [accordion] What is QAT and how does it help?
>
> QAT (quantization-aware training) means Google trained the model to stay accurate at 4-bit from the start, rather than applying quantization after training. The result: `-it-qat` tags hold quality better than a plain post-training `Q4_K_M` quant, at a comparable or smaller download — `gemma4:12b-it-qat` is \~7.2 GB vs \~7.6 GB for `Q4_K_M`, and `gemma4:e4b-it-qat` is \~6.1 GB vs \~9.6 GB. If you are running Gemma 4 at 4-bit, prefer the `-it-qat` tag.

> [accordion] Should I use Gemma 3 or Gemma 4?
>
> Use Gemma 4 — it is the current generation, with multimodal input (text + image across all sizes, audio on `E2B`/`E4B`), Apache 2.0 licensing, and context windows of 131K–262K tokens. Gemma 3 remains useful for very small, CPU-friendly deployments (`270m`, `1b`) where Gemma 4's smallest tier (`E2B`) is still heavier than you need, or if you specifically rely on a Gemma 3 tag that has no Gemma 4 equivalent.

> [card] [Quickstart](https://docs.tokios.com/quickstart.md)
>
> Pair a connector and register your first model end to end.

> [card] [What size model fits your GPU?](https://docs.tokios.com/guides/model-fit.md)
>
> Match Gemma to your available VRAM before you pull it.

> [card] [Quantization & hardware](https://docs.tokios.com/models/quantization-and-hardware.md)
>
> Which of Gemma's formats — `GGUF`, `QAT`, `MLX`, `NVFP4` — runs on your hardware.

> [card] [Which local model for which task?](https://docs.tokios.com/guides/model-by-task.md)
>
> See how Gemma compares to other local models by task.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.