> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Local models in LangChain, LlamaIndex, and Vercel AI SDK

> Point LangChain, LlamaIndex, the Vercel AI SDK, or Pydantic AI at your Tokios deployment using the same OpenAI-compatible base URL each already supports.

All four frameworks let you set a custom OpenAI-compatible base URL for their OpenAI provider — that's the integration point. Point it at `https://api.tokios.com/v1`, pass an `sk-tok-…` API key, and set `model` to your registered **deployment** name (for example, `gemma-tunnel`), not your local backend's model id. If you've ever run a keyless local server, you've probably typed a throwaway string like `"not-needed"` just to satisfy a client that requires an API key field — with Tokios that field takes a real, revocable key instead.

## Prerequisites

> [card] [Install the connector](https://docs.tokios.com/setup/connector-install.md)
>
> Pair `tokios-connector` with your local backend.

> [card] [Register a deployment](https://docs.tokios.com/setup/register-model.md)
>
> Choose the public name your `model` field will use.

> [card] [Create an API key](https://docs.tokios.com/setup/api-keys.md)
>
> Generate an `sk-tok-…` key, scoped to the deployments you need.

> [card] [Supported backends](https://docs.tokios.com/setup/supported-backends.md)
>
> Confirm Ollama, llama.cpp, vLLM, or LM Studio is wired up as the upstream.

## Configure your framework

> [tab] LangChain
>
> LangChain's `ChatOpenAI` class takes a `base_url` and `api_key` like the OpenAI SDK it wraps:
>
> ```python langchain_tokios.py theme={"theme":{"light":"github-light","dark":"css-variables"}}
> from langchain_openai import ChatOpenAI
>
> llm = ChatOpenAI(
>     base_url="https://api.tokios.com/v1",
>     api_key="sk-tok-YOUR_KEY",
>     model="gemma-tunnel",
> )
>
> print(llm.invoke("Summarize this code: print('hello')").content)
> ```

> [tab] LlamaIndex
>
> LlamaIndex talks to OpenAI-compatible servers through `OpenAILike`, which wraps the OpenAI client but defaults to settings tuned for a generic local server rather than a chat-capable model. Set `is_chat_model` and `is_function_calling_model` explicitly if your deployment supports chat and tool calls, and set `context_window` to match your model — otherwise LlamaIndex falls back to conservative defaults (`is_chat_model=False` and a small `context_window`) that quietly degrade behavior. The exact default values can vary between LlamaIndex versions, so confirm them against your installed version.
>
> ```python llamaindex_tokios.py theme={"theme":{"light":"github-light","dark":"css-variables"}}
> from llama_index.llms.openai_like import OpenAILike
>
> llm = OpenAILike(
>     api_base="https://api.tokios.com/v1",
>     api_key="sk-tok-YOUR_KEY",
>     model="gemma-tunnel",
>     is_chat_model=True,
>     is_function_calling_model=True,
>     context_window=8192,
> )
>
> print(llm.complete("Summarize this code: print('hello')"))
> ```

> [tab] Vercel AI SDK
>
> The `@ai-sdk/openai-compatible` package's `createOpenAICompatible` builds a provider from any OpenAI-compatible base URL, then you call it like the built-in OpenAI provider:
>
> ```typescript vercel-ai-tokios.ts theme={"theme":{"light":"github-light","dark":"css-variables"}}
> import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
> import { generateText } from 'ai';
>
> const tokios = createOpenAICompatible({
>   name: 'tokios',
>   baseURL: 'https://api.tokios.com/v1',
>   apiKey: process.env.TOKIOS_API_KEY,
> });
>
> const { text } = await generateText({
>   model: tokios('gemma-tunnel'),
>   prompt: "Summarize this code: print('hello')",
> });
>
> console.log(text);
> ```
>
> Set `TOKIOS_API_KEY` in your environment rather than hardcoding the `sk-tok-…` key.

> [tab] Pydantic AI
>
> Pydantic AI's `OpenAIProvider` accepts a custom `base_url`, and `OpenAIChatModel` takes that provider alongside your deployment name:
>
> ```python pydantic_ai_tokios.py theme={"theme":{"light":"github-light","dark":"css-variables"}}
> from pydantic_ai.models.openai import OpenAIChatModel
> from pydantic_ai.providers.openai import OpenAIProvider
>
> model = OpenAIChatModel(
>     "gemma-tunnel",
>     provider=OpenAIProvider(
>         base_url="https://api.tokios.com/v1",
>         api_key="sk-tok-YOUR_KEY",
>     ),
> )
> ```

> [!NOTE]
>
> Across all four frameworks, `model` is always the Tokios **deployment** name you registered in the [Models](https://tokios.com/console/models) tab — never the local model id your backend serves — and the base URL must include `/v1`. Omitting `/v1` is the most common misconfiguration when adapting a snippet from a different OpenAI-compatible provider.

## FAQ

> [accordion] Why does LlamaIndex need is\_chat\_model and is\_function\_calling\_model set explicitly?
>
> `OpenAILike` is a generic wrapper for OpenAI-compatible servers that don't necessarily behave like OpenAI's own chat models, so it doesn't assume chat or function-calling support out of the box. Confirm the exact default values against your installed LlamaIndex version, and set both explicitly to match what your underlying deployment actually supports.

> [accordion] Do I still need a dummy API key value?
>
> No. Frameworks built against local, keyless servers often expect a placeholder string because the field is required but unchecked. Tokios keys are real `sk-tok-…` credentials — created, scoped, and revoked from the [Keys](https://tokios.com/console/keys) tab — so the API key field does actual authentication instead of satisfying a client-side check.

> [accordion] Does streaming work in these frameworks?
>
> Yes. Tokios returns standard OpenAI-format SSE when a request sets `"stream": true`, so each framework's native streaming support (for example, LangChain's `.stream()` or the Vercel AI SDK's `streamText`) works the same as it would against OpenAI directly.

## Now call it from anywhere

Your backend still runs on localhost — the connector only dials out, and nothing you configured above changes that. What changes is who can reach it: once a teammate or a deployed agent has a scoped `sk-tok-…` key, the same LangChain, LlamaIndex, Vercel AI SDK, or Pydantic AI code that worked against your machine works from theirs.

> [card] [Compatible clients](https://docs.tokios.com/guides/compatible-clients.md)
>
> See the full list of OpenAI- and Anthropic-compatible clients Tokios works with.

> [card] [API keys](https://docs.tokios.com/setup/api-keys.md)
>
> Create a separate scoped key per teammate or agent.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.