> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# What are evals? A practical guide to LLM benchmarks

> Understand MMLU, HumanEval, MT-Bench, and other benchmarks. Know which scores matter for your use case before choosing a local model.

Parameter count alone doesn't tell you whether a model is good. A 7B model trained on the right data can outperform a sloppy 13B model at coding, and a well-tuned 8B model can beat a 70B model at following instructions. Without a way to measure capability, you're guessing.

Evaluation benchmarks — "evals" — are standardized tests that give you comparable numbers across models. They're imperfect, sometimes misleading, and occasionally gamed, but they're the best tool available for comparing models before you commit hardware to one.

This page explains what the major benchmarks measure, how to read their scores, and which ones actually matter for your use case.

## Why evals exist

You can't judge a model by its parameter count, training data size, or architecture alone. Those numbers describe *capacity* — what the model could theoretically do — not *capability*, what it actually does when you give it a prompt.

Benchmarks exist to bridge that gap. A good benchmark:

* **Standardizes comparison** — every model answers the same questions under the same conditions.
* **Reveals strengths and weaknesses** — a model that scores well on math might be mediocre at code.
* **Tracks progress** — you can see whether a new generation of models actually improved or just got bigger.

But every benchmark has blindspots. A high MMLU score doesn't guarantee the model writes good emails. A low HumanEval score doesn't mean the model can't help you debug. Benchmarks are a starting point, not a verdict.

## The benchmark landscape

Here are the major benchmarks you'll encounter on model cards and leaderboards:

| Benchmark | What it tests | Score range | Who publishes | Limitations |
| - | - | - | - | - |
| **MMLU** | Knowledge across 57 academic subjects | 0–100% | Hendrycks et al. | Multiple-choice only; doesn't test open-ended reasoning |
| **HumanEval** | Code generation from docstrings | 0–100% pass\@1 | OpenAI | 164 problems; narrow coverage of programming tasks |
| **MBPP** | Python function generation from descriptions | 0–100% pass\@1 | Google | \~1000 problems; mostly simple algorithmic tasks |
| **MT-Bench** | Multi-turn conversational quality | 1–10 scale | LMSYS | Human-rated but limited to 80 questions; subjective |
| **GSM8K** | Grade-school math word problems | 0–100% | OpenAI | Requires multi-step reasoning but stays at elementary level |
| **MATH** | Competition-level math (AMC, AIME) | 0–100% | Hendrycks et al. | Harder than GSM8K but still narrow in scope |
| **ARC-Challenge** | Science reasoning (elementary/middle school) | 0–100% | Allen Institute | Questions can be solved with common sense, not deep science |
| **HellaSwag** | Commonsense reasoning (sentence completion) | 0–100% | Zellers et al. | Adversarially filtered but still multiple-choice |
| **TruthfulQA** | Factual accuracy and resistance to hallucination | 0–100% | Lin et al. | Multiple-choice format; doesn't capture nuanced misinformation |
| **IFEval** | Instruction following (format, constraints) | 0–100% | Google | Tests rigid constraint-following, not creative instruction interpretation |
| **BigBench Hard** | 23 challenging reasoning tasks | 0–100% | Google Brain | Subset of BIG-bench; tasks are handpicked for difficulty |
| **LMSYS Arena** | Crowdsourced human preference | ELO rating | LMSYS | Slow to update; reflects popular taste, not task-specific quality |
| **GPQA** | Graduate-level physics, chemistry, biology | 0–100% | NYU | Domain experts struggle too; very hard, good for separating frontier models |
| **LiveBench** | Monthly-updated tasks across 6 categories | 0–100% | LiveBench AI | Designed to resist contamination; relatively new, smaller track record |
| **SWE-Bench** | Real-world GitHub issues and pull requests | 0–100% resolved | Princeton | End-to-end software engineering; very hard, models score in single digits |

> [accordion] MMLU in detail
>
> MMLU (Massive Multitask Language Understanding) is the most widely cited benchmark. It covers 57 subjects from STEM, humanities, and social sciences at the high-school to professional level. Each question is multiple-choice with 4 options. A score of 70 means the model gets 70% of these questions right. MMLU is useful as a broad capability signal but doesn't test open-ended generation, creativity, or instruction following. Most model cards report the 5-shot score.

> [accordion] HumanEval and MBPP
>
> These 2 benchmarks measure code generation. HumanEval has 164 hand-written Python problems where the model generates a function from a docstring, then tests are run to check correctness. MBPP is larger (\~1000 problems) but covers simpler algorithmic tasks. Both report pass\@1 — the percentage of problems solved correctly on the first attempt. They're good for comparing coding capability but cover a narrow slice of real-world programming.

> [accordion] MT-Bench and Chatbot Arena
>
> MT-Bench evaluates conversational quality through 80 multi-turn prompts rated by GPT-4 or human judges on a 1–10 scale. Chatbot Arena (LMSYS) is different: it pits 2 anonymous models against each other and lets users vote, producing an ELO ranking like chess ratings. Arena scores reflect real-world user preference but update slowly and favor models that produce polished, confident-sounding answers over models that are correct but terse.

> [accordion] SWE-Bench
>
> SWE-Bench tests whether a model can solve real GitHub issues end-to-end: read the issue, find the bug in a large codebase, and generate a correct pull request. Top models resolve 30–50% of issues on the verified subset. This is the closest benchmark to actual software engineering work, but it's extremely difficult and results vary heavily by language and framework.

> [accordion] LiveBench
>
> LiveBench refreshes its test set monthly using new data, which makes it resistant to benchmark contamination — models can't have memorized answers they haven't seen during training. It covers math, coding, reasoning, language, data analysis, and instruction following. It's newer than MMLU or HumanEval but increasingly cited as a more trustworthy signal.

## How to read a benchmark table

Scores look authoritative, but they need context to be useful.

### What the numbers mean

Most benchmarks report a score from 0 to 100, which is the percentage of questions the model answered correctly. An MMLU score of 70 means the model gets 70% of college-level multiple-choice questions right. ELO ratings (used by Chatbot Arena) are different — they're relative rankings where a 100-point gap means the higher-rated model wins about 64% of head-to-head matchups.

### Margin of error

A 1–2 point difference on any single benchmark is noise. Scoring methodology, prompt formatting, and even random seed variation can shift scores by that much. If Model A scores 72 and Model B scores 74 on MMLU, treat them as equivalent and look at a benchmark that tests the specific capability you care about.

### Contamination

Benchmark contamination happens when test questions leak into a model's training data. The model scores artificially high because it has already seen the answers. This is a growing problem as benchmarks become widely published. LiveBench and some newer benchmarks combat this by refreshing their test sets regularly.

> [!NOTE]
>
> When a model's benchmark score seems suspiciously high compared to its real-world performance, contamination is a likely explanation. Check whether the benchmark was published before the model's training cutoff date.

### Cherry-picking

Never judge a model by a single benchmark. Publishers sometimes highlight the 1 score where their model looks best while burying weaker results. Look for a spread of benchmarks that cover different capabilities — knowledge, code, math, instruction following — before drawing conclusions.

## Which benchmarks matter for your use case

Not every benchmark matters for every task. Focus on the ones that test what you actually need:

> [card] Coding assistant
>
> **Look at:** HumanEval, MBPP, SWE-Bench.
> These directly measure code generation and software engineering. Ignore MMLU for this decision — a model can score high on trivia and still write bad code.

> [card] Chatbot and general conversation
>
> **Look at:** MT-Bench, Chatbot Arena ELO, IFEval.
> Conversational quality is subjective, so human-rated benchmarks matter most. IFEval adds a measure of whether the model follows formatting and constraint instructions reliably.

> [card] Reasoning and analysis
>
> **Look at:** GSM8K, MATH, GPQA, ARC-Challenge, BigBench Hard.
> These test multi-step logic at different difficulty levels. GPQA is the hardest — useful for separating frontier models from merely good ones.

> [card] General assistant
>
> **Look at:** MMLU + MT-Bench + IFEval as a composite.
> No single number captures "general assistant" quality. Check all 3 to get a picture of knowledge breadth, conversational skill, and instruction compliance.

> [!TIP]
>
> When comparing 2 similar models, skip the averages and look at the specific benchmark that matches your task. A 7B coding model and a 7B chat model might have similar MMLU scores but very different HumanEval results.

## Benchmark scores at different quantizations

Benchmark scores are usually reported at full precision (FP16 or BF16). When you run a quantized model locally, expect some score degradation.

### How much does quantization cost?

The impact depends on the benchmark, the model size, and the quantization level:

| Quant level | Approximate MMLU impact | Notes |
| - | - | - |
| `Q8_0` | 0–1 point drop | Near-lossless for most tasks |
| `Q6_K` | 0–1 point drop | Hard to distinguish from full precision |
| `Q5_K_M` | 1–2 point drop | Still very close to original quality |
| `Q4_K_M` | 1–3 point drop | The sweet spot most users target |
| `Q3_K_M` | 3–6 point drop | Noticeable degradation, especially on reasoning |
| `Q2_K` | 8–15 point drop | Severe quality loss; not recommended |

### When quantization hurts more

* **Small models lose more than large models.** A 7B model quantized to `Q4_K_M` drops more than a 70B model at the same quant level, because the smaller model has less redundancy to absorb rounding errors.
* **Precise tasks suffer more.** Math and code generation are more sensitive to weight precision than general chat. If you're running a coding assistant, consider `Q5_K_M` or higher.
* **I-quants can help at low bit-widths.** Imatrix quantization (`IQ4_XS`, `IQ3_XXS`) preserves critical weights more carefully and may recover 1–2 points compared to standard K-quants at the same size.

> [!NOTE]
>
> A 70B model at `Q4_K_M` almost always outperforms a 7B model at full precision on most benchmarks. Model size matters more than quantization level when the size gap is large.

## Running your own evals

Published benchmarks are a starting point. If you need to know how a model performs on *your* tasks with *your* setup, run your own evaluation.

### When to run your own

* You have a domain-specific use case (legal, medical, finance) that standard benchmarks don't cover.
* You're comparing 2 models that score similarly on public benchmarks.
* You're testing a specific quantization or fine-tune and want to measure the real impact.
* You need to validate that a model handles your edge cases before deploying it.

### Tools for running evals

**lm-evaluation-harness** (by EleutherAI) is the standard open-source tool. It supports 60+ academic benchmarks out of the box and runs against any model served through `llama.cpp`, vLLM, or an OpenAI-compatible API:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
pip install lm-eval
lm_eval --model local --model_args pretrained=/path/to/model --tasks mmlu,hellaswag --batch_size auto
```

**OpenAI Evals** and **HELM** (Stanford) are alternatives with different task libraries and reporting formats.

### A simple custom eval

You don't need a framework to test domain-specific quality:

1. Write 50 questions from your actual use case.
2. Send each question to the model and record the response.
3. Score each response on a 1–5 scale (or pass/fail for factual questions).
4. Repeat for each model you're comparing.

50 questions is enough to spot large differences. Use the same questions for every model, and score blind (hide which model produced which response) to avoid bias.

### When to trust published scores

Trust published benchmark scores when:

* The model publisher has a track record of honest reporting (Meta, Google, Mistral, Qwen, Anthropic).
* Multiple independent sources confirm similar numbers.
* The benchmark was published after the model's training cutoff (reducing contamination risk).

Run your own eval when:

* You're using a third-party quantization and want to verify it hasn't degraded quality significantly.
* Your task is narrow enough that general benchmarks don't apply.
* You're making a production decision and need confidence beyond a leaderboard number.

## Red flags in benchmark reporting

Not every benchmark claim is trustworthy. Watch for these patterns:

* **No quantization disclosed.** "Our model scores 78 on MMLU" — at what precision? If they don't say, they might be reporting full-precision scores for a model most users will run quantized.
* **Only reporting the best benchmark.** A model page that highlights 1 score and omits others is cherry-picking. Look for a full benchmark table.
* **Comparing different model sizes.** "Our 7B model beats a 70B model on X" — yes, on a narrow benchmark that favors the 7B's training data. This isn't a fair comparison.
* **Contaminated benchmarks without disclosure.** If a model's training data likely included the benchmark's test set and the publisher doesn't address it, treat the score skeptically.
* **"Beats GPT-4 on..."** Almost always on a narrow, favorable benchmark. GPT-4 is a large proprietary model — beating it on 1 specific task doesn't mean the model is generally better.
* **No methodology described.** If the score doesn't include details like shot count, prompt format, and scoring method, you can't compare it to other published numbers.

## Where to find benchmark data

| Source | URL | What it offers |
| - | - | - |
| **Open LLM Leaderboard** (HuggingFace) | [huggingface.co/spaces/open-llm-leaderboard](https://huggingface.co/spaces/open-llm-leaderboard) | Standardized evals across many open models; updated regularly |
| **LMSYS Chatbot Arena** | [chat.lmsys.org](https://chat.lmsys.org) | ELO rankings from crowdsourced human preferences |
| **LiveBench** | [livebench.ai](https://livebench.ai) | Monthly-refreshed, contamination-resistant benchmarks |
| **Model papers and cards** | HuggingFace model repos, arXiv | Publisher-reported scores; usually the most detailed |
| **lm-evaluation-harness results** | GitHub repos, community posts | Independent evaluations run by the community |

> [!TIP]
>
> Cross-reference at least 2 sources before trusting a benchmark number. If the Open LLM Leaderboard and the model's own paper report different scores, investigate why — it's usually a difference in evaluation settings (shot count, prompt format, or quantization).

## Go deeper

> [card] [Read a model name](https://docs.tokios.com/guides/read-a-model-name.md)
>
> Decode model names, quantization labels, and architecture suffixes before you look at benchmarks.

> [card] [Read a model card](https://docs.tokios.com/concepts/read-a-model-card.md)
>
> Find the parameters, architecture, context length, and license that benchmarks don't cover.

> [card] [Quantization](https://docs.tokios.com/concepts/quantization.md)
>
> Understand how quantization affects the benchmark scores you just learned to read.

> [card] [Which local model for which task?](https://docs.tokios.com/guides/model-by-task.md)
>
> Use benchmark insights to pick a model matched to your specific use case.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.