LLM Benchmarks Explained: MMLU, HumanEval, GPQA, and Beyond

The Major Knowledge and Reasoning Benchmarks

MMLU (Massive Multitask Language Understanding) is the most cited general knowledge benchmark. It consists of approximately 15,000 multiple-choice questions across 57 subjects, from elementary math to professional law, professional medicine, and abstract algebra. The score represents the percentage of questions answered correctly. MMLU measures breadth of knowledge and the ability to select the right answer from four choices, which correlates with but is not the same as the ability to generate a correct answer from scratch. As of mid-2026, frontier models score 88% to 92% on MMLU, which means the benchmark is approaching saturation and can no longer distinguish meaningfully between the top models. MMLU-Pro, a harder variant with 10 answer choices and more challenging questions, has become the replacement for differentiating frontier models.

GPQA (Graduate-Level Google-Proof Q&A) tests expert-level reasoning on questions written by PhD researchers in physics, biology, and chemistry. The questions are designed so that a non-expert with internet access cannot answer them correctly, which means they test deep domain reasoning rather than surface-level recall. GPQA Diamond, the hardest subset, has scores ranging from 50% to 75% for frontier models, making it one of the few benchmarks that still separates the top tier. It is the best available benchmark for assessing a model's ability to reason about complex scientific problems, though its narrow domain coverage (hard science only) limits generalization.

ARC (AI2 Reasoning Challenge) consists of grade-school science questions, split into Easy and Challenge sets. The Challenge set requires multi-step reasoning and background knowledge that is not stated in the question. Frontier models have nearly saturated ARC Challenge (scoring above 95%), so it is more useful for evaluating smaller or open-weight models where there is still spread in the scores.

HellaSwag tests commonsense reasoning by asking the model to complete a scenario with the most plausible continuation. Like ARC, it is mostly saturated for frontier models but remains useful for evaluating smaller models and fine-tuned variants.

Coding Benchmarks

HumanEval is the original coding benchmark, consisting of 164 Python programming problems where the model generates a function body and the output is tested against unit tests. The metric is pass@k: the fraction of problems where at least one of k generated solutions passes all tests. HumanEval is simple enough that frontier models score above 90% on pass@1, so it no longer differentiates the top tier. HumanEval+ adds more rigorous test cases to catch solutions that pass the original tests but are subtly incorrect.

SWE-bench is the benchmark that matters most for real-world coding ability. It consists of actual GitHub issues from popular Python repositories, and the model must generate a patch that resolves the issue and passes the existing test suite. SWE-bench Verified, a curated subset of 500 problems with confirmed solvability, is the standard reference. As of mid-2026, the best agent systems resolve 60% to 75% of SWE-bench Verified problems, which represents a dramatic improvement from 2024 but leaves significant room for improvement. SWE-bench is uniquely valuable because it tests the full software engineering loop, understanding a codebase, diagnosing the problem, and writing a correct fix, rather than isolated function generation.

LiveCodeBench addresses the contamination problem by using only programming competition problems published after the model's training cutoff. This guarantees the model has not seen the solutions during training, which makes the scores more meaningful as a measure of genuine reasoning ability rather than memorization. It includes problems of varying difficulty from platforms like LeetCode and Codeforces.

Instruction Following and Chat Benchmarks

MT-Bench uses a two-turn conversation format with an LLM judge (GPT-4) scoring the responses on a 1-10 scale across eight categories including writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities. The two-turn structure tests whether the model can build on a previous response, which matters for conversational applications. MT-Bench scores range from 7 to 9.5 for current models, with the top tier clustered tightly enough that the differences are within the noise of the judge.

Chatbot Arena (LMSYS) is not a traditional benchmark but a live preference-ranking system where users interact with two anonymous models side by side and vote for the better response. The accumulated votes produce an Elo rating for each model. Arena rankings are the closest available proxy for "which model do humans actually prefer in open-ended conversation," and they frequently disagree with benchmark scores, because human preference includes factors like tone, formatting, and personality that benchmarks do not measure. The limitation is that Arena reflects the preferences of its self-selected user base, which skews toward English-speaking technical users asking relatively simple questions, so it may not generalize to your specific user population.

IFEval (Instruction Following Evaluation) tests whether models follow specific formatting and content constraints in their outputs, such as "write exactly 200 words," "include the word 'quantum' at least 3 times," or "respond in all lowercase." It is a targeted benchmark for instruction adherence, which matters for applications where the model must follow structured output requirements precisely.

Agentic and Tool-Use Benchmarks

The newest category of benchmarks evaluates models as agents that use tools, browse the web, and complete multi-step tasks. GAIA tests general AI assistants on tasks that require multiple steps, tool use, and web interaction, with difficulty levels from simple fact-finding to complex research. TAU-bench evaluates tool-augmented understanding, measuring whether models can correctly identify which tool to call, with what arguments, in what sequence, to complete a task. WebArena and VisualWebArena test web navigation in realistic simulated environments.

These benchmarks are the most relevant for teams building agent-based applications, because they measure the skills that agents actually need: planning, tool selection, error recovery, and multi-step execution. However, they are also the most sensitive to the agent framework and scaffolding around the model, which means the benchmark score reflects the system rather than the model alone. The same model can score dramatically differently depending on the agent framework, the tools provided, and the prompting strategy.

Why Leaderboard Rankings Are Misleading

Benchmark leaderboards present a single ranking that implies a total ordering of model quality, but this ordering is a fiction. Models have different strengths across domains, and the ranking changes depending on which benchmarks you include and how you weight them. A model that leads on coding benchmarks may trail on reasoning benchmarks, and vice versa. A model that scores highest on English benchmarks may be mediocre on multilingual ones. The leaderboard ranking is an editorial decision about what matters, not an objective fact about model quality.

Contamination is the second reason to distrust leaderboards. Models may have been trained on benchmark data, either intentionally or through the inclusion of web-scraped text that happens to include published benchmark questions and answers. When a model has seen the test, its score reflects memorization rather than capability, and it will not generalize to novel inputs. LiveCodeBench and GPQA were designed specifically to address contamination by using problems that postdate training cutoffs or that were never published online, but most older benchmarks have unknown contamination levels.

The third problem is overfitting to the benchmark format. MMLU is multiple-choice, so a model that is excellent at selecting from four options may not be excellent at generating the answer from scratch. HumanEval tests function-level coding, so a model that excels there may struggle with file-level or repository-level changes. Every benchmark has a format, and every format introduces a gap between the benchmark score and real-world performance on tasks that do not match the format.

How to Use Benchmarks Correctly

Benchmarks are most useful as a first filter, not a final decision. When selecting a model for your application, start by identifying which benchmark is closest to your actual use case: coding benchmarks for coding applications, reasoning benchmarks for analysis tasks, Arena rankings for conversational applications. Eliminate models that score poorly on the relevant benchmark, because a model that cannot pass a standardized test in your domain is unlikely to perform well on your specific task. Then evaluate the remaining candidates on your own data with your own metrics, because the benchmark tells you the model has the capability but not that it applies that capability to your particular inputs, prompts, and quality standards.

Custom evaluation on your own task-specific dataset is always more informative than any public benchmark, which is why building an evaluation dataset is the highest-leverage investment a team can make. A 200-example dataset drawn from your actual user queries, scored by your own criteria, will tell you more about model fit than any combination of public benchmarks. Benchmarks help you decide which models to try. Your own evaluation tells you which one to ship.