AI benchmarks directory: 481 LLM and agent benchmarks by task
481 benchmarks used to evaluate language models and AI agents, grouped by what they test, with a one-line description, size and a link to the paper that introduced each one. Free to download and cite.
481
benchmarks listed
Across 26 task areas.
Link63%
of listed benchmarks were introduced in 2025 or 2026
253 of the 399 benchmarks with a known year.
Link101
agent benchmarks
Tool use, computer use and web, deep research, and AI research agents combined.
LinkAgents and tool use
largest task area
42 benchmarks.
Link26
task areas
From coding and math to law, medicine, finance, speech and cybersecurity.
Link
Benchmarks are how the AI field keeps score, and there are now far too many to track by memory. This directory lists 481 benchmarks used to evaluate large language models and AI agents, grouped into 26 task areas, with what each one measures, its size where stated, the year it was introduced and a link to the paper or announcement that introduced it.
The list includes the standard references (MMLU, GPQA, HumanEval, SWE-bench, MATH, GAIA and others) and hundreds of newer, narrower benchmarks for agents, tool use, long context, law, medicine, finance and safety.
Search by name, filter by task area or year, sort any column, or download everything as CSV or JSON. The data is free to reuse under CC BY 4.0 with a link back to this page.
All benchmarks
| Benchmark | Task area | What it measures | Size | Year | Paper |
|---|---|---|---|---|---|
| ACEBench | Agents and tool use | Evaluates LLM tool usage and function-calling capabilities across Normal, Special, and Agent data categories using accuracy-based metrics and process/end-to-end metrics for agent interactions | Not stated | Not stated | doi.org |
| AgentBench | Agents and tool use | Multi-dimensional benchmark to assess LLM-as-Agent reasoning and decision-making abilities across 8 environments; evaluates performance of API-based and OSS LLMs in interactive environments | 8 environments (distinct environments) | 2023 | arxiv.org |
| Berkeley Function Calling Leaderboard | Agents and tool use | Evaluates function calling capability of LLMs across serial/parallel calls, AST evaluation, memory, decision making, long-horizon reasoning in agentic settings | Not stated | 2025 | dl.acm.org |
| ComplexMCP | Agents and tool use | Evaluates LLM agents on dynamic, interdependent tool use across 7 stateful sandboxes with seed-driven environment dynamics; reports success rate and granular bottlenecks like tool retrieval saturation, over-confidence, and strategic defeatism | >300 tools derived from 7 sandboxes | 2026 | arxiv.org |
| CompToolBench | Agents and tool use | Evaluation of LLM tool-use across four composition levels using 200 tasks and 106 real tools; identifies Selection Gap; compares cloud vs local models on structured tool-use tasks | 200 tasks; 106 tools | 2026 | doi.org |
| DICE-BENCH | Agents and tool use | Measures the ability of LLMs to perform function-calling in multi-round, multi-party dialogues using DICE-SCORE and a practical function-calling dataset built from a tool graph and multi-agent dialogues | 1607 high-DICE-SCORE instances in the final dataset | Not stated | doi.org |
| DynamicMCPBench | Agents and tool use | Scores an agent on whether it reproduces the effects observed in trajectories, using path-agnostic effect checkpoints and a pass^3 metric (tasks considered solved only if all three independent attempts succeed) | 24 models; 121 servers; 750 tasks; 15 task categories (50 per category) | 2026 | arxiv.org |
| ETOM | Agents and tool use | Evaluates end-to-end multi-hop tool orchestration by LLM agents within an MCP ecosystem using equal function sets; uses F1 score and robustness metrics to assess tool-use and cross-server orchestration capabilities across five curriculum levels | 5 levels in curriculum; experiments on 491 servers and 2375 tools (as per highli | 2025 | arxiv.org |
| FinMCP-Bench | Agents and tool use | Evaluates LLMs on tool invocation accuracy and reasoning in real-world financial tasks under the Model Context Protocol | 613 samples across 10 main scenarios and 33 sub-scenarios; 65 real financial MCP | 2026 | doi.org |
| FuncBenchGen | Agents and tool use | Evaluates tool-augmented LLMs (TaLMs) and multi-step function calling under contamination-free, controllable difficulty conditions; analyzes performance and failure modes across models; assesses impact of dependency depth, graph size, distractor functions, and state tracking; proposes mitigation by restating prior variable values | 7 (models evaluated) + multiple task configurations; not specifying exact task c | 2025 | arxiv.org |
| FunctionChat-Bench | Agents and tool use | Four output types in tool-use dialogs (Tool Call, Answer Completion, Slot Question, Relevance Detection) and overall generative performance of language models with function calling in single-turn and multi-turn contexts | 700 evaluation items (single call + dialog dataset combined); 8 models; 45 Korea | 2024 | arxiv.org |
| GAIA | Agents and tool use | Assesses a broad set of fundamental abilities for AI assistants (reasoning, multi-modality handling, web browsing, and tool-use proficiency) through 466 questions with automated, quasi-exact evaluation | 466 questions (with 300 answers retained for leaderboard) | 2023 | arxiv.org |
| GeoBenchX | Agents and tool use | Evaluates tool-calling capabilities of LLMs on multi-step geospatial tasks using a 23-function tool-calling agent; comparison of multiple commercial LLMs; evaluation framework with LLM-as-Judge | Not stated | 2025 | arxiv.org |
| GTA | Agents and tool use | Evaluates real-world tool-use capabilities of LLMs with real user queries, real deployed tools, and real multimodal inputs across perception, operation, logic, and creativity categories | 229 tasks | 2024 | arxiv.org |
| LiveMCP-101 | Agents and tool use | Evaluates real-world agent capabilities to coordinate multiple MCP tools across challenging queries using a ground-truth execution plan vs. reference trajectories; reports success rates, ablations, error analysis, and token efficiency | 101 tasks (LiveMCP-101); plus LiveMCP-101-a variant with 101 tasks | 2025 | arxiv.org |
| LiveMCPBench | Agents and tool use | Task success rate across multiple LLMs and analysis of tool retrieval and composition performance | 95 real-world daily tasks; 70 servers with 527 tools; 12 LLMs | Not stated | doi.org |
| LoCoBench-Agent | Agents and tool use | Evaluates LLM agents in long-context software engineering across multi-turn conversations, tool usage efficiency, error recovery, and architectural consistency using 9 metrics across comprehension and efficiency dimensions | undefined | 2025 | arxiv.org |
| M^3-Bench | Agents and tool use | Evaluates multimodal, multi-hop, multi-threaded tool-using MLLM agents with auditable correspondences, semantic fidelity, and workflow consistency; reports end-task completion and grounding via an LLM judge ensemble | 28 servers with 231 tools | 2025 | arxiv.org |
| MCP Security Bench | Agents and tool use | Robustness and security evaluation of MCP-enabled LLM agents via Net Resilient Performance (NRP) across 12 attacks, 9 agents, 10 domains, 405 tools, 2000 attack instances | approximately 405 tools × 9 agents × 10 domains with 2000 attack instances | 2025 | arxiv.org |
| MCP-Atlas | Agents and tool use | Measures tool-use competency of LLM agents against production MCP servers using claim-level scoring and a diagnostic taxonomy across 36 real MCP servers and 220 tools; evaluates 20 frontier models with pass rates up to 82.2% under specific task-level conditions | 1000 natural-language tasks; 36 real MCP servers; 220 tools; 20 models; 500-task | 2026 | arxiv.org |
| MCP-Bench | Agents and tool use | Evaluates LLMs on realistic, multi-step tasks requiring tool use, cross-tool coordination, precise parameter control, planning/reasoning, tool retrieval from fuzzy instructions, multi-hop trajectories, grounding in tool outputs, and cross-domain orchestration | Not stated | 2025 | arxiv.org |
| MCP-GRANITE Benchmark | Agents and tool use | Evaluates tool-interface granularity for MCP-based agents using task completion, tool selection F1, argument accuracy, latency, and resource-usage metrics | 81 multi-step scenarios across 9 domains, instantiated at 4 granularity levels, | 2026 | arxiv.org |
| MCP-RADAR | Agents and tool use | Five-dimensional evaluation across answer accuracy, tool selection efficiency, computational resource efficiency, parameter construction accuracy, and execution speed to assess LLM tool use within the MCP framework | 300 tasks across three core domains (100 tasks per domain) | 2025 | arxiv.org |
| MCP-Universe | Agents and tool use | Execution-based evaluators including format evaluators, static evaluators, dynamic evaluators; real-time ground truth for temporally sensitive tasks; success rates of LLMs | 231 tasks across 6 domains and 11 MCP servers | 2025 | arxiv.org |
| mcp-vs-cli-bench | Agents and tool use | Completion is scored over four independently checkable conditions by reading repository state back through the API after the run; every request crosses a local proxy that records input, output, and cached tokens, the tools offered in each request, and the tools actually called | 1 | Not stated | doi.org |
| MCPAgentBench | Agents and tool use | Metrics for task completion, execution efficiency, including Task Finish Score (TFS), Task Efficiency Finish Score (TEFS), Time Efficiency, and Token Efficiency; evaluates tool-use and multi-step/parallel MCP tool invocations in a sandbox with distractors | 180 task instances (from 841 tasks and over 20000 MCP Tools; 180 high-quality ta | 2025 | arxiv.org |
| MCPEvol-Bench | Agents and tool use | Evaluates LLM agent performance under dynamic evolutions of MCP tool interfaces, using 11 mutation operators across 123 MCP servers and benchmarking 12 LLMs | 1 | 2026 | arxiv.org |
| MCPMark | Agents and tool use | Evaluates MCP use realism and comprehensiveness for LLMs via CRUD-heavy tasks in a tool-calling loop; reports pass@1 and pass^4, average execution turns, and tool calls per task | 127 tasks | 2025 | arxiv.org |
| MCPToolBench++ | Agents and tool use | Evaluates MCP tool use capabilities of LLMs/AI agents across single-step and multi-step tool calls on a large number of MCP servers | over 4k MCP servers from more than 40 categories | 2025 | arxiv.org |
| Multi-Mission Tool Bench | Agents and tool use | Evaluates agent robustness in related and dynamic multi-mission scenarios using a novel evaluation framework (dependency analysis, decision tree construction, path validation) and metrics such as success rate and optimality rate; analyzes agent decisions across multiple interrelated missions | 1024 test entries, each with 1, 4 missions | 2025 | arxiv.org |
| MultiCAT-Bench | Agents and tool use | Recall and Precision metrics for tool use and function calling across ten difficulty categories; analysis of impact factors (number of calls, parameter optionality, number of function parameters); average accuracy by model | -practically 3 789 test cases across ten categories; evaluation of ten LLMs from | Not stated | doi.org |
| NESTFUL | Agents and tool use | Metrics include F1 score for user query evaluation, full sequence match accuracy, and win rate in evaluating nested sequences of API calls | 1800+ (more than 1,800 instances) | 2025 | doi.org |
| OpaqueToolsBench | Agents and tool use | Measures efficiency and performance of LLM agents in opaque tool settings, including improvement in tool documentation and task success across environments, with token-efficiency and cross-trajectory learning | Not stated | 2026 | dl.acm.org |
| OpenMCP | Agents and tool use | Metrics for task completion, efficiency, and agent issues in MCP-enabled computer-use agents | 1 | 2026 | doi.org |
| ReflecTool-Bench | Agents and tool use | Fine-grained metrics for error detection, error classification, correction accuracy, and explanation quality in reflective tool use across critique and self-reflection tasks | 968 dialogues, 88 APIs, 10 domains | Not stated | doi.org |
| SEAL | Agents and tool use | Comprehensive evaluation pipeline for API retrieval, API calls (tool usage), and final responses with deterministic evaluation via a GPT-4-powered API simulator and caching; standard benchmarks with Recall@K, MRR, tool-call accuracy, final response pass rate | Not stated | 2024 | arxiv.org |
| tau-bench | Agents and tool use | Tool-using agents completing tasks with a simulated user under domain policies | Not stated | 2024 | arxiv.org |
| Terminal-Bench | Agents and tool use | Assesses agent performance on hard, realistic terminal tasks; frontier models and agents scoring <65% with error analysis and evaluation harness | 89 tasks | 2026 | arxiv.org |
| The Tool Decathlon | Agents and tool use | Execution-based evaluation of language agents across diverse Apps and tools with verifiable task success and metrics such as tool-calls and turns | 108 tasks; 32 software applications and 604 tools; ~20 turns per task | 2025 | arxiv.org |
| TheMCPCompany | Agents and tool use | Evaluates tool-calling agents on tasks involving interacting with real-world services using MCP tools; analyzes performance and costs with tool retrieval vs ground-truth tools | Not stated | 2025 | arxiv.org |
| TRAJECT-Bench | Agents and tool use | Evaluates trajectory-aware tool-use capability of LLM agents, including trajectory-level diagnostics and final-answer accuracy across diverse tool-use tasks | Not stated | 2025 | arxiv.org |
| UltraTool | Agents and tool use | Comprehensive evaluation of LLMs’ tool utilization across planning, tool creation, and tool usage in real-world complex scenarios | 5824 examples, 22 domains, 2032 tools | 2024 | doi.org |
| AgentOps-Bench | AI research agents | Evaluates tool-using LLM agents across six axes: completion, cost, efficiency, reliability, recovery, safety, using an instrumented tool server, failure-mode injectors, and a 15-entry indirect prompt-injection catalogue; includes 7,200-run pilot across 100 tasks under 3 conditions and 3 runs with 8 agents | 100 tasks x 5 domains x 3 conditions x 3 runs x 8 agents (pilot) plus framework | 2026 | doi.org |
| CSR-Bench | AI research agents | Completion rate: ratio of successfully executed commands to total commands executed; evaluates accuracy, efficiency, and deployment script quality of LLM agents on code deployment tasks | Not stated | 2025 | arxiv.org |
| ELAIPBench | AI research agents | Multiple-choice question accuracy and reasoning ability across 403 questions from 137 AI papers, with difficulty levels and evaluation of LLMs vs human performance | 403 questions, 137 papers | 2025 | arxiv.org |
| FML-bench | AI research agents | Proposes metrics including Exploration Diversity to evaluate ML research agents across 8 tasks and assesses correlation between exploration diversity and performance | 8 tasks | 2025 | arxiv.org |
| FrontierCS 2 | AI research agents | Assesses agent performance on 48 open-ended CS tasks by having agents modify a repository, submit patches, and be scored on development suites across 13 domains; evaluates 12 agents with a FrontierCS Capabilities Index | 48 tasks in 13 domains, plus a private set of more than 100 tasks | 2026 | arxiv.org |
| ML Research Benchmark | AI research agents | Evaluates AI agents on competition-level ML conference tasks reflecting AI research capabilities and development, including planning, model development, pretraining/finetuning, pruning/compression, and multi-step problem solving | 7 tasks | 2024 | arxiv.org |
| ML-Bench | AI research agents | Evaluates LM-to-code generation and autonomous agent execution performance on repository-scale tasks using Pass@k metrics and end-to-end task success rates in Linux sandbox environments | 9641 samples across 18 GitHub repositories | 2023 | arxiv.org |
| ML-Dev-Bench | AI research agents | Evaluation of agentic capabilities across ML development tasks using Calipers framework; metrics include binary task success rate and aggregate success rate across 30 tasks, plus analysis of strengths/limitations of agents | 30 | 2025 | arxiv.org |
| MLAgentBench | AI research agents | Evaluates agents' ability to perform ML experimentation tasks, focusing on task competence (improvement of performance metric) and efficiency (time and number of LM queries) | 13 tasks | 2023 | arxiv.org |
| MLE-bench | AI research agents | Evaluates ML engineering capabilities of AI agents across 75 Kaggle competitions, using pass@k metrics and human baselines; analyzes resource scaling and pre-training contamination | 75 tasks/competitions | 2024 | arxiv.org |
| MLE-Dojo | AI research agents | Evaluates autonomous LLM agents in iterative machine learning engineering workflows using an interactive Gym-style environment, with evaluations across eight frontier LLMs and a continuously updated leaderboard to assess iteration, debugging, and long-horizon solution generation | Over 200 Kaggle competitions; 150 tasks in initial training dataset | Not stated | doi.org |
| MLE-Smith | AI research agents | Assessing scalability, quality, and solvability of automated MLE task generation across a wide range of real-world datasets; evaluation of LLMs on generated MLE tasks and correlation with human-designed tasks | 606 tasks generated from 224 real-world datasets | 2025 | arxiv.org |
| MLGym | AI research agents | Evaluation of AI research task performance of LLM agents across diverse AI research tasks using frontier models; framework to add tasks, generate synthetic data, and evaluate models/agents | 13 tasks in MLGym-bench | 2025 | arxiv.org |
| MLOps-Bench | AI research agents | Measures ability of AI agents to perform repository-grounded MLOps tasks across data pipelines, feature engineering, model training, serving, monitoring, CI/CD and governance; evaluation includes executable lifecycle changes, oracle patches, buggy baselines, fail-to-pass tests, pass-to-pass tests, and provenance metadata; reported success rates across agents | 212 validated tasks from 31 public ML repositories; 636 fail-to-pass tests; 856 | 2026 | arxiv.org |
| PaperBench | AI research agents | Evaluates AI agents' ability to replicate state-of-the-art AI research using a rubric-based, multi-task replication benchmark and an LLM-based judge; reports replication scores and human baselines across ICML 2024 papers | 8,316 tasks | 2025 | arxiv.org |
| RE-Bench | AI research agents | Evaluates frontier AI R&D capabilities of language model agents against human experts using best-of-k scoring across environments and time budgets; compares agent vs human performance on 7 ML research engineering environments with 71 8-hour human attempts across 61 experts | 7 tasks/environments (7 ML research engineering environments); 71 attempts by 61 | 2024 | arxiv.org |
| ResearchCodeBench | AI research agents | Evaluates LLMs' ability to translate cutting-edge ML contributions from top 2024-2025 research papers into executable code across a benchmark of 212 coding challenges (and ResearchCodeBench-HARD with 109 tasks) and reports pass rates and error patterns | 212 tasks (ResearchCodeBench) + 109 tasks (ResearchCodeBench-HARD) | 2026 | dl.acm.org |
| ReX-MLE | AI research agents | End-to-end autonomous agent performance on medical imaging challenges across data preprocessing, model training, and submission under realistic compute and time constraints | 20 | 2025 | arxiv.org |
| TimeSeriesGym | AI research agents | Hybrid evaluation combining quantitative metrics (accuracy, mean absolute error), programmatic analysis, and qualitative evaluation (LLM-as-a-judge) for AI agents on time series ML engineering tasks | 34 challenges across 8 time series problems, more than 15 domains | 2025 | arxiv.org |
| Aider Polyglot | Coding | Evaluates coding model performance across multiple programming languages (C++, Go, Java, JavaScript, Python, Rust) on Exercism problems to differentiate strongest coding models and calibrate scoring across a 5%-50% top-score range | 225 (hard problems) + 697 total problems in Exercism set; 7 models tested | 2024 | aider.chat |
| BDD-Bench | Coding | Evaluates coding agents on executable Gherkin specifications across the application lifecycle, including lifecycle progress, coverage of production code, and failure taxonomy under an agent harness | 111 instances across five Python repositories (lifecycle benchmark) | Not stated | doi.org |
| CODE2BENCH | Coding | Dynamic benchmark construction using Dual Scaling with 100% branch coverage tests and property-based testing to evaluate code-generating LLMs across real-world code corpora and multiple languages | 1 | 2025 | arxiv.org |
| COMPASS | Coding | Multidimensional code generation evaluation across correctness, performance efficiency, and maintainability with human-referenced normalization | 50 programming problems | 2026 | dl.acm.org |
| ComplexCodeEval | Coding | Evaluates large code models on multiple coding tasks: code generation, code completion, API recommendation, and test case generation; assesses impact of context information and time on performance (CodeBLEU, F1, etc.) | 3897 Java samples + 7184 Python samples from GitHub repositories (as described i | Not stated | doi.org |
| CORE | Coding | Evaluates LLMs on fundamental static analysis tasks including data dependency, control dependency, and information flow across programs in C/C++, Java, and Python; analyzes semantic reasoning and multi-step code understanding | 12553 | 2026 | dl.acm.org |
| DSCodeBench | Coding | Pass@1 for LLMs on data science code generation tasks and related evaluation metrics (test cases, code length, etc.) | 1000 tasks | 2026 | doi.org |
| EvoCodeBench | Coding | Evaluation metrics for code generation aligned with real-world repositories, including Pass@k and Recall@k; repository-level code generation task | 275 samples from 25 real-world repositories (EvoCodeBench-2403) | 2024 | arxiv.org |
| HumanEval | Coding | Functional correctness of Python functions generated from docstrings | Not stated | 2021 | arxiv.org |
| LeetCodeDataset | Coding | Evaluates and trains code-generation models with temporal splits, contamination-free evaluation, and efficient supervised fine-tuning (SFT) on LeetCode Python problems | Not stated | 2025 | arxiv.org |
| LiveCodeBench | Coding | Evaluation of LLMs across code-related capabilities (code generation, self-repair, code execution, test output prediction) using contamination-free, live-updated problems | 511 problems collected May 2023, May 2024; 18 base models and 34 instruction-tune | 2024 | arxiv.org |
| LoCoBench | Coding | A 17-metric, 4-dimension evaluation framework for long-context code understanding across 8,000 evaluation scenarios with context lengths from 10K to 1M tokens | 8000 tasks (evaluation scenarios) across 8 long-context task categories | 2025 | arxiv.org |
| MBPP | Coding | Entry-level Python programming problems solved from a short description | Not stated | 2021 | arxiv.org |
| OmniCode | Coding | Evaluates software engineering tasks across bug fixing, test generation, code review fixing, and style fixing using a diverse set of 1794 tasks across Python, Java, and C++; analyzes agent performance (e.g., SWE-Agent reaching up to 20.9% on Java Test Generation with DeepSeek-V3.1) | 1794 tasks; 963 Python tasks, 418 Java tasks, and (remaining in other languages) | 2026 | arxiv.org |
| PolyCodeEval | Coding | Execution-based evaluation protocol across multilingual, multi-granularity code generation tasks; reports correct function/file/repository generation percentages | 2590 tasks | 2026 | arxiv.org |
| PRDBench | Coding | 1,258 evaluation metrics across QA stages and Shell Interaction metrics; evaluation uses PRDJudge fine-tuned on Qwen3-Coder-30B with >90% human alignment in fixed-interface scenarios | 50 tasks (real-world Python projects) across 20 domains; 1,258 evaluation metric | Not stated | doi.org |
| PROBE | Coding | Evaluates code generation along three dimensions: functional correctness, proximity to valid solutions, and code quality; uses diverse workload, prompt templates, and evaluation procedure to assess model performance across multiple languages and prompting strategies | Not stated | 2026 | link.springer.com |
| ProjectEval | Coding | Pass@K type metric (Pass@K) adapted from HumanEval/MBPP-style benchmarks to evaluate code generation in project-level tasks; reported average Pass@K across 284 test cases | 20 tasks with 284 test cases | 2025 | doi.org |
| REAP | Coding | Evaluation of coding agents using production-derived benchmarks (Harvest) with solve rates across frontier models and multi-run stability validation | Not stated | 2026 | dl.acm.org |
| RepoMasterEval | Coding | Evaluates code completion models using real-world repository snippets masked within source files, with mutation testing to assess test adequacy and coverage of model-generated code; reports correlation with real-world performance and variance across models | Not stated | 2025 | dl.acm.org |
| rStar-Coder | Coding | Evaluates code reasoning capabilities of LLMs on a large-scale, verified dataset of competitive programming problems with test cases and long-reasoning solutions; reports pass@k and accuracy improvements across models | 418K code problems + 580K long-reasoning solutions + rich test cases | 2026 | dl.acm.org |
| SWE-bench goes live! | Coding | Evaluates code LLMs and code agents on bug-fixing tasks using SWE-bench-Live with live GitHub issues, automated curation, and executable Docker environments | 1,890 tasks from 223 repositories | 2026 | dl.acm.org |
| SWE-Bench Pro V2 | Coding | Evaluates long-horizon software engineering tasks for coding agents; assesses robustness of benchmarks and agent performance under hardened protocol; evaluates task completion and separation of systems under Hard split | 731 tasks (original), 642 V2 tasks frontier, 51-task Hard split (from highlights | 2026 | arxiv.org |
| SWE-Journey | Coding | Assesses realism of coding assistant evaluation across long-horizon tasks and multi-turn interactions using a weak-to-strong synthesis pipeline and a user-simulation agent; analyzes interaction capabilities (asking right, finding right, fixing right) and success rates across coder vs non-coder user personas | 1 | 2026 | arxiv.org |
| Tests as Prompt | Coding | Evaluates LLMs on test-driven development (TDD) code generation using test cases as prompt and verification across diverse challenges | 1000 challenges across 20 domains | 2025 | arxiv.org |
| Turbulence | Coding | Three complementary scores: accuracy, correctness-potential, and consistent-correctness | Not stated | 2025 | doi.org |
| BEARCUBS | Computer use and web | Evaluates a web agent's ability to search, browse, and identify factual information from the web through live web content and multimodal interactions | 111 questions | 2025 | arxiv.org |
| BrowseComp | Computer use and web | Tests persistent web browsing ability, including searching, verifying factuality, persistence and creativity in finding information; 1,266 questions | 1266 | 2025 | arxiv.org |
| BrowseComp-$V^3$ | Computer use and web | Assesses multimodal browsing agent capability with deep search, multi-modal evidence gathering, subgoal-driven process evaluation, and final-answer accuracy | 300 | 2026 | arxiv.org |
| CAP | Computer use and web | Evaluates browser agents on cross-site, human-like web tasks requiring non-trivial UI interactions and visual understanding; enables fine-grained diagnosis through a decomposition-recomposition pipeline and verifies agent performance using a verifiable agent-as-a-judge framework | 420 tasks across 108 websites and 24 domains | 2026 | arxiv.org |
| CUA-Lite benchmarks | Computer use and web | Provides unified environment interface, unified supervised data format, and model harnesses with verifiable tasks across desktop, browser, and mobile; reports reference scores across integrated benchmarks and RL/SFT improvements | 12 benchmarks, over 30,000 verifiable tasks | 2026 | arxiv.org |
| KNOWS | Computer use and web | Evaluates web agents on synthesizing, organizing, and displaying knowledge across multi-step browser workflows using produced artifacts; includes deterministic checks and LLM judgments for evaluation | Not stated | 2026 | arxiv.org |
| macOSWorld | Computer use and web | Multilingual GUI agent benchmarking across macOS; task success rate (SR), safety benchmarking subset, language-specific performance degradation | 202 tasks across 30 applications (28 macOS-exclusive); 7 categories; 5 languages | 2025 | doi.org |
| MCPWorld | Computer use and web | Task completion accuracy of LLM-powered computer use agents, evaluated on 201 tasks with code-instrumented verification and GPU-accelerated, containerized setup; reports 75.12% task completion accuracy in preliminary experiments | 201 tasks | 2025 | arxiv.org |
| MYPCBENCH | Computer use and web | Tests computer-use agents as personal assistants on a Linux desktop with 17 web apps and 184 tasks; compares models on task completion, rubric-graded evaluation, and trajectory metrics | 184 tasks across 17 applications with a single canonical persona | 2026 | arxiv.org |
| Odysseys | Computer use and web | Rubric-based evaluation with an average of 6.1 graded rubrics per task; Trajectory Efficiency metric (rubric score per step); binary pass/fail is inadequate; evaluation on live Internet with 200 long-horizon web tasks | 200 tasks | 2026 | arxiv.org |
| OSUniverse | Computer use and web | Multimodal GUI-navigation task performance with automated validation and manual scoring; covers increasing complexity from basic clicking to multi-application tasks; reports SOTA less than 50% and automated validation error rate < 2% | Not stated | 2025 | arxiv.org |
| OSWORLD | Computer use and web | Comprehensive evaluation of multimodal agents on real computer tasks across Ubuntu, Windows, and macOS; human performance vs agent performance on 369 tasks; GUI grounding and operational knowledge proficiency; open-ended task execution in real desktop/web apps | 369 | 2024 | dl.acm.org |
| OSWorld 2.0 | Computer use and web | Primary binary-completion metric at 500 steps evaluating agent performance on long-horizon computer-use workflows; reports completion rate and partial score, token efficiency, and qualitative behavior. Includes safety reports | 108 tasks (long-horizon workflows) | 2026 | arxiv.org |
| REDT-EAMC? | Computer use and web | Attack success rate (ASR) and attempt rate (AR) for adversarial testing of computer-use agents in hybrid web-OS environments | 864 examples (RTC-BENCH) | 2025 | arxiv.org |
| TheAgentCompany | Computer use and web | Evaluates AI agents on real-world workplace tasks through autonomy, task completion, and a specialized metric with partial credit in a simulated software company environment | Not stated | Not stated | doi.org |
| WARC-Bench | Computer use and web | Measures task completion success rate of multimodal AI agents on GUI subtasks in web navigation using verifiable rewards and benchmark datasets | 438 tasks in benchmark; test set 200 real-world examples; train/development set | 2025 | arxiv.org |
| WeaveBench | Computer use and web | Evaluates long-horizon, real-world computer-use agents across GUI, CLI/code, and hybrid interfaces using a trajectory-aware judge and 114 tasks across 8 domains | 114 tasks | 2026 | arxiv.org |
| Web Agents and Benchmarks | Computer use and web | Overview of benchmarks and performance metrics for web agents; discusses task success rate, step success rate, and benchmarking coverage across multiple benchmarks | multiple benchmarks and 79 state-of-the-art web agents and benchmarks reviewed ( | Not stated | doi.org |
| WebArena | Computer use and web | Evaluates functional correctness of task executions for autonomous agents on a realistic web environment | 812 tasks | 2023 | arxiv.org |
| WebCanvas | Computer use and web | Evaluates web agents in online, dynamic web environments using progress-aware evaluation with key node annotations, plus step and task scores (URL, Element Path, Element Value targets with Exact/Include/Semantic matches); reports task success rate and completion rate on Mind2Web-Live | 542 tasks with 2439 intermediate evaluation states | 2024 | arxiv.org |
| WebChoreArena | Computer use and web | Assesses capabilities of web browsing agents on realistic, tedious web tasks including memory, memory-based calculation, and long-term memory across multiple websites | 532 tasks (WebChoreArena); 4 websites used (Shopping, Shopping Admin, Reddit, Gi | 2025 | arxiv.org |
| WebGames | Computer use and web | Evaluates general-purpose web-browsing AI agents across 50+ interactive challenges; reports human vs AI performance and success rates, aiming to measure capability gaps in web interactions | 50+ tasks | 2025 | arxiv.org |
| WebPageBench | Computer use and web | Verifies task success via interface event logs and assesses sensitivity to UI variants using controlled re-renders; 152 tasks across 65 canonical scenarios and 87 control variants; leaderboard with model-harness pairs; gap between declared finished and log-confirmed tasks | 152 tasks, 65 canonical scenarios, 87 control variants, six browser/DOM harness | 2026 | arxiv.org |
| WebSuite | Computer use and web | Diagnostic benchmark to systematically evaluate why web agents fail by disaggregating task failures into specific web actions across taxonomy of category, action, and interaction; provides both individual tasks and end-to-end tasks; evaluates failures of agents on action-level and E2E performance | Not stated | 2024 | arxiv.org |
| AutoPenBench | Cybersecurity | Benchmarks generative agents for automated penetration testing using 33 tasks across two difficulty levels, with milestones and success rate metrics | 33 tasks | 2024 | arxiv.org |
| Capture the Flags | Cybersecurity | Assess robustness and generalization of agentic LLMs to semantics-preserving transformations of CTF challenges, across 13 LLM configurations with tool access | Not stated | 2026 | arxiv.org |
| CTF | Cybersecurity | Assesses LLM-based multi-agent systems solving CTF problems across Crypto and Misc categories, including independent solving rates, hint-assisted improvements, and planning/solution-generation performance on GITW and THUCTF datasets | multiple datasets and puzzles (GITW, THUCTF, ZH puzzles; Crypto and Misc categor | 2025 | mdpi.com |
| CTFExplorer | Cybersecurity | Evaluation of LLM offensive agents in a multi-target web CTF setting, including exploration, target prioritization, multi-stage coordination, and reasoning traces; behavioral evaluation beyond binary flag capture | 40 web-based vulnerable services in a single environment | 2026 | arxiv.org |
| CTFKnow | Cybersecurity | Measures LLMs' ability to understand CTF knowledge and apply it to solve CTF challenges; introduces a benchmark (CTFKnow) and evaluates CTFAgent with retrieval-augmented generation and environmental augmentation | 3992 | 2025 | dl.acm.org |
| CTFTiny | Cybersecurity | CTF Competency Index (CCI) for partial correctness; analysis of LLM hyperparameters; multi-agent coordination settings; framework CTFTiny benchmark of 50 CTF challenges across binary exploitation, web, reverse engineering, forensics, cryptography; CTFJudge for analysis of agent trajectories | 50 | 2025 | arxiv.org |
| CTFusion | Cybersecurity | Evaluates LLM-based agents in cybersecurity Capture The Flag (CTF) benchmarks using a streaming Live CTF-based evaluation framework with per-agent independence and first-correct-flag forwarding | Not stated | 2026 | arxiv.org |
| Cybench | Cybersecurity | Evaluation of cybersecurity task-solving capabilities of LM agents across 40 professionally-level CTF tasks, including subtasks and multiple agent scaffolds, across 8 models, with analysis of performance and task difficulty | 40 tasks | 2024 | arxiv.org |
| CyberGym | Cybersecurity | Evaluates AI agents' ability to generate and reproduce PoC tests for real-world vulnerabilities across large codebases, measured by vulnerability reproduction success rate and discovery of zero-days | 1507 tasks across 188 projects | 2025 | arxiv.org |
| Cybersecurity AI Benchmark | Cybersecurity | Evaluation of LLM models and agents across offensive and defensive cybersecurity domains using five evaluation categories and over 10,000 instances; assesses knowledge vs. capability, adversarial multi-step performance, and robotics/privacy-focused benchmarks | over 10,000 instances | 2025 | arxiv.org |
| DeepRed | Cybersecurity | Partial-credit scoring of checkpoint completion derived from public writeups, using an automated summarise-then-judge labelling pipeline to assign checkpoint labels from execution logs | 10 tasks (ten VM-based CTF challenges) evaluating ten LLM-based agents, three ru | 2026 | arxiv.org |
| DefenderBench | Cybersecurity | DefenderBench score evaluating language agents across offense, defense, and cybersecurity knowledge-based tasks using a standardized agentic framework | 1 | Not stated | doi.org |
| LD-CIPHER | Cybersecurity | Mult-agent LLM framework for collaborative CTF challenge solving; demonstrates improved challenges solved and cost (e.g., NYU CTF Bench) using Planner-Executor agents and a scoring system | Not stated | Not stated | doi.org |
| NYU CTF Bench | Cybersecurity | Evaluates LLMs on 200 CTF challenges with tool integration and automated solving framework across both open-source and black-box models | 200 | 2024 | arxiv.org |
| PACEbench | Cybersecurity | Assesses end-to-end autonomous cyber-offense capabilities of LLM-driven agents across four CVE-based scenarios using a CTF-style flag verification | Not stated | 2025 | arxiv.org |
| RedTeamAI | Cybersecurity | Assesses autonomous cybersecurity agent performance across diverse offensive security tasks using a structured Cybench-based environment with automatic evaluation against reference solutions | not clearly specified | Not stated | doi.org |
| ADRA-Bank | Deep research agents | Evaluates planning, retrieval, and reasoning capabilities of academic deep research (DR) agents using a modular evaluation paradigm with end-to-end and isolated backbone-based assessments | 200 | 2025 | arxiv.org |
| DeepResearch Arena | Deep research agents | Evaluates how well LLMs perform research tasks framed as seminar-grounded tasks using MAHTG to generate and curate over 10,000 tasks across 12 disciplines; reports performance gaps between modern models | over 10,000 tasks from over 200 seminars across 12 disciplines | 2025 | arxiv.org |
| DeepResearch Bench | Deep research agents | Two novel methodologies to evaluate Deep Research Agents: (1) a reference-based method with adaptive criteria to assess the quality of generated research reports; (2) a framework to evaluate information retrieval and collection capabilities by assessing effective citation count and citation accuracy | 100 tasks | 2025 | arxiv.org |
| DeepResearch Bench II | Deep research agents | 9,430 fine-grained binary rubrics evaluating reports across three dimensions: Information Recall, Analysis, and Presentation; 132 tasks across 22 domains; end-to-end rubric satisfaction judged by an LLM and human review | 132 tasks; 9,430 rubrics | 2026 | arxiv.org |
| DeepResearch-9K | Deep research agents | Evaluates multi-step web exploration, targeted retrieval, and sophisticated question answering for deep-research agents; assesses dataset difficulty levels (L1-L3) via number of tool invocations and reasoning chains; supports RL and supervised fine-tuning approaches; provides low-cost data synthesis pipeline | 9000 questions | Not stated | doi.org |
| DEEPWEB-BENCH | Deep research agents | Assesses capability across four families (Retrieval, Derivation, Reasoning, Calibration) using task matrices, per-cell and per-family signals, source-provenance records, and cross-source checks; evaluates nine frontier models with analyses of retrieval, derivation, calibration failures, and cross-domain agreement | Not stated | 2026 | arxiv.org |
| DR-Arena | Deep research agents | Evaluates Deep Research (DR) agents through automated, dynamic investigation using Information Trees, an automated Examiner, and Adaptive Evolvement Loop to test deep reasoning and wide coverage; reports correlation with LM-SYS Search Arena leaderboard and human alignment without manual adjudication | Not stated | Not stated | doi.org |
| Dr. Bench | Deep research agents | Multidimensional evaluation framework and integrated scoring metrics for long-form report-style outputs of Deep Research Agents (DRAs), assessing semantic quality, topical focus, and retrieval trustworthiness | 214 | 2025 | arxiv.org |
| DR3-Eval | Deep research agents | Multi-dimensional evaluation framework: Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality | Not stated | 2026 | arxiv.org |
| KDR-Bench | Deep research agents | General-purpose, knowledge-centric, and vision-enhanced evaluation metrics for Knowledgeable Deep Research (KDR) agents | 41 expert-level questions; 1,252 tables; 9 domains | 2026 | dl.acm.org |
| LiveResearchBench | Deep research agents | DeepEval evaluates content- and report-level quality for citation-grounded long-form reports; supports four evaluation protocols; benchmark comprises 100 expert-curated tasks with real-time web search and synthesis; evaluation of 17 frontier deep research systems | 100 tasks | 2025 | arxiv.org |
| Personalized Deep Research Bench | Deep research agents | Joint evaluation framework (PQR Evaluation Framework) assessing Personalization Alignment, Content Quality, and Factual Reliability for personalized deep research tasks | 50 tasks across 10 domains with 25 authentic user profiles ( yielding 250 user-t | 2025 | arxiv.org |
| ResearchArena | Deep research agents | Evaluates LLMs' ability to conduct academic surveys through three sub-tasks: information discovery, information selection, and information organization, with mind-map as a bonus task; dataset includes 12M full-text papers and 7.9K survey papers; preliminary evaluations compare LLM-based approaches to keyword-based retrieval | 3 tasks (information discovery, information selection, information organization) | 2025 | doi.org |
| ResearchRubrics | Deep research agents | Assesses factual grounding, reasoning soundness, and clarity using a rubrics-based evaluation of deep research agents; includes prompts, rubrics, and evaluation code | 1 task (paper) with 2,500+ rubrics and 2,800+ hours of human labor reported | 2025 | arxiv.org |
| RuWikiBench | Deep research agents | Evaluation of LLMs' ability to reproduce encyclopedia-style articles from Russian Wikipedia sources across source retrieval, article structuring, and section generation tasks | Not stated | Not stated | doi.org |
| Total Recall QA | Deep research agents | Verifiable evaluation suite for deep research agents using total recall QA with structured knowledge base and text corpus | 1 | 2026 | doi.org |
| AI Productivity Index | Economic tasks | Tests whether frontier AI models can perform knowledge work with high economic value using domain-specific tasks and a rubric-based autograding system evaluated by LLM judges | 200 test cases in APEX-v1.0; results for 23 models from 13 providers | Not stated | doi.org |
| Anchor | Economic tasks | Measures ability to generate auditable, constrained task environments and evaluate agent performance on long-horizon, economically valuable tasks with controlled difficulty and known optimal solutions; assesses artifact drift mitigation and task generation quality | 300 | 2026 | arxiv.org |
| APEX-Agents | Economic tasks | Pass@1 (and related metrics like Pass@8, mean score) for evaluating long-horizon, cross-application agent performance on professional-services style tasks | n = 480 tasks across 33 worlds | 2026 | arxiv.org |
| BizAgentBench | Economic tasks | Evaluates goal attainment and execution validity for long-horizon agents in real-world business tasks; reports strict success rates and tool-use overhead to assess execution quality beyond final task completion | 50 | 2026 | arxiv.org |
| Cost Per Correct | Economic tasks | Evaluates cost per correct answer: total spend in USD divided by number of correct answers for LLM agents | 5 | Not stated | doi.org |
| DAYJOB | Economic tasks | Assesses AI agents on long-horizon, professional tasks via containerized Harbor environments with expert rubrics across healthcare and finance, reporting pass rates and task completion metrics | 130 tasks (50 healthcare, 80 finance) | 2026 | arxiv.org |
| ExtractBench | Economic tasks | An end-to-end benchmark and evaluation framework for PDF-to-JSON structured extraction with per-field scoring metrics defined by the JSON Schema; evaluates accuracy, reliability, and handling of schema breadth across economically valuable domains; reports performance of frontier models under various schema complexities | 12,867 evaluatable fields across 35 PDF documents with JSON Schemas and gold lab | 2026 | arxiv.org |
| GDPVAL | Economic tasks | Win rate against human expert baseline for two primary components: a request and a deliverable, with an automated grader for a gold subset; evaluation emphasizes realism, subjectivity, and long-horizon task completion time; tasks are evaluated via head-to-head human expert comparison | approximately 1,320 tasks in the full GDPval set; 220 gold subset tasks; 44 occu | Not stated | doi.org |
| GDPval Revolution | Economic tasks | GDPval benchmark evaluating AI model performance on economically valuable real-world tasks across knowledge-work occupations | 44 occupations across nine sectors | 2025 | doi.org |
| Hedge-Bench | Economic tasks | Deterministic grading against verified expert steps; agents score on open-ended financial reasoning tasks | 102 | 2026 | arxiv.org |
| Joules-to-GDP | Economic tasks | Measures economic value per unit energy of AI outputs using Net Value per Joule (NVJ) and Time Saving per Joule (TSJ) across GDPval tasks, occupations, and models | 220 tasks across 44 occupations, 5 tasks per occupation, 13 categories | 2026 | arxiv.org |
| Market-Bench | Economic tasks | Evaluates LLMs on economically-relevant tasks including economic, operational, and semantic metrics in a multi-agent supply-chain economy; assesses profit, net profit margin, stockout rate, fill rate, stability indices, and language-persona alignment | Not stated | Not stated | doi.org |
| MBABench-2.0 | Economic tasks | Evaluates AI agents on constructing financial spreadsheets using 129 criteria across 12 quality categories, beyond mere verifiability; reports pass rates and performance across agents/models/harness types | 101 tasks | 2026 | arxiv.org |
| OmegaUse-OfficeVal | Economic tasks | Evaluates LLM agents on long-horizon office-suite tasks with task-level economic grounding using human labor time and task price proxy | 100 tasks | 2026 | arxiv.org |
| SWE-Lancer | Economic tasks | Evaluates frontier LLMs on real-world freelance software engineering tasks and maps model performance to monetary value in a benchmarked economy | over 1,400 tasks | 2025 | arxiv.org |
| WAGE-Bench | Economic tasks | Monetary value of AI’s practical usefulness by eliciting willingness to accept compensation for performing realistic tasks under different AI conditions (no AI, with AI assistance, or with alternative models) | 1 | Not stated | osf.io |
| xbench | Economic tasks | Professions-aligned evaluation suite measuring productivity value and TMF prediction across real-world tasks in professional domains; tracks product capabilities over time | Not stated | 2025 | arxiv.org |
| DefAn | Factuality | Assesses factual accuracy, faithfulness to the prompt, and consistency of generated responses using a large-scale benchmark (~75,000 prompts across eight domains) to evaluate LLM hallucinations and factuality | 75000 | 2024 | arxiv.org |
| DiaHalu | Factuality | Evaluates dialogue-level hallucination (factuality and faithfulness) in multi-turn dialogues across four domains with three-step dataset construction and human annotations; classifies faithfulness into Incoherence, Irrelevance, Overreliance and includes Reasoning Error in reasoning dialogues | Not stated | 2024 | arxiv.org |
| Drowzee | Factuality | Evaluates ability to detect fact-conflicting hallucinations (FCH) in LLMs using metamorphic testing with semantic-aware oracles; reports hallucination rates across six LLMs and nine domains (24.7% to 59.8%) | 6 tasks? (six LLMs) across nine domains; explicit task count not provided | 2024 | dl.acm.org |
| ERBench | Factuality | Tests whether LLM answers are correct and whether rationales are verifiable by checking for critical keywords; evaluates answer accuracy, rationale accuracy, and hallucination rates across single-hop, multi-hop, and multimodal questions | Not stated | 2024 | arxiv.org |
| FactBench | Factuality | Factuality evaluation of language models in real-world interactions, verifiability of content units, hallucination scores, prompts usefulness and verifiability | 1000 prompts across 150 topics (FACTBENCH); 382 unique tasks identified from LMS | 2024 | arxiv.org |
| FactCHD | Factuality | Evaluates detection of fact-conflicting hallucinations in LLM outputs with interpretable evidence chains and various factuality patterns (vanilla, multi-hop, comparison, set-operation) and tools-assisted reasoning | multiple samples including FACTCHD dataset with 6,960 evaluation samples plus la | 2023 | arxiv.org |
| Factcheck-Bench | Factuality | Evaluates automatic fact-checkers using a fine-grained, eight-subtask annotation framework and document-level factuality benchmark with F1 performance (best 0.63) reported | 678 open-domain claims across long documents | 2023 | arxiv.org |
| FaithBench | Factuality | Evaluates hallucination rates of LLMs and the accuracy of hallucination detection models on a summarization hallucination benchmark | 10 models across 8 families (FaithBench corpus) | Not stated | doi.org |
| FELM | Factuality | Assesses factuality evaluation performance of LLM-based evaluators on a new benchmark across multiple domains | 817 samples and 3948 segments | 2023 | arxiv.org |
| HalluDial | Factuality | Evaluates dialogue-level hallucination detection, localization, and rationale provision for LLMs across spontaneous and induced scenarios | 4094 dialogues, 146856 samples | 2024 | arxiv.org |
| HalluHard | Factuality | Assesses groundedness and factuality in multi-turn LLM responses using inline citations, a web-search based evidence retrieval and judging pipeline, and evaluation across models | 950 seed questions | 2026 | arxiv.org |
| HalluLens | Factuality | Tests LLM hallucination by extrinsic and intrinsic evaluation tasks, with a taxonomy distinguishing extrinsic vs intrinsic hallucinations; includes dynamic test set generation to mitigate data leakage and analyzes existing benchmarks; proposes a taxonomy and new extrinsic tasks (PreciseWikiQA, LongWiki, NonExistentRefusal) | 3 tasks described (PreciseWikiQA, LongWiki, NonExistentRefusal) | 2025 | arxiv.org |
| HALOGEN | Factuality | Introduces a comprehensive, multi-domain benchmark (HALOGEN) to measure hallucination in long-form generations of LLMs, including automatic verifiers and new metrics (Hallucination Score, Response Ratio, Utility Score) across nine use-cases | ~10,923 prompts; ~150,000 generations from 14 LLMs; nine tasks | 2025 | doi.org |
| HaluEval 2.0 | Factuality | Evaluates factuality hallucination in LLMs using the HaluEval 2.0 benchmark across five domains and analyzes detection, source, and mitigation of hallucinations | 8870 questions total (1535 biomedicine, 1125 finance, 1409 science, 1701 educati | 2024 | arxiv.org |
| KG-FPQ | Factuality | Assesses factuality hallucination in LLMs using knowledge-graph based false premise questions (FPQs); evaluation across discriminative and generative tasks; automated FPQ generation and FPQ Judge evaluator; 93% accuracy on manually annotated set | approximately 178k FPQs across three knowledge domains, six confusability levels | 2024 | arxiv.org |
| KGHaluBench | Factuality | Evaluates truthfulness/accuracy and hallucination rates of LLMs across breadth and depth of knowledge using a knowledge-graph-based framework; introduces new metrics Wa (weighted accuracy), HaluBOK, HaluDOK; includes dynamic question generation and two-layer response verification | 25 models evaluated | 2026 | arxiv.org |
| LLM-OASIS | Factuality | End-to-end factuality evaluation capability; ability to detect factuality in long texts and evidence-based claim verification; GPT-4o accuracy up to 60% in their end-to-end factuality task | 162,550 passages; 681,201 claims; 81,275 Wikipedia pages; ~2k manually-curated s | 2024 | arxiv.org |
| MultiHal | Factuality | Evaluates multilingual, multi-hop factual grounding and semantic similarity in KG-RAG vs vanilla QA across multiple languages and models | ~25.9k high-quality KG-paths (from an initial 140k mined KG-paths) across multip | 2025 | arxiv.org |
| RealFactBench | Factuality | Not stated | 6K claims | 2025 | dl.acm.org |
| SimpleQA | Factuality | Evaluates the ability of language models to answer short, fact-seeking questions with a single, indisputable answer; grading into correct/incorrect/not attempted; assesses factuality in short-form questions | 4,326 questions | 2024 | arxiv.org |
| The Hallucinations Leaderboard | Factuality | Two scores (factuality score and faithfulness score) by averaging evaluation metrics across task categories to quantify LLM hallucinations (factuality and faithfulness) | Not stated | 2024 | arxiv.org |
| TruthfulQA | Factuality | Whether language models generate truthful answers to questions in a zero-shot setting, avoiding imitative falsehoods | 817 questions | 2022 | doi.org |
| WILDHALLUCINATIONS | Factuality | Evaluates factuality of long-form LLM generations about real-world entities using automatic fact-checking with a knowledge source; introduces metrics WILDFACTSCORE and WILDFACTSCORE-STRICT to quantify supported atomic facts and complete correctness respectively | 7,919 entities; 118,785 generations across 15 LLMs | 2024 | arxiv.org |
| AVENIBENCH | Finance | Evaluates six finance-related skills (tabular reasoning, numerical reasoning, question answering, long context modelling, summarisation and dialogue) across 20 LLMs with a ranking-based aggregation and two task difficulty modes (easy/hard) | 8 datasets in AVENIBENCH; 20 models evaluated | Not stated | research.ed.ac.uk |
| BacktestBench | Finance | Evaluates end-to-end performance of LLMs on automated quantitative backtesting across four task categories: metrics calculation, ticker selection, strategy selection, and parameter confirmation; analyzes factors influencing end-to-end performance and grounded verification requirements | 18,246 QA pairs across four task categories; built from over 6 million real mark | 2026 | dl.acm.org |
| BIGFINANCEBENCH | Finance | Evaluates financial-research derivation quality via a 928-item, point-weighted rubric across 36,241 rubric points to assess auditable workflow steps and derivation-level performance of financial-research agents | 928 items, 36,241 rubric points | 2026 | arxiv.org |
| BizBench | Finance | Tests quantitative reasoning in finance/business through program synthesis, quantity extraction, and domain knowledge across eight tasks | 8 tasks | Not stated | doi.org |
| BizFinBench | Finance | Objectives and subjective metrics across five dimensions (numerical calculation, reasoning, information extraction, prediction recognition, and knowledge-based question answering) with 9 fine-grained categories; includes IteraJudge to reduce bias in objective metrics; evaluates 25 models | 6781 tasks (queries) in Chinese | 2025 | arxiv.org |
| Fin-RATE | Finance | Domain-grounded Likert scoring with a 13-type error taxonomy to quantify model degradation and identify bottlenecks across three QA tasks (Detail Reasoning, Enterprise Comparison, Longitudinal Tracking) and under ground-truth vs retrieval-augmented contexts | Not stated | Not stated | doi.org |
| FinAgentBench | Finance | Evaluation of retrieval accuracy and reasoning depth in agentic retrieval, using document-type and chunk-level ranking with top-k metrics (MRR, MAP, nDCG) | about 26K samples | 2025 | dl.acm.org |
| Finance Agent Benchmark | Finance | Tests LLM agent performance on real-world finance research tasks using tools (Google Search, EDGAR) with metrics including accuracy and cost per query | 537 questions | 2025 | arxiv.org |
| FINANCEBENCH | Finance | Evaluates LLMs on open-book financial question answering with 10,231 questions, 361 public filings, and evidence strings; tests 16 model configurations on a sample of 150 cases with manual review (n=2,400 answers) | 10231 | 2023 | arxiv.org |
| FinanceQA | Finance | Evaluates LLMs' performance on complex numerical financial analysis tasks mirroring real-world investment work and financial reasoning, including precision recalculations, accounting conventions, assumption handling, and multi-step calculations | Not stated | 2025 | arxiv.org |
| FinanceReasoning | Finance | Evaluates reasoning capabilities of large reasoning models in financial numerical reasoning; reports accuracy on Hard subset (89.1% for OpenAI o1 with PoT) and improvements via Reasoner/Programmer, coverage and credibility metrics | 2,238 problems in dataset; 3,133 Python-formatted functions | Not stated | doi.org |
| Financial Touchstone | Finance | Evaluates accuracy and hallucination rates of frontier language models on long-context annual report comprehension across 2,878 QA triplets from 480 reports; highlights retrieval as main bottleneck and benchmarking across 11 models | 2878 questions across 480 annual reports (dataset contains 83M tokens) | 2025 | dl.acm.org |
| FinAuditing | Finance | Tests financial concept retrieval, taxonomy-aware relation modeling, and cross-document numerical reasoning using three tasks: FinSM, FinRE, FinMR | 1102 annotated instances | Not stated | doi.org |
| FinBen | Finance | Assesses LLM capabilities across seven financial domains with 36 datasets spanning 24 tasks, including information extraction, textual analysis, QA, text generation, risk management, forecasting, and decision-making; introduces agent-based evaluation and Retrieval-Augmented Generation (RAG); provides open-source datasets, results, and code; evaluates 15 representative LLMs | 36 datasets, 24 financial tasks | 2024 | arxiv.org |
| FinCDM | Finance | Evaluates financial LLMs at knowledge-skill level using cognitive diagnosis across skill-tagged tasks rather than a single aggregate score | 30 tasks evaluated across 30 LLMs (proprietary, open-source, and domain-specific | 2025 | arxiv.org |
| FinDABench | Finance | Tests financial data analysis abilities of LLMs across three dimensions (Foundational Ability, Reasoning Ability, Technical Skill) and six sub-tasks across three task types: classification, extraction, generation; includes numerical reasoning, corporate sentiment risk assessment, chart data to insights, report interpretation/anomaly detection, multi-perspective analysis and visualization generation | 2400 instances (as per text) | 2024 | arxiv.org |
| FinMR | Finance | Evaluates expert-level financial reasoning capabilities of multimodal language models, including knowledge-intensive reasoning, perception-driven visual reasoning, and advanced mathematical reasoning | 3200 | 2025 | arxiv.org |
| FinTradeBench | Finance | Evaluates financial reasoning ability of LLMs across fundamentals, trading signals, and hybrid cross-signal questions, using zero-shot and retrieval-augmented settings with calibration-then-scaling framework | 1400 questions | 2026 | arxiv.org |
| MME-Finance | Finance | Evaluates multimodal large language models on finance domain understanding, perception, and reasoning across open-ended, binary-choice, and multi-turn questions with visual context in a bilingual benchmark | 4751 samples (2,274 open-ended questions, 2,000 binary-choice questions, 477 mul | 2025 | doi.org |
| MultiFinBen | Finance | Evaluation of large language models on multilingual and multimodal financial reasoning tasks across five languages and modalities (text, vision, audio); includes multilingual financial reasoning and financial OCR task families; reports overall performance and modality-specific strengths/weaknesses | Not stated | 2025 | arxiv.org |
| XFinBench | Finance | Assesses LLMs' ability to solve complex, knowledge-intensive financial problems with multi-modal context across four core capabilities: Terminology Understanding, Temporal Reasoning, Future Forecasting, Scenario Planning, and Numerical Modelling; and evaluates performance on three tasks (statement judging, multi-choice with visual data, financial calculation) using 18 LLMs with knowledge augmentation and error analysis | likely 4,235 examples; 3,032 finance terms in knowledge bank; 18 models evaluate | Not stated | doi.org |
| HELM | Holistic | Holistic evaluation of language models across many scenarios and metrics | Not stated | 2022 | arxiv.org |
| Arena-Hard | Human preference | Hard, open-ended prompts drawn from live user queries, judged automatically against a baseline model | 500 prompts in Arena-Hard-Auto benchmark | 2024 | arxiv.org |
| Chatbot Arena | Human preference | Evaluation of LLMs based on human preferences using pairwise comparisons and crowdsourced votes; ranking of models | Not stated | 2024 | proceedings.mlr.press |
| MT-Bench | Human preference | Evaluates alignment of LLMs to human preferences using LLM-as-a-judge on MT-bench and Chatbot Arena; assesses agreement with human preferences and robustness of LLM judges | MT-bench: 80 multi-turn questions; MT-bench questions: ~3K expert votes; Chatbot | 2023 | arxiv.org |
| WildBench | Human preference | Challenging real-world user tasks scored with checklists and automated judges | 1024 examples | 2024 | arxiv.org |
| CFBench | Instruction following | Assesses how well LLM outputs follow complex, real-world constraints across multiple categories and tasks with multi-dimensional assessment criteria and requirement prioritization | 1000 samples; 200 real-life scenarios; 50+ NLP tasks | 2024 | arxiv.org |
| CoDI-Eval | Instruction following | Accuracy of controllable generation across five CTG tasks (Sentiment, Topic, Length, Keyword, Toxicity Avoidance) with diversified instructions; average accuracy across tasks to rank LLMs' | Not stated | 2024 | arxiv.org |
| EIFBENCH | Instruction following | Assesses instruction following ability of LLMs across multi-task scenarios with multifaceted constraints and the SegPO algorithm for per-instruction output optimization | Not stated | Not stated | doi.org |
| FollowBench | Instruction following | Evaluates instruction-following ability of LLMs across multi-level, fine-grained constraint categories; proposes multi-level constraint evolution and hybrid rule-based/model-based evaluation | 820 instructions from over 50 NLP tasks | 2023 | arxiv.org |
| FollowEval | Instruction following | Assesses large language models’ instruction-following ability across multiple dimensions (string manipulation, common sense reasoning, logical reasoning, spatial reasoning, and response constraints) using manually curated English and Chinese test instances | 200 manually written test instances | 2023 | arxiv.org |
| IF-RewardBench | Instruction following | Evaluates judge models’ ability to rank multiple responses to instructions (listwise evaluation) and its correlation with downstream task performance | 842 instructions; 6,011 responses; 16 LLMs; 21 judge models | 2026 | arxiv.org |
| IFEval | Instruction following | Evaluates instruction-following ability of large language models using verifiable instructions and prompts; automatic/accurate assessment of compliance | 541 prompts across 25 verifiable instructions; evaluation of models such as GPT- | 2023 | arxiv.org |
| IHEval | Instruction following | Assessing how well language models follow an instruction hierarchy and resolve conflicts between high- and low-priority instructions across nine tasks with 3,538 examples | 3538 | 2025 | arxiv.org |
| InFoBench | Instruction following | Proposes Decomposed Requirements Following Ratio (DRFR) and INFOBENCH benchmark to evaluate LLMs instruction-following ability across decomposed criteria and constraint categories | 500 instructions and 2,250 decomposed questions | 2024 | arxiv.org |
| Instruction Following Score | Instruction following | Introduces Instruction Following Score (IFS) as a metric to evaluate language models' ability to follow instructions and as an early stopping criterion for instruct tuning | Not stated | 2023 | arxiv.org |
| LIFBench | Instruction following | A rubric-based LIFEVAL evaluation method and a dataset LIFBENCH to assess instruction-following capability and stability of LLMs in long-context scenarios across multiple tasks and length intervals | 2766 instructions across six length intervals, 11 tasks across three scenarios | 2024 | arxiv.org |
| LIFT-Bench and LIFT-Train | Instruction following | Benchmarks to evaluate and train instruction following (IF) with scoped constraints, including verifiable and judged constraints, and training counterpart LIFT-Train; assesses model adherence to constraints and reward hacking resistance | 784 examples in LIFT-Bench; 34,258 examples in LIFT-Train | 2026 | arxiv.org |
| LLMBAR | Instruction following | Evaluates how well LLM evaluators can discern instruction-following outputs using a curated 419-instance meta-evaluation benchmark with natural and adversarial subsets | 419 | 2023 | arxiv.org |
| MCBench | Instruction following | Objective evaluation of LLM outputs for following step-by-step instructions, numerical computation, and long-range consistency using parallel reference code; final accuracy, format following, and following depth as metrics | Not stated | 2025 | arxiv.org |
| MCJudgeBench | Instruction following | Tests constraint-level judge evaluation for multi-constraint instruction following, including correctness and inconsistency metrics across per-constraint labels (yes/partial/no) and judge stability under perturbations | 1 | 2026 | arxiv.org |
| Meeseeks | Instruction following | Evaluates instruction-following capability of LLMs through a multi-round iterative, feedback-driven self-correction framework across three evaluation dimensions: Intent Recognition, Granular Content Validation, and Output Structure Validation | Not stated | 2025 | arxiv.org |
| Multi-IF | Instruction following | Instruction-level strict accuracy and conversation-level strict accuracy across multi-turn and multilingual prompts | 4501 multilingual conversations, 8 languages, 3-turn per conversation | 2024 | arxiv.org |
| ReasonIF | Instruction following | Assesses reasoning instruction following (instruction adherence) in reasoning traces of LRMs; reports Instruction Following Score (IFS) in reasoning traces, correlation with model accuracy, and effects of mitigation strategies like multi-turn reasoning and Reasoning Instruction Finetuning (RIF) | 300 samples in ReasonIF benchmark | 2025 | doi.org |
| RefuteBench | Instruction following | Assesses whether LLMs positively accept refuting feedback and consistently adhere to user demands across single- and multi-feedback dialogues; evaluates responsiveness to feedback, memory of feedback, and ability to generalize and follow instructions under refuting prompts | Not stated | 2024 | arxiv.org |
| REIFE | Instruction following | Evaluation accuracy of LLM-based evaluators for instruction-following, across base LLMs and evaluation protocols, using human evaluations as gold standard | Not stated | 2024 | arxiv.org |
| MMLU | Knowledge | Tests multitask accuracy across 57 tasks spanning elementary mathematics, US history, computer science, law, and more to assess broad model understanding; evaluates breadth and depth of academic and professional knowledge | 57 tasks | 2020 | arxiv.org |
| MMLU-Pro | Knowledge | Discriminative multi-domain language understanding benchmark with more challenging, reasoning-focused questions and increased choice set (ten options) to reduce random guessing; robustness to prompts and effectiveness of Chain of Thought vs direct answering | over 12,000 questions across 14 domains | 2024 | arxiv.org |
| ALARB | Law | Evaluates legal reasoning capabilities of LLMs in Arabic law via tasks including verdict prediction, multistep reasoning completion, and identification of relevant regulations | 13K+ structured legal cases dataset | 2025 | arxiv.org |
| BriefMe | Law | Assesses performance of language models on three tasks for assisting legal professionals in writing briefs: argument summarization, argument completion, and case retrieval | Not stated | Not stated | doi.org |
| CrossLex | Law | Grounded joint evaluation of answer correctness and legal-source grounding across single-jurisdiction, cross-jurisdictional comparison, and fine-grained cross-jurisdictional evaluation | 6,149 instances; 385 fact groups | 2026 | arxiv.org |
| GREEKBARBENCH | Law | Three-dimensional scoring system (Facts, Cited Articles, Analysis) + LLM-judge framework; span-based rubrics; correlation with human expert evaluations; 13 models evaluated | Not stated | 2025 | arxiv.org |
| JurisBench | Law | Workflow-level task execution across four sequential, dependency-aware phases of Chinese civil litigation to diagnose legal reasoning and error propagation; end-to-end cognitive process evaluation | Not stated | 2026 | doi.org |
| KBL | Law | Evaluates LLMs on Korean legal language understanding across 7 knowledge tasks (510 examples), 4 legal reasoning tasks (288 examples), and Korean bar exam (4 domains, 53 tasks, 2,510 examples) in closed-book and retrieval-augmented (RAG) settings | 3 datasets/components: (1) legal knowledge tasks 510 examples; (2) legal reasoni | 2024 | arxiv.org |
| KoBLEX | Law | Evaluates provision-grounded, multi-hop legal reasoning and fidelity of generated answers using ParSeR and LF-Eval; includes a 226-instance benchmark with supporting provisions | 226 | 2025 | arxiv.org |
| Korean Canonical Legal Benchmark | Law | Evaluates legal reasoning independent of parameterized knowledge via two benchmarks (KCL-MCQA and KCL-Essay) using question-aligned precedents and instance-level rubrics for automated evaluation; assesses knowledge coverage and evidence-grounded reasoning | 283 MCQA questions + 169 essay questions | Not stated | doi.org |
| LawBench | Law | Assessment of LLMs' legal knowledge across three cognitive levels (memorization, understanding, applying) and five task types (SLC, MLC, regression, extraction, generation) over 51 LLMs with evaluation results and code | 20 diverse tasks | 2023 | arxiv.org |
| LEGALBENCH | Law | Assesses six types of legal reasoning in large language models across 162 tasks | 162 tasks | 2023 | arxiv.org |
| LeMAJ | Law | Evaluates LLM outputs in legal domain by decomposing answers into Legal Data Points (LDPs) and scoring Correctness and Relevance; correlates with human judgments and improves inter-annotator agreement; includes open-source release of LDPs for LegalBench subset | Not stated | 2025 | arxiv.org |
| LEXAM | Law | Assesses process-based and outcome-based legal reasoning ability of LLMs on long-form/open-ended questions and multiple-choice questions, including evaluation of model-generated reasoning steps via human expert validation | 4886 questions (2841 long-form/open-ended + 2045 MCQs) from 340 exams across 116 | 2025 | arxiv.org |
| LexEval | Law | Standard evaluation methods and metrics across a taxonomy of six legal cognitive abilities and 23 tasks with 14,150 questions | 14,150 questions across 23 tasks | 2024 | arxiv.org |
| oab-bench | Law | Evaluates LLMs on open-ended legal writing questions using reference materials and scoring by a judge model; correlates LLM scores with human scores | 105 questions across seven areas of law from three editions of the OAB exam (39t | 2026 | dl.acm.org |
| One Law, Many Languages | Law | Evaluates multilingual legal reasoning across long-document processing, domain knowledge, multitasking (IR, CVG, LDS, CE, and eight text classification tasks), and reasoning across five languages; reports Macro F1 and other baselines for various tasks | datasets in the hundreds of thousands of examples across multiple tasks | 2023 | arxiv.org |
| PLAWBENCH | Law | Fine-grained legal reasoning using a task-specific rubric across three hierarchical legal workflow tasks, with six evaluation dimensions, evaluated by an LLM-based evaluator aligned to human judgments | 850 questions across 13 practical legal scenarios (approximately 12,500 rubric i | Not stated | doi.org |
| TW-LegalBench | Law | Evaluates LLMs on Taiwanese law using MCQs (accuracy), open-ended essays (LLM-as-Judge framework with rubric-based scoring), and legal judgment prediction (LJP) with sentencing accuracy and statute citation | >16,000 MCQs; 117 OEQs; >14,000 LJP instances | Not stated | doi.org |
| AntiLeakBench | Live and contamination-free | Automatic construction of contamination-free benchmarks with updated real-world knowledge and a fully automated workflow to build/update benchmarks without human labor | Not stated | Not stated | doi.org |
| ArenaBencher | Live and contamination-free | Multi-model evaluation, cross-model fairness, model separability, task alignment, iterative in-context refinement, and contamination-resilient benchmark evolution | Not stated | 2025 | arxiv.org |
| ArxivRoll | Live and contamination-free | RS (Rugged Scores) and proportion of public benchmark contamination; SCP (Sequencing, Cloze, and Prediction) generates private test cases; monthly/biannual benchmark updates (every six months) | Not stated | 2025 | arxiv.org |
| BenchChecker | Live and contamination-free | Evaluates contamination detection capability and robustness for LLM-based bug-fixing benchmarks using AUC ~0.9 and effect on reported resolution rates after filtering contaminated samples | 6 benchmarks evaluated; multiple models and datasets mentioned (StarCoder-15B, S | 2026 | arxiv.org |
| C2LEVA | Live and contamination-free | Evaluates LLMs on contamination-free, comprehensive evaluation across 22 tasks with data-contamination prevention and monthly updated results via a continuously maintained leaderboard | 22 tasks | Not stated | doi.org |
| HelloFresh | Live and contamination-free | Dynamic, real-world, contamination-free continuous evaluation of LLMs on streams of real-world human editorial actions from X community notes and Wikipedia edits, with zero-shot classification and web-search based evaluation | Not stated | 2024 | arxiv.org |
| LatestEval | Live and contamination-free | Evaluates data contamination in language model evaluation and robustness when using time-sensitive test construction; demonstrates reduced memorisation and contamination risk with LatestEval | Not stated | Not stated | doi.org |
| LiveBench | Live and contamination-free | Assesses robustness of LLMs to contamination leakage and regeneration in recursive corpora benchmarks | null | 2024 | arxiv.org |
| LiveMedBench | Live and contamination-free | Automated rubric-based evaluation with contamination-limited, weekly-updated medical benchmarks across 38 specialties and 2 languages; evaluation of 38 LLMs with 16,702 evaluation criteria | ~2,756 real-world cases; ~16,702 evaluation criteria | Not stated | doi.org |
| LiveSearchBench | Live and contamination-free | Assesses retrieval-augmented reasoning over dynamic, temporally grounded knowledge; evaluates how well models handle up-to-date facts vs memorization | Not stated | 2025 | arxiv.org |
| LiveVerilogEval | Live and contamination-free | Assesses contamination-free, automatically scalable Verilog code generation benchmarks and semantic-preserving mutation of RTL designs to evaluate LLMs | Not stated | 2026 | doi.org |
| MACEval | Live and contamination-free | Defines a dynamic continual evaluation framework (MACEval) with multi-agent collaboration, in-process data generation, and AUC-inspired routing metrics to evaluate large models over time across open-ended tasks | 9 | 2025 | arxiv.org |
| 100-LongBench | Long context | A novel metric LongScore to disentangle baseline knowledge from long-context capabilities and a length-controllable long-context benchmark (LongBench) for evaluation of long-context abilities in LLMs | Not stated | Not stated | doi.org |
| Ada-LEval | Long context | Accuracy-based evaluation of long-context understanding across length-adaptable benchmarks for LLMs | 4 closed-source API models + 6 open-source models evaluated | 2024 | arxiv.org |
| BABILong | Long context | Evaluates long-context reasoning capabilities of language models across facts distributed in extremely long documents; reports context utilization and performance across various reasoning tasks and context lengths | 1 | 2024 | dl.acm.org |
| BAMBOO | Long context | Comprehensive evaluation of long-text modeling capacities across generation, knowledge utilization, reasoning, and tool manipulation over long texts using ten datasets from five tasks | 10 datasets from 5 long-text understanding tasks; two length subsets (BAMBOO-4k | 2023 | arxiv.org |
| CLongEval | Long context | Evaluates long-context Chinese LLMs across 7 tasks with context windows from 1K to 100K and 7,267 test samples; analyzes performance of 6 open-source long-context LLMs and 2 commercial models | 7 tasks, 7,267 test samples | 2024 | arxiv.org |
| DebateBench | Long context | Evaluates long-context reasoning in LLMs via speech scoring, speech ranking, and house ordering on British Parliamentary debate transcripts with detailed speech-level scores and house rankings | 256 speeches across 32 debates | 2025 | arxiv.org |
| ETHIC | Long context | Information coverage (IC) to quantify the proportion of input context necessary for answering queries and assess LLMs’ ability to utilize the full long context | 1986 tasks across four long-context domains | 2025 | doi.org |
| Find the Origin | Long context | Tests LLM reasoning degradation as context window size increases across multiple models, using a benchmark called Find the Origin to evaluate reasoning with long contexts | 14 models evaluated; multiple tests and figures described in the abstract | 2024 | doi.org |
| L-Eval | Long context | Assesses standardized evaluation for long context language models (LCLMs) using dataset construction and evaluation metrics, with LIE (Length-Instruction-Enhanced) evaluation and LLM judges; compares metrics against human judgment across multiple LLMs | 20 sub-tasks, 508 long documents, over 2,000 human-labeled query-response pairs | 2023 | arxiv.org |
| LCFO | Long context | Evaluates gradual summarization and summary expansion across diverse domains using human evaluations and state-of-the-art LLMs; reports correlations between automatic metrics and human judgments | approximately 1 dataset plus QA pairs: long inputs (~5k words) with three summar | 2025 | doi.org |
| LongBench | Long context | Automatic evaluation of long-context understanding using metrics such as ROUGE-L and F1 across 21 tasks in 6 categories, bilingual (English and Chinese), with LongBench and LongBench-E variants | 21 tasks across 6 categories; 4,750 test instances | 2023 | arxiv.org |
| LongBench Pro | Long context | Evaluates long-context understanding across bilingual English-Chinese samples with task-specific metrics and a multi-dimensional taxonomy of context requirement, length, and difficulty; analyzes 46 long-context LLMs | 1500 tasks/samples | 2026 | arxiv.org |
| LongBench v2 | Long context | Assesses long-context understanding and reasoning across realistic multitask questions for language models, using 503 multiple-choice questions with contexts from 8k to 2M words | 503 questions | Not stated | doi.org |
| LongLeader | Long context | Evaluation of long-context comprehension abilities of diverse LLMs and context length extension strategies across a unified benchmark with two sets (32K and 128K) and multiple dimensions (Synthetic Retrieval, Free-form QA, Multi-choice QA, In Context Learning, Summarization, Logical Reasoning) | Not stated | Not stated | doi.org |
| LooGLE v2 | Long context | Evaluates LLMs' long-context understanding across real-world long-dependency tasks; reports overall score and context window limitations | 1934 QA instances across 10 domain-specific long-dependency tasks | 2026 | dl.acm.org |
| MILEBENCH | Long context | Evaluates multimodal long-context capabilities of MLLMs across diagnostic and realistic evaluations | 6440 multimodal long-context samples from 21 datasets; average 15.2 images and 4 | 2024 | arxiv.org |
| MiniLongBench | Long context | Evaluates long-context understanding efficiency and correlation with LongBench across many LLMs; reports cost reduction to 4.5% and rank correlation ~0.97 | 237 test samples, 6 major task categories, 21 distinct tasks | 2025 | arxiv.org |
| NOLIMA | Long context | Evaluates long-context retrieval and reasoning performance of LLMs on a needle-in-a-haystack task with minimal lexical overlap between needle and haystack | 13 models evaluated across long-context settings (up to 128K tokens) with perfor | 2025 | dl.acm.org |
| OOLONG | Long context | Tests long-context reasoning and information aggregation capabilities, including multi-hop reasoning, classification and counting in-context, and aggregation over long inputs across two task sets (OOLONG-synth and OOLONG-real) | Not stated | 2025 | arxiv.org |
| RULER | Long context | Evaluates long-context modeling capabilities of language models using a synthetic benchmark with flexible sequence lengths and task complexities across four task categories ( Retrieval, Multi-hop Tracing, Aggregation, Question Answering ) | 13 tasks; 17 long-context LMs evaluated | 2024 | arxiv.org |
| TLDM | Long context | Evaluates long-context understanding in LLMs via narrative tasks (summarization, storyworld description, and narrative time estimation) across novels ranging from <32k to >128k tokens; assesses stability of long-context predictions versus short-context predictions | forty English-language novels; seven frontier LLMs evaluated | 2026 | doi.org |
| XL$^2$Bench | Long context | Evaluates long-context understanding across four tasks (Memory Retrieval, Detailed Understanding, Overall Understanding, Open-ended Generation) over extremely long documents with average length 100K+ words (English) / 200K+ characters (Chinese) using 27 subtasks | 27 subtasks in English and Chinese; 632K questions; tasks across three scenarios | 2024 | arxiv.org |
| ∞Bench | Long context | Evaluates long-context processing ability of LLMs beyond 100K tokens across multiple domains and languages with a bilingual, multi-task benchmark; analyzes model performance degradation and long-context reasoning capabilities | 3946 examples, 12 tasks | Not stated | doi.org |
| AIMO 2025 | Math | Evaluation of augmented LLMs on AI Mathematical Olympiad (AIMO) 2025 dataset, focusing on mathematical reasoning, accuracy, consistency, and interpretability; compares augmented LLMs with base LLMs | 110 problems in the AIMO 2025 dataset (algebra, combinatorics, geometry, number | Not stated | doi.org |
| AIMO-2 Winning Solution | Math | State-of-the-art mathematical reasoning benchmarks;107? (summary) | 540K problems; 3.2M long-reasoning solutions; 1.7M TIR solutions; 566K training | 2025 | arxiv.org |
| AMO-Bench | Math | Evaluates mathematical reasoning of LLMs on Olympiad-level problems; final-answer-based grading; token consumption and accuracy across 26 LLMs; difficulty at IMO level | 50 problems | 2025 | arxiv.org |
| Beyond Benchmarks | Math | Evaluation platform for mathematical reasoning with LLMs; reports model performances on tasks including USA Math Olympiad and research-level questions | Not stated | 2026 | arxiv.org |
| ComBench | Math | Evaluates and diagnoses combinatorial reasoning capabilities of large language models via analysis-centric proof grading and construction-centric verification, reporting overall Avg and Best@4 across frontier models | 100 | 2026 | arxiv.org |
| CombiGraph-Vis | Math | Assesses proof-grading and partial-credit calibration for discrete mathematical reasoning using a corpus of solutions and rubric-based grading workflows; compares different grading workflows and reports agreement with human grades | 90 | 2025 | arxiv.org |
| DOoM | Math | Benchmark to assess capabilities of language models in solving mathematics and physics problems; evaluation methodology and initial results | Not stated | 2025 | arxiv.org |
| EEFSUVA | Math | Evaluates mathematical reasoning of large language models using a benchmark derived from Eastern Europe and former Soviet Union Olympiads to assess reasoning beyond standard IMO-style problems | Not stated | 2025 | arxiv.org |
| FrontierMath | Math | Assesses AI models on the ability to solve exceptionally challenging, original mathematical problems with automated verification and data-contamination safeguards; reports that current state-of-the-art models solve under 2% of problems | hundreds of original problems (as stated) | 2024 | arxiv.org |
| GSM8K | Math | Grade-school math word problems that need multi-step arithmetic reasoning | Not stated | 2021 | arxiv.org |
| HARP | Math | Math reasoning capability across short-answer and multiple-choice questions with human-annotated solutions and programmatic answer checking; evaluates frontier models on difficult olympiad-style problems and provides analysis and code for dataset construction and evaluation | 5409 | 2024 | arxiv.org |
| IMO-Bench | Math | Benchmarks to evaluate robust mathematical reasoning and proof-writing at IMO level, with short answers and full proofs, plus automatic grading via autograders; reports model performance on IMO-AnswerBench and IMO-ProofBench, and correlation with human evaluations | 400 (IMO-AnswerBench) + 60 (IMO-ProofBench) problems | Not stated | doi.org |
| LiveAoPSBench | Math | Assesses math reasoning capabilities of LLMs with an evolving, contamination-resistant evaluation set and QA-based training data from AoPS forum | 652K QA pairs in AoPS-Instruct; LiveAoPSBench dynamically evolving | 2025 | arxiv.org |
| MATH | Math | Evaluates mathematical problem solving capability of ML models using a dataset of 12,500 competition mathematics problems with step-by-step solutions and an auxiliary pretraining dataset; examines model accuracy and scalability limitations in mathematical reasoning | 12500 | 2021 | arxiv.org |
| MATH-Perturb | Math | Evaluate robustness of LLMs' mathematical reasoning to simple vs hard perturbations in math problems; report performance drops on MATH-P-Hard across multiple models | 279 problems each for MATH-P-Simple and MATH-P-Hard; 18 LLMs evaluated | 2025 | arxiv.org |
| Mathador-LM | Math | Tests mathematical reasoning capabilities of LLMs, including ruleset interpretation, planning, problem-solving, and robustness to test-set leakage; evaluates open and closed-source LLMs on dynamic, one-time instances with a target difficulty | multiple tasks across model evaluations; the paper discusses broad experimental | 2024 | arxiv.org |
| MATHARENA | Math | Evaluates LLMs on uncontaminated math competitions to assess mathematical reasoning and proof-writing capabilities across multiple competitions with live, forward-looking problem streams | 149 problems across 5 competitions (30 models evaluated) | Not stated | doi.org |
| MathNet | Math | Benchmarks for three tasks: Problem Solving on MathNet-Solve, Math-Aware Retrieval on MathNet-Retrieve, and Retrieval-Augmented Problem Solving on MathNet-RAG (evaluated with automatic and human expert grading) | Not stated | 2026 | arxiv.org |
| MathOdyssey | Math | Evaluates mathematical problem-solving/reasoning capabilities of large language models across problem types, difficulties (High School, University, Olympiad), and answer formats with detailed solution reasoning annotations | 387 problems | 2025 | nature.com |
| OlymMATH | Math | Tests mathematical reasoning across bilingual (English/Chinese) problems with both numerical-answer rule-based verification and formal Lean 4 verification, covering four core mathematical domains; evaluates LLMs on computational and theorem-proving style tasks | 350 problems total (200 computational + 150 formal Lean 4) | Not stated | doi.org |
| OlympiadBench | Math | Evaluates multimodal mathematical and physical reasoning capabilities of LLMs/LMMs using 8,476 Olympiad-level problems with expert-level annotations; reports average performance scores (e.g., GPT-4V ~17.97% overall, 21.70% math, 10.74% physics) and analyzes errors such as hallucinations and logical fallacies | 8476 | 2024 | arxiv.org |
| Omni-MATH | Math | Assesses mathematical reasoning of LLMs at Olympiad level across 33+ sub-domains and 10+ difficulty levels; evaluates using GPT-4o-based model evaluation and Omni-Judge verifier; reports model accuracies on Olympiad-level problems | 4428 | 2024 | arxiv.org |
| RIMO | Math | Tests advanced mathematical reasoning and problem-solving ability of large language models on Olympiad-level tasks with deterministic (RIMO-N) and stepwise (RIMO-P) evaluation, including evaluation of both single-answer reformulated problems and decomposed proof problems | Thousands of problems: 335 RIMO-N problems and 456 RIMO-P problems | 2025 | arxiv.org |
| SKYLENAGE Technical Report | Math | Assesses mathematical reasoning with SKYLENAGE-ReasoningMATH (100 items) and SKYLENAGE-MATH (150 items) across multiple grades and subjects, reporting per-item metadata and cross-model performance | 2 benchmarks (100-item ReasoningMATH + 150-item MATH), plus models evaluated (15 | 2025 | arxiv.org |
| AgentClinic | Medicine | Multimodal, sequential decision-making evaluation of LLMs in simulated clinical environments across nine medical specialties and seven languages; tool usage and performance under incomplete information | Not stated | 2026 | doi.org |
| CLIBENCH | Medicine | Micro precision, recall, and F1 across multiple clinical decision tasks (diagnosis, procedures, lab tests, prescriptions) with multi-granular evaluation | around 1,000 testing cases per task (diagnosis, procedure, lab tests, prescripti | 2024 | arxiv.org |
| CliMedBench | Medicine | Assesses medical ability of LLMs across seven pivot dimensions (clinical question answering, knowledge application, reasoning, information retrieval, summarization, hallucination, toxicity) using 33,735 questions across 14 core clinical scenarios and evaluated across seven dimensions with automatic metrics such as medical correctness, completeness, fluency, and friendliness | 33735 | 2024 | arxiv.org |
| ClinicalLab | Medicine | Four novel metrics (ClinicalMetrics) to evaluate the effectiveness of LLMs in clinical diagnostic tasks across department guidance and end-to-end clinical diagnostic capabilities | ClinicalBench: 8 tasks, 24 departments, 150 diseases, 1500 samples (10 cases per | 2024 | doi.org |
| ClinicBench | Medicine | Evaluates LLMs across open-ended clinical tasks, reasoning, generation, and understanding; zero-shot and few-shot performance; human evaluation of clinical usefulness (factuality, completeness, preference, safety) | twenty-two LLMs; eleven existing datasets plus six novel datasets; seventeen dat | Not stated | doi.org |
| HealthBench | Medicine | Evaluation of health-care LLMs across accuracy, completeness, context awareness, communication, and instruction-following using 5000 synthetic conversations benchmarked against 48,562 clinician-developed criteria | 1 | 2025 | doi.org |
| LLMEval-Med | Medicine | Evaluates LLMs across five core medical dimensions using 2,996 real-world clinical questions with expert-validated references and an automated LLM-as-Judge evaluation pipeline | 2996 | 2025 | arxiv.org |
| MedAgentBench | Medicine | Assesses agent capabilities of large language models (LLMs) in medical records contexts using a Fast Healthcare Interoperability Resources, compliant interactive environment and 300 clinically derived tasks across 10 categories; reports task success rates (e.g., Claude 3.5 Sonnet v2 ~69.67%) and variation across task categories | 300 tasks, 100 patient profiles, 700,000 data elements | Not stated | doi.org |
| MedBench | Medicine | Tests medical knowledge recall and reasoning abilities of medical language models using a large-scale Chinese benchmark with 40,041 questions from authentic exams and real-world EHR cases across multiple medical branches | 40041 | Not stated | doi.org |
| MedHELM | Medicine | Assesses LLM performance on medical tasks across a taxonomy and benchmark suite; compares LLMs using LLM-jury and cost-performance analysis | 35 benchmarks; 9 frontier LLMs evaluated; 121 tasks in taxonomy; 81 authors list | 2025 | arxiv.org |
| MedQA | Medicine | Medical licensing exam multiple-choice questions | Not stated | 2020 | arxiv.org |
| MedQA-CS | Medicine | Evaluates LLM clinical skills via AI-SCE framework inspired by OSCEs; two tasks: LLM-as-medical-student and LLM-as-examiner; includes quantitative and qualitative assessment and expert-annotated data; compares with traditional benchmarks; assesses information gathering, physical examination, closure, diagnosis; evaluates correlation with expert scoring | Not stated | 2024 | arxiv.org |
| MedS-Bench | Medicine | Evaluates large language models across 11 high-level clinical tasks using instruction-prompted QA, reporting performance gaps between MCQA benchmarks and real clinical tasks; includes an open-source instruction-tuning dataset MedS-Ins and a proof-of-concept model MMedIns-Llama 3 | 11 tasks in MedS-Bench; MedS-Ins comprises 5M instances with 19K instructions ac | 2025 | nature.com |
| MedThink-Bench | Medicine | Assesses medical reasoning capabilities of large language models via a rigorous, scalable benchmark with expert-annotated rationales and an evaluation framework (LLM-w-Rationale) measuring correlation with expert judgments and reasoning trajectories | 500 tasks (QA pairs) | 2025 | doi.org |
| MSDiagnosis | Medicine | Evaluates multi-step clinical diagnostic capability of LLMs, using primary/differential/final diagnosis with macro-recall on key points; BLEU/ROUGE used but limited for rationale and evidence | 2225 medical records from 12 departments, each containing five related diagnosti | 2024 | arxiv.org |
| VivaBench | Medicine | Evaluates sequential clinical reasoning in LLMs through a multi-turn viva voce style benchmark | 1152 physician-curated clinical vignettes | 2026 | dl.acm.org |
| BLEnD | Multilingual | Evaluates LLMs' everyday cultural knowledge across 16 countries/regions and 13 languages using short-answer and multiple-choice formats; reports performance gaps across cultures and languages | 52.6k question-answer pairs (15k short-answer, 37.6k multiple-choice) | 2024 | arxiv.org |
| BUFFET | Multilingual | Evaluates few-shot cross-lingual transfer across 15 tasks and 54 languages using fixed few-shot demonstrations and text-to-text format; compares multilingual LLMs with in-context learning and fine-tuning | Not stated | 2024 | doi.org |
| ChiKhaPo | Multilingual | Evaluates lexical comprehension and generation abilities of generative models across a massively multilingual set of languages and tasks | 8 subtasks across 2 evaluation directions (4 tasks × 2 directions) with language | Not stated | doi.org |
| CMiLBench | Multilingual | Evaluation of multilingual LLMs on 17 task categories across language understanding and generation for Tibetan, Mongolian, and Uyghur using automatic metrics and LLM-as-a-Judge scoring | 24,663 samples across 17 task categories | Not stated | doi.org |
| DEEPSEA | Multilingual | Evaluates multilingual understanding and generation across five tasks (mathematical reasoning, scientific reasoning, question answering, abstractive summarization, and machine translation) over 12 Southeast Asian languages; analyzes performance across high- and low-resource languages and cross-lingual effects | 5 tasks x 12 languages (DEEPSEA benchmark) plus detailed language/resource strat | 2026 | arxiv.org |
| Eka-Eval | Multilingual | Evaluates LLMs across 55+ benchmarks in 9 categories with multilingual, low-resource focus; includes zero-code web UI and interactive CLI; demonstrates improvements in usability, setup time, and reproducibility | 55+ benchmarks across 9 categories | 2025 | arxiv.org |
| EthioBenchmark | Multilingual | Evaluates multilingual LLMs on five Ethiopian languages across multiple downstream NLP tasks using EthioBenchmark datasets and task-specific fine-tuned models | Not stated | 2024 | arxiv.org |
| GlotEval | Multilingual | Unified evaluation of LLMs across 27 benchmarks and 9 tasks using ISO 639-3 language codes, enabling language-specific, cross-benchmark analysis and non-English-centric evaluations across 1500+ languages | 27 benchmarks, 9 tasks, 1500+ languages | Not stated | doi.org |
| INCLUDE | Multilingual | Assesses multilingual LLM capabilities across 44 languages using 197,243 QA pairs spanning regional knowledge, cultural knowledge, and exam-style questions; includes two subsets INCLUDE-BASE and INCLUDE-LITE for multilingual evaluation under different resource settings | 197243 | 2024 | arxiv.org |
| IrokoBench | Multilingual | Evaluates zero-shot, few-shot, and translate-test settings for multiple tasks (NLI, mathematical reasoning, and knowledge-based QA) across 17 typologically diverse African languages using open and proprietary LLMs | 3 tasks across 17 languages, evaluated across 10 open and 6 proprietary LLMs | 2024 | arxiv.org |
| LaoBench | Multilingual | Multidimensional evaluation of LLMs on Lao across knowledge application, K12 education, and bilingual translation; includes held-out and open-source subsets; black-box evaluation protocol for fairness and data security | 17,000+ samples; Lao-7k, Lao-10k, Lao-500 subsets | Not stated | doi.org |
| M3Exam | Multilingual | Assesses multilingual, multimodal, and multilevel proficiency of large language models using real and official human exam questions | 12317 | Not stated | doi.org |
| MEGAVERSE | Multilingual | Evaluates multilingual capabilities of LLMs across 83 languages over 22 datasets (MEGA-based extended benchmark) including multimodal evaluation and contamination analysis | 22 datasets, 83 languages | Not stated | doi.org |
| MGSM | Multilingual | Evaluates reasoning abilities of large language models in multilingual settings using the MGSM benchmark derived from GSM8K translated into 10 languages; shows chain-of-thought prompting scales with model size and multilingual reasoning abilities across tasks (math, commonsense, word-in-context) | null | 2022 | arxiv.org |
| MiLiC-Eval | Multilingual | Evaluates multilingual LLM performance on minority languages of China across 9 tasks with 24K instances, focusing on underrepresented writing systems and parallelism across languages and task formats; assesses linguistic and problem-solving skills, tokenization, and handling of multiple scripts | 24000 | Not stated | doi.org |
| MINERS | Multilingual | Evaluates multilingual LMs as semantic retrievers across bitext retrieval, retrieval-based classification, and in-context learning (ICL) classification over 200+ languages; assesses robustness across language diversity and cross-lingual/code-switching settings | 200+ languages; 11 encoder LMs; 11 generative LMs (open-source and commercial AP | 2024 | arxiv.org |
| MLRBench | Multilingual | Tests multilingual long-context reasoning for LLMs beyond retrieval, including multi-hop inference, aggregation, and epistemic reasoning across seven languages; evaluates context-length scalability and leakage resistance | Not stated | 2026 | doi.org |
| MMLU-ProX | Multilingual | Assesses multilingual LLM performance across 29 languages using identical questions; includes a lite version with 658 questions per language; evaluates 36 state-of-the-art LLMs; analyzes cross-linguistic reasoning and resource disparity | New benchmark with 29 languages (11,829 identical questions per language; lite v | 2025 | arxiv.org |
| MuBench | Multilingual | Evaluates multilingual LLM capabilities across 61 languages and introduces Multilingual Consistency (MLC) as a complementary metric to accuracy | 61 languages | Not stated | doi.org |
| ONE RULER to measure them all | Multilingual | Evaluates multilingual long-context language models across 26 languages using synthetic tasks that test retrieval and aggregation, including needle-in-a-haystack variants; analyzes performance across languages and context lengths (8K to 128K tokens); cross-lingual instruction-context effects | 26 tasks/languages plus multiple synthetic task variants; multiple models evalua | 2025 | arxiv.org |
| SemEval-2026 Task 7 | Multilingual | Evaluation of adaptability of LLMs and NLP systems across multiple languages and cultures using two tracks (Short-Answer Questions and Multiple-Choice Questions) with gold-label matching and language-specific normalization | 33 datasets; more than 30 language-culture pairs; 140+ participants; 62 final su | Not stated | doi.org |
| TLUE | Multilingual | Evaluates Tibetan language understanding and safety across 5 domains, 67 subdomains for knowledge plus 7 safety subdomains; total 22,963 evaluation questions (Ti-MMLU: 11,528; Ti-SafetyBench: 11,435) | 22963 | 2025 | doi.org |
| xLLMs-100 | Multilingual | Evaluates multilingual understanding and generation across 100 languages on five benchmarks (PAWS-X, XCOPA, XL-Sum, FLORES-101, SELF-Instruct) using zero-shot setting and DPO-aligned crosslingual feedback | five benchmarks with datasets spanning 100 languages (instruction data for 100 l | Not stated | doi.org |
| XTREME-UP | Multilingual | Evaluates multilingual language models across 88 under-represented languages on 9 user-centric technologies (ASR, OCR, MT, information access, etc.) in scarce-data scenarios; creates new datasets for OCR, autocomplete, semantic parsing, transliteration; evaluates text-only and multimodal settings including in-context learning and supervised tuning | Not stated | 2023 | arxiv.org |
| EXAMS-V | Multimodal | Evaluation of vision-language models on a large, multilingual, multimodal exam benchmark with 20,932 multiple-choice questions across 20 subjects in 11 languages; assesses reasoning over combined text and visual content | 20932 | 2024 | arxiv.org |
| LVLM-EHub | Multimodal | Comprehensive evaluation of LVLMs via quantitative capability evaluation and an online arena platform across 42 in-domain benchmarks and multiple LVLMs | 13 LVLMs evaluated; 42 benchmarks; arena platform | Not stated | doi.org |
| M$^3$-VQA | Multimodal | Tests multimodal multi-entity reasoning, multi-hop VQA with evidence and retrieval-augmented input across MLLMs | Not stated | 2026 | arxiv.org |
| MathVista | Multimodal | Evaluates mathematical reasoning in visual contexts across 12 foundation models; overall accuracy metrics and qualitative analyses; self-verification and self-consistency insights; visual and textual reasoning performance gaps | 6141 examples | 2023 | arxiv.org |
| MCTBench | Multimodal | Three categories of tasks (perception, reasoning, and content-creation); multiple-choice QA for perception and reasoning; open-ended generation for content-creation; automated evaluation pipeline using MLLMs as evaluator | approximately 5.2k images and 8.5k QA pairs; 2,734 perception MCs, 2,602 reasoni | 2024 | arxiv.org |
| MDK12-Bench | Multimodal | Evaluates multimodal reasoning capabilities of multimodal large language models (MLLMs) on real-world K-12 examinations across six disciplines, with instance-level knowledge points, explanations, difficulty labels, cross-year partitions, and a dynamic evaluation framework to mitigate data contamination | 140K reasoning instances; 141.3K reasoning questions | 2025 | arxiv.org |
| MLLM-Bench | Multimodal | Evaluates MLLMs with per-sample criteria across six cognitive levels using a pairwise comparison framework with a powerful judge (GPT-4V) to align model responses with per-sample criteria; reports agreement with human evaluation | Not stated | 2023 | arxiv.org |
| MM-BigBench | Multimodal | A comprehensive evaluation framework for multimodal content comprehension across models and instructions using metrics including Best Performance, Model Mean Relative Gain, Instruction Mean Relative Gain, Model Stability, Instruction Stability, and Adaptability | 20 models (14 MLLMs) on 14 multimodal datasets across 6 tasks with 10 instructio | 2023 | arxiv.org |
| MM-Vet v2 | Multimodal | Evaluates integrated vision-language capabilities including recognition, knowledge, spatial awareness, language generation, OCR, math, and a new image-text sequence understanding capability; benchmarks large multimodal models and provides a code/data/leaderboard | Not stated | 2024 | arxiv.org |
| MMBench | Multimodal | Multimodal ability evaluation for vision-language models using a curated, bilingual (English-Chinese) benchmark with over 3000 multiple-choice questions across 20 ability dimensions; evaluation uses CircularEval and GPT-4-based choice matching to map predictions to predefined options | over 3000 questions across 20 dimensions (each dimension >125 questions) | 2023 | arxiv.org |
| MMIE | Multimodal | Evaluates interleaved multimodal comprehension and generation for Large Vision-Language Models using a large-scale benchmark with diverse question types and a proposed automated evaluation metric | 20000 | 2024 | arxiv.org |
| MMMU | Multimodal | Evaluates expert-level multimodal understanding across perception, knowledge, and reasoning for foundation models, across 6 disciplines, 30 subfields; assesses performance on 11.5K questions with various image types and modalities | 11550 questions; 11.5K questions total; 6 disciplines / 30 subjects / 183 subfie | 2023 | arxiv.org |
| MMT-Bench | Multimodal | Assesses LVLMs across 32 core meta-tasks and 162 subtasks in multimodal understanding via a task map and 31,325 curated multi-choice visual questions | 31325 | 2024 | arxiv.org |
| MULTI | Multimodal | Evaluation of multimodal large language models (MLLMs) on authentic examination-style questions requiring image-text understanding, complex reasoning, and knowledge recall | over 18000 questions in MULTI; 500 in MULTI-Elite; 4500+ in MULTI-Extend | 2025 | link.springer.com |
| PCA-Bench | Multimodal | Error localization and decision accuracy across perception, cognition, and action; PCA-Eval automatic evaluation protocol; comparison of 10 MLLMs; GPT-4 Vision vs open-source models; improvement from Embodied-Instruction-Evolution | Not stated | 2024 | arxiv.org |
| ReForm-Eval | Multimodal | Evaluation of various capabilities of LVLMs (large vision-language models) using unified reformulated benchmarks | Not stated | 2024 | dl.acm.org |
| SEED-Bench | Multimodal | Evaluates hierarchical multimodal capabilities of MLLMs across 27 dimensions using 24K multiple-choice questions with ground-truth options | 24000 | Not stated | doi.org |
| TinyLVLM-eHub | Multimodal | Assesses six multimodal capabilities of LVLMs (visual perception, visual knowledge acquisition, visual reasoning, visual commonsense, object hallucination, embodied intelligence) using 2.1K image-text pairs and ChatGPT-based Ensemble Evaluation (CEE) | 2.1K image-text pairs | 2023 | arxiv.org |
| UrBench | Multimodal | Evaluates large multimodal models (LMMs) across multi-view urban scenarios using 11.6K questions over 14 tasks and 4 dimensions (Geo-Localization, Scene Reasoning, Scene Understanding, Object Understanding) to compare LMMs against human performance, with cross-view relations and multi-view annotations | 11.6K questions, 14 task types, 4 dimensions | 2025 | doi.org |
| VL-GLUE | Multimodal | Assesses visuo-linguistic reasoning across seven tasks totaling 106k image-passage-question-answer tuples and evaluates robustness of vision-language models in zero-shot and fine-tuned settings | 7 tasks; 106k problems/images-passage-question-answer tuples | 2024 | arxiv.org |
| VLMEvalKit | Multimodal | Evaluates large multi-modality models using over 70 models, 20+ benchmarks, end-to-end evaluation pipeline including data prep, distributed inference, post-processing, and metrics; provides leaderboard and open-source toolkit | Not stated | 2024 | dl.acm.org |
| ARC-AGI | Reasoning | Abstract reasoning puzzles that test how efficiently a system acquires new skills on novel tasks | null | 2019 | arxiv.org |
| ARC-AGI-2 | Reasoning | Evaluates abstract reasoning and problem-solving abilities at higher levels of fluid intelligence using an upgraded ARC-AGI benchmark with a more granular signal for abstract reasoning in frontier AI systems | Not stated | 2025 | arxiv.org |
| BEYONDBENCH | Reasoning | Evaluates reasoning and algorithmic problem-solving capabilities of language models using contamination-resistant, on-the-fly generated algorithmic problems across multiple task suites and difficulty levels | 44 algorithmic tasks with 117 variations across three suites (Easy, Medium, Hard | 2025 | arxiv.org |
| BIG-Bench Extra Hard | Reasoning | Evaluates general-purpose and reasoning-specialized LLM models on a harder variant of BBH, reporting accuracy figures (23.9% for general-purpose, 54.2% for reasoning-specialized) and qualitative failure analyses | Not stated | Not stated | doi.org |
| BIG-Bench Hard | Reasoning | The 23 hardest BIG-Bench tasks, used to test multi-step reasoning | 23 tasks (27 subtasks) in BBH | 2023 | doi.org |
| Easy2Hard-Bench | Reasoning | Datasets with continuous numerical difficulty labels for problems across six domains; evaluation of LLM performance across varying difficulty, using IRT and Glicko-2 for difficulty estimation | 6 datasets | 2024 | arxiv.org |
| EnigmaEval | Reasoning | Evaluates long multimodal reasoning capabilities of frontier language and vision-language models across normal and hard puzzle splits with accuracy metrics, including OCR/parsing sensitivity and transcribed vs original formats | 1184 puzzles | 2025 | arxiv.org |
| FATE | Reasoning | Evaluation of state-of-the-art LLM provers on FATE showing pass@64 accuracy for FATE-H and 0% on FATE-X; two-stage evaluation of natural-language reasoning vs formalization | 200 problems (100 in FATE-H and 100 in FATE-X) | 2025 | arxiv.org |
| FormulaOne | Reasoning | Tests depth of algorithmic reasoning of frontier AI models on a graph/logic/algorithms benchmark beyond competitive programming, including an evaluation framework and warmup tasks | Not stated | 2025 | arxiv.org |
| Hard2Verify | Reasoning | Assesses step-level verification of frontier LLMs on open-ended math problems, including step-level annotations, first-error identification, and balanced F1 metric; evaluates generative critics and process reward models | approximately 1860 rigorously graded steps across 200 unique model responses | Not stated | doi.org |
| HardcoreLogic | Reasoning | Evaluation of large reasoning models on a long-tail set of logic puzzle games, focusing on robustness and handling of increasing complexity, uncommon elements, and unsolvability; analyzes model performance drops and error categories | over 5,000 puzzles across 10 games | 2025 | arxiv.org |
| HellaSwag | Reasoning | Tests ability of models to perform commonsense natural language inference by selecting the most likely continuation for an event description, evaluating model vs human performance on the HellaSwag dataset | Not stated | 2019 | arxiv.org |
| LongCoT | Reasoning | Tests long-horizon chain-of-thought reasoning capabilities of frontier models using 2,500 expert-designed problems across multiple domains | 2500 | 2026 | arxiv.org |
| MathConstraint | Reasoning | Evaluates combinatorial reasoning capabilities of LLMs with and without tool use; assesses accuracy under varying tool access and budgets | ? (329 instances in MathConstraint; 266 in MathConstraint-Easy) | 2026 | arxiv.org |
| MultiChallenge | Reasoning | Evaluates frontier LLMs on multi-turn conversation capabilities via four challenge categories and an LLM-as-judge with instance-level rubrics; reports accuracy of models on MultiChallenge and correlation with human judgments | Not stated | Not stated | doi.org |
| NazoNazo Benchmark | Reasoning | Evaluates insight-like representational restructuring and metacognitive evaluation in reasoning, including accuracy on a curated riddles subset and verification/endorsement gap analysis | 201 riddles total; 120-item subset with human reference (n=126) | 2025 | arxiv.org |
| null | Reasoning | Assesses model performance on paired P and NP-complete problems under adversarial multi-agent rewriting to separate reasoning from recall | 2114 | 2026 | arxiv.org |
| PUTNAMBENCH | Reasoning | Evaluates neural and symbolic theorem-provers on Putnam competition problems across multiple proof languages; reports how many problems are solved across languages | 1692 hand-constructed formalizations of 640 theorems | 2024 | arxiv.org |
| RiddleBench | Reasoning | Assesses general generative reasoning capabilities of LLMs across multi-faceted tasks including logical deduction, spatial awareness, and constraint satisfaction; evaluates robustness, self-correction, and resistance to hallucination cascades | 1737 | 2025 | arxiv.org |
| Soohak | Reasoning | Evaluates research-level mathematical reasoning capabilities of LLMs, including a Challenge subset and a refusal subset to probe handling of ill-posed problems and confident-only-answer behavior | 439 tasks | 2026 | arxiv.org |
| Behavior Beyond Text | Safety | Assesses whether refusal behavior in text transfers to tool use by measuring tool-call safety across text-only, standard tool-enabled, and adversarial tool-enabled prompts using 20 predefined forbidden actions across 2,304 prompt triplets in four domains; reports refusal rates in chat vs tool-enabled contexts and activation signal analyses | 2304 | 2026 | arxiv.org |
| BLINDSPOT | Safety | Trajectory-level safety calibration across long-horizon tool-using agent interactions, including unsafe completion, appropriate refusal, benign utility, over-refusal, repeated-run robustness, and post-refusal failure | 22 attack families and 35 scenarios across seven domains; >2,500 long-horizon tr | 2026 | arxiv.org |
| CFSafety | Safety | Assesses safety of LLMs using 10 safety categories with a 0-5 safety rating scale; evaluates NLG capabilities across 8 LLMs; provides a dataset of 25k prompts in 10 safety categories (2500 questions per category); bilingual Chinese-English evaluation; combines moral judgment with token-probability weighted safety score | 25000 prompts / 2500 questions per category | 2024 | arxiv.org |
| DiagSafe | Safety | Fine-grained prompt-level rubrics with two dimensions (safety and usefulness) across 84 safety labels; agreement with human annotations (95.34%), inter-judge consistency (Cohen’s kappa = 0.864); unsafe rates across models; performance in Chinese and multi-turn contexts | 84-label safety taxonomy; two subsets Diag Base and DiagAttack | 2026 | arxiv.org |
| DUAL-Bench | Safety | Tests over-refusal and robustness/safe completion in vision-language models across hazard categories under semantics-preserving perturbations | 18 models, 12 hazard categories | 2025 | arxiv.org |
| Exaggerated Safety Benchmark | Safety | Assesses exaggerated safety behaviors and false refusals in LLMs across single-turn and multi-turn prompts; evaluates compliance on safe prompts and refusal rates on unsafe prompts using scenario-based diagnostics and post-hoc mitigation strategies | XSB: 12 prompt types, 340 safe prompts and 240 unsafe prompts; MS-XSB: 30 scenar | 2025 | arxiv.org |
| Forbidden Science | Safety | Assesses safety/refusal behavior of LLMs across prompts and prompt variations, measuring refusals vs allowances and response consistency | 512 prompts initially constructed (for Scientific Refusal Test); multiple models | 2025 | arxiv.org |
| FORTRESS | Safety | Average Risk Score (ARS) and Over-Refusal Score (ORS) across multiple LLMs evaluating safeguard robustness | 500 prompts with 10 subcategories across 3 domains | 2025 | arxiv.org |
| FUSE | Safety | Evaluates dangerous capabilities of LLMs via K (Knowledge), D (Defense), H (Harm) pipelines and reports cross-judge consistency and inter-correlations | 12 models evaluated across four families | 2026 | arxiv.org |
| GUARDSET-X | Safety | Evaluates guardrail models across multi-domain safety risks with policy-grounded risk construction; reports F1 scores and domain coverage/robustness against adversarial prompts | Not stated | 2026 | dl.acm.org |
| HarmBench | Safety | Attack success rate (ASR) as primary measure of a red-teaming method's success | 18 red teaming methods, 33 LLMs and defenses, 510 harmful behaviors (400 textual | 2024 | arxiv.org |
| OR-Bench | Safety | Measures over-refusal behavior of large language models across multiple rejection categories and prompts; dataset size and evaluation across 32 LLMs | 80,000 safe prompts plus ~1,000 hard prompts plus 600 toxic prompts (80K prompts | 2024 | arxiv.org |
| RefusalBench | Safety | Generative evaluation of selective refusal performance across models using perturbation-based diagnostic test cases and three intensity levels across six categories of informational uncertainty; reports refusal detection and categorization skills, and the impact of model scale and alignment | approximately 30+ models evaluated; two benchmarks released (RefusalBench-NQ and | 2025 | arxiv.org |
| RiskCueBench | Safety | Evaluates safety behavior conditioned on recoverability of user intent; measures refusal rate, global safety score, and unsafe cooperation under recoverable vs non-recoverable prompts | 160 latent intents and 640 abstract non-operational surface prompts across eight | 2026 | arxiv.org |
| SG-Bench | Safety | Assesses generalization of LLM safety across diverse tasks and prompt types, including open-end generation, multiple-choice, safety judgments, and effects of prompt engineering and jailbreak on safety | not clearly stated (mentions 1,442 malicious queries and subsets across 4 genera | 2024 | arxiv.org |
| SORRY-Bench | Safety | Evaluates whether LLM responses refuse or fulfill unsafe instructions using a fine-grained 44-class taxonomy, 440 unsafe instructions, and 7k+ human annotations; analyzes LLM-as-judge designs and compares automated safety evaluators | 440 instructions; 7k+ annotations; 50+ models evaluated | 2024 | arxiv.org |
| TIER | Safety | Six-label behavior scale assessed by two independent LLM judges across four risk domains and four threat levels | Not stated | 2026 | arxiv.org |
| WalledEval | Safety | Comprehensive safety evaluation across multiple LLM safety dimensions (harmful behavior, refusal behavior, multilingual safety, exaggerated safety, prompt injections) using 35+ safety benchmarks and custom mutators | 35+ datasets/benchmarks | 2024 | arxiv.org |
| ATLAS | Science | Multidisciplinary scientific reasoning with high-difficulty, open-ended questions across seven core fields; evaluation of frontier models with cross-disciplinary integration and multi-step reasoning; includes LaTeX-formatted expressions and expert-quality judgments | approximately 800 original problems | 2025 | arxiv.org |
| Ever-Evolving Science Exam | Science | Dynamic benchmark evaluating scientific understanding across multiple disciplines with leakage-resilient, low-overhead evaluation and rigor across Range, Reach, and Rigor | over 100K instances in EESE-Pool; 500-instance periodically updated subset | 2025 | arxiv.org |
| Expert-validated STEM QA | Science | Tests expert-validated STEM knowledge across Physics, Chemistry, Biology, and Mathematics using a dataset of 398 questions with expert consensus and evaluation of frontier AI models | 398 questions in the dataset | 2026 | arxiv.org |
| FrontierScience | Science | Granular rubric-based evaluation framework to assess model capabilities throughout the process of solving a research task and in-depth evaluation of expert-level scientific reasoning across two tracks (Olympiad and Research) | several hundred questions (including 160 in the open-sourced gold set) | 2026 | arxiv.org |
| Genome-Bench | Science | Tests scientific reasoning capability via automated pipeline converting real-world expert discussions into a multiple-choice benchmark with 3000+ QA pairs in genomics | 3000+ questions | Not stated | doi.org |
| GPQA | Science | Assesses difficulty of graduate-level multidisciplinary multiple-choice questions and the ability of humans and AI to answer them (Google-proof Q&A; expert vs non-expert vs AI baselines) | 448 | 2023 | arxiv.org |
| HARDMath2 | Science | Evaluation of LLMs on a dataset of 211 original applied mathematics problems covering graduate-level topics; assesses mathematical reasoning, approximation-based problem solving, and ability to match solutions with numerical ground truths and peer-validated work | 211 | Not stated | doi.org |
| Humanity’s Last Exam | Science | Assesses expert-level knowledge across dozens of subjects using multi-modal, open-ended and multiple-choice questions with automated grading to evaluate LLM capabilities at an expert level | 2500 | 2026 | link.springer.com |
| MSQA | Science | Evaluates LLMs on graduate-level materials science reasoning and knowledge across factual accuracy and multi-step reasoning in seven sub-fields | 1757 | 2025 | arxiv.org |
| Physics | Science | LLM-based grading protocol evaluating individual answers and complete multi-part problems across numeric, symbolic, and text responses | 1034 problems | 2025 | arxiv.org |
| Probing the Critical Point | Science | Tests LLMs on unpublished, research-level reasoning tasks across multiple physics domains using composite challenges and checkpoint tasks with automated, physics-specific grading | 71 composite research challenges; 190 checkpoint tasks | 2025 | arxiv.org |
| R-Bench | Science | Evaluates complex reasoning capabilities of language and multimodal models across a large, multi-disciplinary question set in English and Chinese | approximately 1,094 questions for language evaluation across 108 subjects; 665 q | 2025 | arxiv.org |
| SciCode | Science | Evaluates code generation abilities of language models on scientist-curated natural science problems, including subproblems, gold-standard solutions, and test cases; reports model performance on 16 domains and 338 subproblems | 338 subproblems from 80 main problems | 2024 | arxiv.org |
| SciEx | Science | Performance of LLMs on university computer science exam questions using expert grading and automatic grading (Pearson correlation with expert grading, exam-grade percentages) | Not stated | 2024 | arxiv.org |
| SciKnowEval | Science | Assesses LLMs across five progressive levels of scientific understanding (memory, comprehension, reasoning, discernment, application) using a 28K-question dataset spanning biology, chemistry, physics, and materials science, evaluated on 20 LLMs; reports on strengths and challenges especially in reasoning and real-world application | 28000 | 2024 | arxiv.org |
| SciQA | Science | Benchmarks QA over scholarly knowledge using SPARQL queries over ORKG; includes 100 handcrafted questions and 2465 autogenerated questions with corresponding SPARQL queries and ground-truth results | 2765 | 2023 | nature.com |
| SciVQR | Science | Assesses multimodal scientific reasoning across 6 core domains (mathematics, physics, chemistry, geography, astronomy, biology) with questions ranging from easy to hard and includes expert-authored solution traces for detailed reasoning evaluation | 54 subfields across 6 domains | Not stated | doi.org |
| SuperGPQA | Science | Evaluates graduate-level knowledge and reasoning capabilities across 285 disciplines; reports model accuracy and insights from a large-scale annotation process | 285 | 2025 | arxiv.org |
| Theoretical physics benchmark | Science | Evaluates AI reasoning capability in theoretical physics across problem difficulties (from undergraduate to research level) and assesses auto-verifiability, grading, and model performance on TPBench problems | 57 problems in the first iteration | Not stated | doi.org |
| TuringQ | Science | Evaluates LLM reasoning in theory of computation, using Chain of Thought prompting and expert human assessment; evaluates open-source LLMs and GPT-4; automated LLM-based evaluation; fine-tuning improvements on out-of-domain tasks | 4006 Q, A pairs | Not stated | doi.org |
| Virology Capabilities Test | Science | Tests expert-level virology troubleshooting capability using multimodal questions; evaluates LLMs vs expert virologists on practical virology lab knowledge | 322 | 2025 | arxiv.org |
| AGENTISSUE-BENCH | Software engineering | Evaluates state-of-the-art software engineering agents on agent issue resolution tasks and reports resolution and localization rates | 50 tasks in AGENTISSUE-BENCH (reproducible benchmark); 201 analyzed agent issues | 2025 | arxiv.org |
| CONTEXTBENCH | Software engineering | Process-oriented evaluation of context retrieval in coding agents, measuring recall, precision, and efficiency of code context retrieval during issue resolution across file/block/line granularities | 1136 tasks (and 522,115 lines of gold contexts, 23,116 classes/functions, 4,548 | 2026 | arxiv.org |
| FAUN-Eval | Software engineering | Evaluation of LLMs across QA, fault localization, and code editing using metrics such as Exact Match (EM), BLEU, CodeBLEU, ROUGE-L, ROUGE-1, and Edit Similarity | 300 entries | 2024 | arxiv.org |
| Hunk-SWE | Software engineering | Repairs accuracy on multi-hunk defects across agents, file-level localization, and operational efficiency (tokens, runtime) using a per-instance Docker pipeline and SWE-bench harness | 32 | 2026 | doi.org |
| Multi-SWE-bench | Software engineering | Evaluates LLMs on issue-resolving tasks across multiple programming languages using three methods (Agentless, SWE-agent, OpenHands) and analyzes performance across models and languages; includes 4,723 RL-oriented data samples for Multi-SWE-RL | 1632 tasks in main benchmark; 4723 RL instances | Not stated | doi.org |
| OmniGIRL | Software engineering | LLM-based resolution performance on GitHub issue resolution tasks across multiple languages, domains, and modalities | 959 task instances | 2025 | dl.acm.org |
| Rust-SWE-bench | Software engineering | Percentage of real-world repository-level Rust issues resolved by LLM-based agents on Rust-SWE-bench | 500 tasks | 2026 | dl.acm.org |
| SWE-bench | Software engineering | Percentage of task instances resolved by a model after applying a generated patch and running tests | 2294 tasks | 2023 | arxiv.org |
| SWE-bench / SWE-bench Lite | Software engineering | Evaluates ability of LLM-based SE agents to solve real-world GitHub issues using repository-level information, repository navigation, fault locating, debugging, code generation and program repair; reports resolved instances and patch application rate on SWE-bench Lite and SWE-bench datasets | 300 task instances (SWE-bench Lite); multiple repositories across SWE-bench; pro | 2024 | arxiv.org |
| SWE-bench Live | Software engineering | Evaluates real-world bug-fixing capabilities of code LLMs and code agents using a live-updatable SWE-bench-Live benchmark with 1,890 tasks across 223 repositories; includes Docker images, automated curation pipeline, and empirical evaluation | 1,890 tasks across 223 repositories | 2026 | dl.acm.org |
| SWE-bench Verified | Software engineering | Evaluates AI models’ ability to solve real-world software issues using a human-validated subset of SWE-bench with 500 samples and accompanying annotations, plus an evaluation harness and performance metrics on SWE-bench Verified | 500 samples (SWE-bench Verified); annotations for 1,699 samples used to produce | 2024 | openai.com |
| SWE-bench-java | Software engineering | Resolved Rate (%) of issues resolved by LLM-based approaches on SWE-bench-java-verified | likely 2,294 issues from 12 Python libraries in SWE-bench; this variant SWE-benc | 2024 | arxiv.org |
| SWE-MERA | Software engineering | Evaluates agentic LLM performance on software engineering tasks via a dynamic, continuously updated benchmark with real-world GitHub issues and quality validation | approximately 10,000 potential tasks with 728 samples currently available | 2025 | arxiv.org |
| SWT-Bench | Software engineering | Tests generation effectiveness for issue reproduction, coverage changes, reproduction rate, and precision of proposed fixes; evaluates Code Agents for code repair vs test generation | over 1,900 samples | 2024 | arxiv.org |
| ADU-Bench | Speech and audio | Evaluates open-ended audio dialogue understanding for large audio-language models (LALMs) across 4 datasets, 3 scenarios, 12 skills, 9 languages, and 4 ambiguity categories; reports performance of 13 LALMs | over 20,000 open-ended audio dialogues (specifically 20,715) | Not stated | doi.org |
| AHELM | Speech and audio | Holistic evaluation of audio-language models across 10 aspects including audio perception, knowledge, reasoning, emotion detection, bias/fairness, multilinguality, robustness, toxicity, and safety; standardized prompts, inference parameters, and evaluation metrics; compares 14 ALMs and baselines | 14 | 2025 | arxiv.org |
| AIR-Bench | Speech and audio | Evaluates large audio-language models' generative comprehension across audio understanding and instruction-following tasks using both single-choice and open-ended QA evaluated with GPT-4-based scoring and human evaluation | ~19k single-choice questions (foundation benchmark) and ~2k open-ended QA instan | Not stated | doi.org |
| Audio MultiChallenge | Speech and audio | Average Pass Rate (APR) across four axes (Inference Memory, Instruction Retention, Self Coherence, Voice Editing) for end-to-end spoken dialogue systems in natural multi-turn audio interactions | 452 conversations, 1,712 instance-specific rubrics | Not stated | doi.org |
| AudioBench | Speech and audio | Evaluates Audio Large Language Models (AudioLLMs) across three aspects: speech understanding, audio scene understanding, and voice understanding (paralinguistic); uses 8 tasks and 26 datasets with 400+ hours of audio and multiple evaluation metrics | 8 tasks; 26 datasets (7 newly proposed); 400+ hours of audio across 100k+ sample | 2024 | arxiv.org |
| AudioMarathon | Speech and audio | Assess long-context audio understanding and inference efficiency for long-form audio; evaluate task performance and inference efficiency; study token pruning and KV cache eviction | Not stated | 2025 | arxiv.org |
| CodecBench | Speech and audio | Evaluates audio codec performance from acoustic and semantic perspectives across four data domains | Not stated | 2025 | arxiv.org |
| DASB | Speech and audio | Benchmarks discrete audio tokens across discriminative and generative speech tasks to compare semantic, compression, and hybrid tokenizers | Not stated | 2024 | arxiv.org |
| EXAM$^2$ | Speech and audio | Evaluates state-of-the-art open-source and proprietary LALMs as well as multimodal LLMs to reveal performance gaps in multilingual and cross-modal understanding | Not stated | 2026 | semanticscholar.org |
| GlobeAudio | Speech and audio | Evaluates naturalistic audio understanding and higher-level auditory reasoning across multilingual, multicultural contexts using 5,637 MCQs in 6 languages; benchmarks LALMs and ASR→LLM pipelines under natural acoustic conditions | 5637 | 2026 | arxiv.org |
| ISA-Bench | Speech and audio | Assesses instruction sensitivity of large audio language models (LALMs) across instruction description, output format, and task composition; evaluates compliance and accuracy under instruction variations on five atomic tasks (ASR, S2TT, SER, GR, AAC) using task-native metrics and compliant replacements | Not stated | 2025 | arxiv.org |
| MMSU | Speech and audio | Assessment of spoken language understanding and complex reasoning abilities of SpeechLLMs across 47 tasks (5,000 audio-question-answer triplets) | 5000 | 2025 | arxiv.org |
| mSTEB | Speech and audio | Evaluates LLMs on multilingual speech and text tasks including language identification, text classification, question answering, and translation across speech and text modalities; compares high-resource and low-resource languages | Not stated | 2025 | arxiv.org |
| MuLA-Bench | Speech and audio | Assesses multilingual long-form audio understanding across language, domain, and acoustic tasks using a suite of 5,038 open-ended questions over 1,769 in-the-wild recordings (16 languages, eight domains) and multiple evaluation tracks; evaluates audio-language models and diagnostics across long-range retrieval, clock alignment, and factual grounding | 5038 questions; 1769 recordings; 1377.9 hours | 2026 | arxiv.org |
| PolySpeech-100 | Speech and audio | Evaluates native-level speech comprehension across 110 linguistic variants and analyzes performance of 22 state-of-the-art models on open-source vs commercial systems, including dialect robustness and zero-shot prompting effects | 110 linguistic variants (languages/dialects) | 2026 | dl.acm.org |
| SCENEBench | Speech and audio | Assess four audio understanding tasks (background sound understanding, noise localization, cross-linguistic speech understanding, vocal characterizer recognition) plus model latency | Not stated | Not stated | doi.org |
| UltraEval-Audio | Speech and audio | Comprehensive, multi-dimensional evaluation framework for audio foundation models including one-command evaluation, multilingual support (10 languages), 14 task categories, 24 models, 36 benchmarks; three-dimensional codec evaluation (semantic accuracy, timbre fidelity, acoustic quality); Chinese evaluation benchmarks SpeechCMMLU and SpeechHSK; standard metrics include WER, BLEU, accuracy, ASR-WER/UTMOS for codec, cosine similarity for timbre, etc | 36 benchmarks, 14 tasks, 10 languages, 24 models | Not stated | doi.org |
| URO-Bench | Speech and audio | Evaluates Understanding, Reasoning, and Oral conversation in end-to-end spoken dialogue models across basic and pro tracks; uses 4 metrics: task accomplish score, UTMOS, WER/CER, and first packet latency | 40 | Not stated | doi.org |
| VCB Bench | Speech and audio | Evaluates audio-grounded LLMs along instruction following, knowledge understanding, and robustness using authentic human speech; assesses Chinese voice conversation capabilities across speech-level control, general knowledge, reasoning, daily dialogue, and robustness to perturbations | Not stated | 2025 | doi.org |
| ALLVB | Video | Nine integrated video understanding tasks converted into video QA formats; 1,376 long videos across 16 categories; 252k Q&As; average ~2 hours per video; 9 question templates for 9 task types; automated GPT-4o annotation pipeline; 183 Q&As per video; benchmarks outperform across long-video duration and Q&A counts compared to prior works | 1,376 videos; 252,000 Q&As; 16 categories; average video length ~2 hours | Not stated | doi.org |
| E.T. Bench | Video | Assesses event-level video understanding across referring, grounding, dense captioning, and complex understanding using 7.8K high-quality annotations over 7.3K samples, 12 tasks, 20 sub-tasks, 8 domains, evaluated across 20 models | 7.3K samples, 7K videos (251.4h), 12 tasks, 8 domains, 20 models evaluated | 2024 | arxiv.org |
| FAVOR-Bench | Video | Assesses fine-grained video motion understanding with close-ended (8,184 MCQ pairs across six sub-tasks) and open-ended (GPT-assisted) evaluation; includes FAVOR-Train for fine-grained motion annotations; evaluates state-of-the-art MLLMs on temporal dynamics in video motions | 1,776 videos in FAVOR-Bench; 8,184 MCQ pairs; 17,152 videos in FAVOR-Train | 2026 | dl.acm.org |
| InfiniBench | Video | Tests long-form video understanding across grounding-based and reasoning-based skills using over 1,000 hours of video content and a large set of QA pairs in both multiple-choice and open-ended formats | around totalSampleNumber questions (approximate in text) | Not stated | doi.org |
| LongVideoBench | Video | Multimodal video question answering on long-context inputs with referring reasoning and 6,678 multiple-choice questions across 3,763 long videos | 3,763 videos; 6,678 questions | 2024 | arxiv.org |
| LVBench | Video | Assesses long-form video understanding across six core temporal understanding capabilities and information extraction in long videos, evaluating multimodal models' long-term memory and extended comprehension | Not stated | Not stated | doi.org |
| Meta-Video | Video | Assesses the impact of evaluation configurations on video understanding benchmarks and provides a standardized evaluation protocol plus a 1,000-question Meta-Video benchmark across 791 videos to measure progress reliably and efficiently | 1000 questions; 791 videos; five models; seven benchmarks | 2026 | arxiv.org |
| MMBench-Video | Video | Quantitative benchmark to evaluate LVLMs’ proficiency in long-form video understanding with free-form questions and GPT-4-based automated scoring, focusing on temporal reasoning, contextual richness, and temporal indispensability | approximately 600 web videos, ~2,000 QA pairs, 16 categories, videos 30s to 6 mi | 2024 | arxiv.org |
| MT-Video-Bench | Video | Six core capabilities spanning perceptivity and interactivity evaluated over 987 multi-turn dialogues and 5,805 QA pairs | 987 dialogues, 5,805 QA pairs | Not stated | doi.org |
| MVBench | Video | Temporal understanding and comprehensive evaluation of video understanding across 20 tasks using a static-to-dynamic transformation and multiple-choice QA; baseline VideoChat2 with multi-modal training; evaluation fairness with ground-truth annotations; open-source data and models | 20 tasks | Not stated | doi.org |
| MVU-Eval | Video | Evaluates eight core competencies of multi-video perception and reasoning across 1,824 QA pairs spanning 4,959 videos, organized into Perception and Reasoning protocols | 1824 | 2025 | arxiv.org |
| Needle In A Video Haystack | Video | Evaluates video understanding capabilities via synthetic benchmark tasks (retrieval, ordering, counting) focusing on temporal perception, chronological ordering, and spatio-temporal coherence | VNBench tasks across video understanding dimensions; includes retrieval, orderin | 2024 | arxiv.org |
| Neptune | Video | Open-ended Gemma Equivalence Metric (GEM) for evaluating long video understanding; multiple choice and open-ended evaluation protocols; dataset of 3,268 QAD annotations for 2,405 videos; NEPTUNE-MMH subset; evaluation against open-source and proprietary VideoQA models | ~3,268 QAD annotations on 2,405 videos plus subset NEPTUNE-MMH | 2024 | arxiv.org |
| OmniCompass | Video | Evaluates omni-modal video grounding and understanding across two tracks (OmniCompass-VL and OmniCompass-AV) with tasks including temporal grounding, spatial grounding, and question answering; covers perception and reasoning categories; provides two paired settings (text-only and text+image); assesses model capabilities on visual, audio-visual, and grounding tasks | 820 videos with 2,460 instances for OmniCompass-VL; 240 videos with 720 instance | 2026 | arxiv.org |
| OmniVideoBench | Video | Tests audio-visual understanding and multimodal reasoning capabilities of MLLMs through 1000 QA pairs across 13 task types with step-by-step reasoning traces on 628 videos | 1000 questions (QA pairs) across 13 task types from 628 videos | 2025 | arxiv.org |
| Q-Bench-Video | Video | Evaluates video quality understanding of large multimodal models (LMMs) across video source types, distortions, and question types; 2,378 QA pairs tested on 12 open-source & 5 proprietary LMMs | 2378 QA pairs; 12 open-source & 5 proprietary LMMs | 2025 | doi.org |
| TUNA | Video | Evaluates temporal understanding on dense dynamic videos via two tasks: TUNA-CAP (captioning) and TUNA-MCQ (VQA), across multiple dimensions including visual characteristics, temporal elements, and video complexities; provides automated evaluation pipeline and human-aligned metrics | ~1,000 videos (TUNA-1K) with 1,432 MCQ questions and captioning annotations | 2025 | arxiv.org |
| Video-Bench | Video | Evaluates Video-LLMs across three capability levels and computes final scores using an automatic evaluation toolkit | 10 tasks in benchmark; evaluation of 8 Video-LLMs | 2023 | arxiv.org |
| Video-Index | Video | Audits and benchmarks evaluating video understanding models against shortcut attacks and robustness across 4 capability groups; release of 210 hardest verified items per group and 840 items overall; compares model performance under fixed inputs and with agent tools | 4 capability groups; 210 items per group (audited set); 840 items overall; 505,5 | 2026 | arxiv.org |
| Video-MME | Video | Multimodal video understanding evaluation across diverse video domains, modalities (video, subtitles, audio), duration ranges, with QA/correct answers; reports average accuracy of models (e.g., Gemini 1.5 Pro ~75%, GPT-4o ~71.9%) | 900 videos; 2,700 QA items (3 per video) | 2024 | arxiv.org |
| VIDEOEVAL-PRO | Video | Open-ended short-answer QA performance on long video understanding, assessing segment-level and full-video understanding via perception and reasoning tasks | 1,289 question-answer pairs based on 465 videos | 2025 | arxiv.org |
Methodology
The directory was compiled in October 2026 from research papers, published since January 2023, that introduce a benchmark or evaluation dataset for language models or AI agents, collected across 24 task areas. Each paper was read for the benchmark's name, what it measures and its size. Papers that do not introduce a benchmark were removed, and duplicates were merged by name.
Thirty-four widely cited benchmarks that predate or sit outside that collection, including MMLU, GPQA, HumanEval, MATH, GSM8K and GAIA, were added individually, and their paper links were checked against arXiv.
Field definitions. Task area is the area under which the benchmark was collected, so a benchmark can fit more than one area but appears once. Size is the number of tasks or questions as the paper states it. Year comes from the arXiv identifier or the publication date.
Corrections. Send missing benchmarks or corrections to editors@thefrontierwire.com with a link to the paper.
Questions readers ask
What are AI benchmarks?
Standardized test sets used to measure what AI models can do, such as answering exam questions (MMLU, GPQA), writing code that passes tests (HumanEval, SWE-bench), solving math problems (MATH, FrontierMath) or completing multi-step tasks as an agent (GAIA, WebArena).
How many AI benchmarks are there?
This directory lists 481, and it is not exhaustive. 63% of the dated benchmarks here were introduced in 2025 or 2026.
Which benchmarks matter most for coding?
HumanEval and MBPP test short functions; SWE-bench and SWE-bench Verified test fixing real GitHub issues; LiveCodeBench refreshes its problems to limit contamination. Filter the table by Coding or Software engineering for the full set.
What is benchmark saturation?
A benchmark is saturated when top models score near its ceiling, so it no longer separates them. That is one reason new, harder benchmarks keep appearing.
Can I use this data?
Yes. The table is published under CC BY 4.0. Download the CSV or JSON and reuse it with attribution and a link to this page.
Sources
Cite this dataset
The Frontier Wire. (2026). AI benchmarks directory: 481 LLM and agent benchmarks by task [Data set]. https://thefrontierwire.com/data/ai-benchmarks/. Updated October 29, 2026.
Link to it
<a href="https://thefrontierwire.com/data/ai-benchmarks/">AI benchmarks directory: 481 LLM and agent benchmarks by task</a> (The Frontier Wire, updated October 29, 2026)