AI Benchmarks

Crowdsourced, randomized battle platform for LLMs based on anonymous human preference voting. Elo rating system.

Coding-specific subset of Chatbot Arena.

HuggingFace's comprehensive open-source model evaluation across multiple benchmarks.

Updated HuggingFace leaderboard with harder benchmarks: MMLU-Pro, GPQA, MATH, MuSR, IFEval, BBH.

Composite index across 39 benchmarks measuring overall AI capability.

Stanford's comprehensive framework for evaluating language models across 42 scenarios.

Automated evaluation of instruction-following LLMs using GPT-4 as judge.

80 high-quality multi-turn questions across 8 categories. Uses GPT-4 as judge.

Automated version of Chatbot Arena using GPT-4 as judge. 500 hard questions.

Benchmarking LLMs with harder, real-world user queries from the wild.

Subset of Chatbot Arena with the most challenging prompts.

Human-verified subset of SWE-bench. Measures ability to resolve real GitHub issues from popular Python repositories.

Extension of SWE-bench for repositories requiring UI, screenshots, or visual context to resolve issues.

Lightweight version of SWE-bench with 300 curated instances for faster evaluation.

Enhanced SWE-bench with improved test setup and removed flaky tests for more reliable evaluation.

164 hand-written Python programming problems with unit tests. The original code generation benchmark.

Enhanced HumanEval with 80x more test cases per problem for more rigorous evaluation.

974 entry-level Python programming problems with test cases. Tests basic programming concepts.

Enhanced MBPP with 35x more test cases for robust evaluation of code generation.

Holistic and contamination-free evaluation of LLMs for code. Continuously collects new problems to prevent data leakage.

Tests LLMs on 225 challenging Exercism coding exercises across C++, Go, Java, JavaScript, Python, and Rust.

Evaluates how effectively LLMs edit Python source files to complete 133 coding exercises from Exercism.

Evaluates LLMs on real-world terminal and shell command tasks. Tests ability to navigate file systems, run commands, and debug issues.

Comprehensive evaluation of LLMs' tool-use capabilities through the Model Context Protocol (MCP).

Benchmarks LLMs on navigating and using MCP tool ecosystems. Tests discovery and composition of tools.

Evaluates LLMs on generating scalable vector graphics from text descriptions.

Comprehensive benchmark for code generation with complex, real-world programming tasks requiring multiple function calls.

1000 real-world data science tasks across 7 libraries (NumPy, Pandas, SciPy, etc.).

800 Python functions for evaluating LLMs' ability to predict code output (CRUXEval-O) and understand inputs (CRUXEval-I).

Automated Programming Progress Standard. 10,000 coding problems from competitive programming platforms.

2359 competitive programming problems from Codeforces and other platforms, with test cases.

Multilingual version of HumanEval covering Python, Java, JavaScript, Go, and C++.

Translates HumanEval to 18 programming languages for multilingual code generation evaluation.

Evaluation suite for StarCoder models across diverse coding tasks.

12,751 examples of real-world text-to-SQL with complex databases. Tests real-world performance.

Evaluates instruction-following ability of code models through complex prompts.

HumanEval extended to 8 languages with code generation, explanation, and synthesis tasks.

835 real bugs from 17 Java projects for evaluating bug-fixing ability.

505 bugs from 97 Python projects for evaluating automated debugging.

Measures fluid intelligence — the ability to solve novel reasoning problems without prior knowledge.

Interactive reasoning benchmark — agents learn in novel turn-based environments.

AI2 Reasoning Challenge — Grade 3-9 science questions. The challenging subset of ARC.

203 challenging tasks from BIG-Bench that language models previously struggled with.

Beyond the Imitation Game Benchmark. 200+ tasks contributed by researchers worldwide.

57 subjects across STEM, humanities, social sciences. 15,000+ multiple-choice questions.

Updated MMLU with harder questions, 10 choices instead of 4, and reduced ambiguity.

Refined version of MMLU with corrected labels and improved question quality.

Graduate-level science questions in biology, chemistry, and physics. Expert-validated, adversarially filtered.

The main set of Graduate-level Program QA questions across sciences.

Adversarial filtering for natural language inference. Tests commonsense reasoning.

Large-scale adversarial dataset for commonsense reasoning, inspired by Winograd Schema Challenge.

817 questions that language models often answer falsely. Tests truthfulness and calibration.

26,000+ logical reasoning questions from Chinese civil service exams, translated and curated.

Questions requiring multi-step implicit reasoning strategies to answer.

Physical commonsense reasoning — understanding how the physical world works.

4th-grade science exam questions with an open book. Requires combining facts with reasoning.

Yes/no questions derived from Google search queries and Wikipedia paragraphs.

Difficult reasoning benchmark covering math, logic, coding, and general reasoning from contest-level problems.

Multistate soft reasoning requiring combining knowledge from multiple sentences with uncertainty.

The $600K competition for solving ARC-AGI. Tests general intelligence.

American Invitational Mathematics Examination problems. Tests advanced mathematical problem-solving.

2025 AIME competition problems for the most current math reasoning evaluation.

Extremely difficult research-level math problems. Tiers 1-3 and Tier 4.

8,500 grade school math word problems requiring multi-step reasoning.

12,500 competition math problems from AMC, AIME, and Olympiad levels.

Curated 500-problem subset of MATH for efficient benchmarking.

Multi-step mathematical reasoning for Minerva models. Graduate-level math problems.

Evaluates mathematical reasoning in visual contexts — charts, diagrams, geometry.

Multimodal math reasoning benchmark with visual elements essential for problem solving.

Symbolic variant of GSM8K testing robust mathematical reasoning with varied numbers.

Large-scale math dataset from math competitions with chain-of-thought solutions.

Enhanced GSM8K with test-time augmentations for more robust math evaluation.

37K math problems with explanations across arithmetic, algebra, and geometry.

Extremely difficult benchmark of expert-level questions across many domains. Designed to be challenging even for top AI systems.

College-level science benchmark covering physics, chemistry, math, and biology.

Biomedical question answering from PubMed abstracts. Yes/No/Maybe answers.

Graduate-level program-style science questions requiring deep domain expertise.

Medical licensing exam questions from USMLE. Tests medical knowledge at professional level.

11,500 questions requiring college-level subject knowledge and visual reasoning.

Enhanced MMMU with harder questions, more options, and reduced visual dependency.

Comprehensive multimodal benchmark with 3,000+ questions across 20 abilities.

19K multiple-choice questions across 12 evaluation dimensions for multimodal LLMs.

Evaluates multimodal models as integrated vision-language assistants.

Challenging multimodal benchmark requiring visual perception AND reasoning.

Science diagrams from grade school textbooks. Tests visual reasoning on diagrams.

9,600 questions about charts requiring visual and logical reasoning.

12,767 questions about document images requiring OCR and understanding.

Questions requiring reading and reasoning about text in images.

Comprehensive OCR evaluation covering text recognition, document understanding, and chart parsing.

Evaluates hallucination in vision-language models through visual question answering.

Evaluates object hallucination via binary yes/no questions about image contents.

Real-world visual question answering for autonomous driving and robotics applications.

Open-ended evaluation of visual instruction following with diverse real-world images.

Multi-turn visual conversation benchmark testing knowledge, reasoning, and perception.

466 questions requiring multi-step reasoning with real-world tools. Tests general-purpose assistants.

Comprehensive video multimodal evaluation across short, medium, and long videos.

9 tasks requiring deeper language understanding than GLUE. Includes coreference, QA, and more.

General Language Understanding Evaluation. 9 tasks for evaluating language understanding.

Cross-lingual benchmark covering 9 tasks across 40 languages.

Refined XTREME with 10 tasks across 50+ languages. More balanced evaluation.

13,948 multiple-choice questions across 52 subjects in Chinese.

67 tasks covering 68 subjects in Chinese for measuring massive multitask language understanding.

Chinese National College Entrance Exam (Gaokao) problems for evaluating AI systems.

Evaluates how well LLMs follow precise instructions with verifiable criteria.

Reading comprehension requiring discrete reasoning (counting, sorting, comparison).

Conversational question answering where questions depend on previous context.

7,000 conversations with 270,000 question-answer pairs across 7 domains.

Reading comprehension on full-length books and movie scripts. Tests narrative understanding.

Typologically diverse question answering in 11 languages.

Cross-lingual extractive QA dataset covering 11 languages.

Open-ended question answering requiring knowledge synthesis from multiple sources.

Evaluates LLMs as agents across 8 environments: OS, DB, KG, web, card games, etc.

Realistic web environment benchmark with 812 tasks across 4 websites.

Realistic computer operating system environment for evaluating AI agents.

Evaluates AI agents on realistic customer service scenarios with policy adherence.

16,464 real-world APIs from RapidAPI for evaluating tool-use capabilities.

Benchmark for evaluating tool-use capabilities with 73 API tools.

Evaluates AI agents' ability to browse and find specific information on the web.

100+ web interaction tasks for training and evaluating web agents.

Evaluates tool utilization capability — selection, evaluation, and execution.

Standardized evaluation of AI safety — tests resistance to jailbreaks and harmful content generation.

Comprehensive benchmark for trustworthiness of LLMs across 6 dimensions.

58,492 questions testing social biases across 11 categories.

100K naturally-occurring prompts for evaluating language model toxicity.

274K machine-generated and human-annotated toxic and benign texts across 13 demographics.

1,508 sentence pairs testing nine types of stereotypical biases.

2,282 sentences testing stereotypical biases across 4 categories.

11,435 multiple-choice questions for evaluating LLM safety across 7 categories.

Evaluates how effectively LLMs refuse harmful requests.

Chinese safety benchmark for evaluating AI ethics and safety.

Evaluates text embeddings across 8 tasks and 56+ datasets.

Heterogeneous benchmark for evaluating information retrieval across 18 datasets.

Benchmark for evaluating Retrieval-Augmented Generation systems.

Benchmark for evaluating and developing Corrective RAG systems.

Standard RL benchmark: 26 Atari games with only 100K environment interactions.

Continuous control benchmark with 30 tasks of varying difficulty.

16 procedurally generated environments for evaluating generalization in RL.

Open-world survival game benchmark for evaluating memory and long-horizon planning in RL.

Comprehensive benchmark for offline and online reinforcement learning.

14M images across 20K+ categories. The foundational vision benchmark.

82K images with object detection, segmentation, and captioning annotations.

20K images with pixel-level semantic segmentation annotations.

Autonomous driving dataset with 1.4M camera images, LiDAR, and radar.

1000 hours of read English speech from audiobooks. Standard ASR benchmark.

Mozilla's crowdsourced speech dataset across 100+ languages.

Speech processing benchmark covering ASR, SER, PR, SD, and more.

Annual shared task for machine translation. The gold standard for translation evaluation.

200 language translation benchmark with high-quality sentence pairs.

Parallel corpus covering 100 language pairs for translation training and evaluation.

162 legal tasks for evaluating LLM capabilities in legal reasoning.

Financial domain benchmark covering reasoning, knowledge, and calculation.

Benchmark for evaluating AI knowledge of sneaker culture and brand expertise.