Aider Polyglot
- Tests
- Multi-language code editing and test-passing benchmark for practical coding assistants.
- Best for
- Code edit models across languages
- Classification
- benchmark · current · mixed
AI evaluation domain · 17 source-backed records
Coding and Software Engineering Benchmarks
These benchmarks cover code generation, repository-level issue resolution, terminal work, debugging, and multi-hour software tasks. The strongest choices run generated work against tests in a reproducible environment.
Decision guidance
What are the best benchmarks for coding agents?
Use repository repair for end-to-end coding agents, terminal benchmarks for shell-native execution, and fresh problem sets for code generation. Record the agent scaffold, tool permissions, time budget, and pass criteria with every result.
17
Benchmarks in this domain
Sources first · alphabetical collection order
Runnability describes access to a usable repository, dataset, or harness—not whether setup is easy.
2026-07-15
ALE-Bench
- Tests
- Long-horizon algorithm-engineering contest tasks.
- Best for
- Optimization and algorithm engineering agents
- Classification
- benchmark · current · open
BigCodeBench
- Tests
- Software-engineering-oriented code-generation tasks designed to go beyond HumanEval and MBPP.
- Best for
- Modern code-generation baselines
- Classification
- benchmark · current · open
CursorBench
- Tests
- Cursor's proprietary/internal offline eval suite from real Cursor sessions, focused on correctness, code quality, efficiency, and interaction behavior.
- Best for
- Editor-agent evaluation context
- Classification
- benchmark · current · commercial
DeepSWE
- Tests
- Original long-horizon software engineering tasks across several languages with isolated environments and verifiers.
- Best for
- Emerging SWE agent evaluation
- Classification
- agent benchmark · emerging · open
GBA-Eval
- Tests
- Single high-quality long-horizon Game Boy Advance SWE eval case; useful signal, not a complete coding benchmark.
- Best for
- Experimental coding-agent signal
- Classification
- benchmark · current · mixed
HumanEval
- Tests
- Classic Python function synthesis benchmark with unit tests.
- Best for
- Legacy coding baseline
- Classification
- benchmark · legacy · open
LiveCodeBench
- Tests
- Recent contest-style coding problems with contamination-aware releases.
- Best for
- Code-generation and algorithmic coding signal
- Classification
- benchmark · current · open
MBPP
- Tests
- Mostly Basic Programming Problems for simple Python coding tasks.
- Best for
- Small-model and baseline coding checks
- Classification
- benchmark · legacy · open
NVIDIA ComputeEval
- Tests
- Correctness and runtime performance of generated GPU compute code.
- Best for
- Evaluating coding systems that optimize kernels or write CUDA.
- Classification
- benchmark · emerging · open
ProgramBench
- Tests
- Rebuilding programs from compiled binaries and documentation.
- Best for
- Reverse engineering and deep coding
- Classification
- benchmark · current · open
SciCode
- Tests
- Scientist-curated coding problems from real natural-science contexts.
- Best for
- Scientific coding agents
- Classification
- benchmark · current · open
SWE-bench
- Tests
- Real GitHub issues that require modifying repositories and passing tests.
- Best for
- Software engineering agent capability
- Classification
- agent benchmark · current · open
SWE-bench Pro
- Tests
- Harder held-out/private-repo software engineering tasks intended to reduce contamination.
- Best for
- Frontier SWE agents and benchmark saturation checks
- Classification
- agent benchmark · current · commercial
SWE-bench Verified
- Tests
- Human-filtered subset of SWE-bench with higher-quality real issue tasks.
- Best for
- Default SWE benchmark slice
- Classification
- agent benchmark · current · open
SWE-Marathon
- Tests
- Ultra-long-horizon software engineering tasks.
- Best for
- Long-horizon coding agents
- Classification
- agent benchmark · emerging · mixed
Terminal-Bench 2.0
- Tests
- Terminal-based task execution benchmark for agents working in shell environments.
- Best for
- CLI agents and software task execution
- Classification
- agent benchmark · current · mixed