AgentHarm
- Tests
- Harmful multi-step agent tasks for evaluating agent safety under tool-use settings.
- Best for
- Agent safety research
- Classification
- agent benchmark · current · open
AI evaluation domain · 11 source-backed records
AI Safety and Security Benchmarks
Safety and security benchmarks probe policy behavior, misuse resistance, harmful capabilities, and cyber tasks. Some sources are intentionally restricted or hazardous and require careful handling.
Decision guidance
How should teams use AI safety and security benchmarks?
Use least-privilege environments, approved datasets, controlled disclosure, and qualified reviewers. Keep capability evaluation separate from release policy, and never treat a public safety score as evidence that a deployed system is safe in your context.
11
Benchmarks in this domain
Sources first · alphabetical collection order
Runnability describes access to a usable repository, dataset, or harness—not whether setup is easy.
2026-07-15
CyberSecEval / PurpleLlama
- Tests
- Cybersecurity risk and capability evaluations for LLMs.
- Best for
- Defensive cyber-risk evaluation
- Classification
- benchmark · current · open
EVMbench
- Tests
- Sandboxed smart-contract benchmark for detecting, patching, and exploiting EVM vulnerabilities.
- Best for
- Smart-contract security agents
- Classification
- agent benchmark · current · open
ExploitBench
- Tests
- Capability ladder from vulnerability identification toward exploitation outcomes.
- Best for
- Cyber capability research with defensive caution
- Classification
- benchmark · current · mixed
HarmBench
- Tests
- Robust refusal and red-teaming benchmark for evaluating whether models comply with harmful requests.
- Best for
- Safety refusal and red-team evaluation
- Classification
- benchmark · current · open
MedSafetyBench
- Tests
- Medical safety benchmark for risky clinical responses and unsafe medical advice.
- Best for
- Medical safety evaluation
- Classification
- benchmark · current · open
OS-Harm
- Tests
- Safety benchmark for computer-use agents and harmful action sequences.
- Best for
- Computer-use safety
- Classification
- agent benchmark · current · open
SafetyBench
- Tests
- Broad static safety benchmark for LLMs across safety categories.
- Best for
- Static safety baseline
- Classification
- benchmark · legacy · open
SciRisk-Bench
- Tests
- AI-for-science safety across explicit risk dimensions and scientific disciplines.
- Best for
- Mapping scientific-agent failure modes to concrete risk categories.
- Classification
- benchmark · emerging · open
SEC-bench
- Tests
- Software-security tasks including vulnerability discovery, patching, and proof-of-concept style scenarios.
- Best for
- Security engineering agents
- Classification
- agent benchmark · current · open
SOSBench
- Tests
- Safety alignment on 3,000 regulation-grounded prompts across six high-risk scientific domains.
- Best for
- Testing refusal and policy behavior on knowledge-intensive scientific misuse.
- Classification
- benchmark · emerging · open