BioASQ
- Tests
- Biomedical semantic indexing and QA challenges.
- Best for
- Biomedical search and QA
- Classification
- benchmark · current · mixed
AI evaluation domain · 15 source-backed records
Healthcare and Biomedical AI Benchmarks
Healthcare benchmarks cover medical knowledge, clinical reasoning, biomedical research, safety, and agentic workflows. High aggregate scores do not establish clinical readiness.
Decision guidance
How should healthcare AI benchmarks be interpreted?
Separate factual knowledge from clinical workflow reliability. Validate with qualified domain experts, patient-specific safety constraints, current guidance, abstention behavior, and the exact decisions the system is allowed to influence.
15
Benchmarks in this domain
Sources first · alphabetical collection order
Runnability describes access to a usable repository, dataset, or harness—not whether setup is easy.
2026-07-15
CliBench
- Tests
- Clinical decisions on diagnoses, procedures, lab-test orders, and prescriptions with structured output ontologies.
- Best for
- Clinical decision granularity
- Classification
- benchmark · emerging · mixed
ClinicBench
- Tests
- Clinical language generation, understanding, and reasoning tasks including open-ended clinical decision-making.
- Best for
- Clinical decision support research
- Classification
- benchmark · current · open
HealthBench
- Tests
- Realistic health conversations with physician-created rubrics.
- Best for
- Health assistant response quality
- Classification
- benchmark · current · open
HealthBench Professional
- Tests
- Professional healthcare benchmark variant focused on clinician workflows and higher-stakes medical tasks.
- Best for
- Clinical workflow response quality
- Classification
- benchmark · emerging · mixed
MedAgentBench
- Tests
- Virtual EHR/FHIR environment with clinically relevant tasks requiring agents to retrieve, record, order, and act in medical-record settings.
- Best for
- Clinical workflow agents
- Classification
- agent benchmark · current · open
MedCalc-Bench
- Tests
- Medical calculation tasks for evaluating whether LLMs can serve as clinical calculators.
- Best for
- Medical arithmetic and calculators
- Classification
- benchmark · current · open
MedHELM
- Tests
- HELM-style medical evaluation framework with a clinical taxonomy, healthcare task categories, and public/gated/private medical benchmarks.
- Best for
- Holistic medical evals
- Classification
- harness · current · mixed
MedMCQA
- Tests
- Large-scale medical entrance-exam question set across healthcare subjects.
- Best for
- Medical multiple-choice baseline
- Classification
- benchmark · legacy · open
MedQA
- Tests
- Medical board-style QA for clinical knowledge and exam reasoning.
- Best for
- Medical QA baseline
- Classification
- benchmark · legacy · open
MultiMedQA
- Tests
- Composite medical QA benchmark spanning MedQA, MedMCQA, PubMedQA, LiveQA, MedicationQA, MMLU clinical topics, and HealthSearchQA.
- Best for
- Medical QA suite comparison
- Classification
- benchmark · current · mixed
OpenMed
- Tests
- Open-source healthcare NLP toolkit, model hub, and curated clinical/biomedical resources; useful source layer rather than a benchmark leaderboard.
- Best for
- Medical NLP resources and PHI-safe extraction
- Classification
- toolkit · current · open
PubMedQA
- Tests
- Biomedical research QA requiring reasoning over PubMed abstracts.
- Best for
- Biomedical literature QA
- Classification
- benchmark · current · open
VoxClinBench
- Tests
- Clinical voice benchmark with cross-lingual expansion.
- Best for
- Medical speech and clinical voice agents
- Classification
- benchmark · emerging · mixed
χ-Bench
- Tests
- Long-horizon healthcare workflows with policy, process, and tool constraints.
- Best for
- Evaluating clinical operations agents beyond medical question answering.
- Classification
- benchmark · emerging · open