EVA-Bench
- Tests
- End-to-end voice-agent evaluation framework for realistic simulated conversations and voice-specific failure modes.
- Best for
- Voice-agent quality beyond transcripts
- Classification
- benchmark · emerging · mixed
AI evaluation domain · 16 source-backed records
Multimodal and Voice AI Benchmarks
Multimodal and voice evals add perception, timing, interaction, and media quality to ordinary language-model evaluation. End-to-end latency and recoverability often matter as much as semantic accuracy.
Decision guidance
How should multimodal and voice agents be evaluated?
Measure perception and task completion separately, then add end-to-end checks for latency, interruptions, turn taking, tool use, channel noise, accessibility, and graceful recovery from misunderstood input.
16
Benchmarks in this domain
Sources first · alphabetical collection order
Runnability describes access to a usable repository, dataset, or harness—not whether setup is easy.
2026-07-15
MathVista
- Tests
- Mathematical reasoning over visual inputs.
- Best for
- Visual math reasoning
- Classification
- benchmark · current · open
MM-BrowseComp
- Tests
- Emerging multimodal browsing benchmark for web tasks where visual context matters.
- Best for
- Multimodal browsing agents
- Classification
- benchmark · emerging · mixed
MMBench
- Tests
- Broad multimodal model evaluation suite.
- Best for
- General VLM comparison
- Classification
- benchmark · current · open
MMMU / MMMU-Pro
- Tests
- Expert multimodal reasoning where images materially affect the answer.
- Best for
- Vision-language reasoning
- Classification
- benchmark · current · open
MultiVox
- Tests
- Voice assistants on spoken and visual cues including emotion, pitch, timbre, and ambient audio.
- Best for
- Evaluating omni assistants that must combine paralinguistic speech with images or video.
- Classification
- benchmark · emerging · open
Open ASR Leaderboard
- Tests
- Speech-to-text systems across public ASR datasets and efficiency measures.
- Best for
- Shortlisting open speech-recognition models with source-linked results.
- Classification
- scoreboard · emerging · hosted
PBench
- Tests
- Pixel-level visual grounding from referring expressions.
- Best for
- Specialist comparison of multimodal grounding and segmentation systems.
- Classification
- benchmark · emerging · open
ScreenSpot-Pro
- Tests
- Visual grounding of interface elements in high-resolution professional software.
- Best for
- Choosing vision-language models for computer-use agents.
- Classification
- benchmark · emerging · open
Vaani Benchmark
- Tests
- Hindi speech recognition across real acoustic and language conditions.
- Best for
- Selecting ASR systems for Hindi-language products.
- Classification
- benchmark · emerging · open
Video-MME
- Tests
- Video understanding benchmark across temporal and multimodal questions.
- Best for
- Video model comparison
- Classification
- benchmark · current · open
VLABench
- Tests
- Vision-language-action systems on primitive robotic manipulation tasks.
- Best for
- Comparing embodied models on reproducible task primitives.
- Classification
- benchmark · emerging · open
VocalBench
- Tests
- Speech-interaction benchmark for vocal communication and multi-round voice tasks.
- Best for
- Speech interaction models
- Classification
- benchmark · emerging · open
VoiceAgentBench
- Tests
- Speech-based agentic tasks with spoken queries, tool/function specifications, multi-turn dialogue, and safety cases.
- Best for
- Voice agents with tools
- Classification
- agent benchmark · emerging · open
VoiceBench
- Tests
- LLM-based voice assistant benchmark across speech QA, reasoning, instruction following, safety, and robustness.
- Best for
- Voice assistant model comparison
- Classification
- benchmark · current · open
WBench
- Tests
- Interactive video world models across multiple behavioral dimensions and metrics.
- Best for
- Comparing action-conditioned world models rather than passive video QA.
- Classification
- benchmark · emerging · open