ArguAna
- Tests
- Retrieval of counterarguments for a given argumentative claim.
- Best for
- Diagnosing semantic retrieval beyond topical similarity.
- Classification
- benchmark · current · open
AI evaluation domain · 13 source-backed records
RAG, Retrieval, and Embedding Benchmarks
RAG evaluation should separate retrieval quality from answer quality. This collection includes embedding and retrieval suites, grounded generation tests, long-context probes, and document understanding benchmarks.
Decision guidance
How should teams evaluate a RAG system?
Measure retrieval with labeled query-document pairs and metrics such as Recall@k or MRR. Evaluate generation separately for faithfulness, answer relevance, citation support, refusals, and domain-specific failure modes.
13
Benchmarks in this domain
Sources first · alphabetical collection order
Runnability describes access to a usable repository, dataset, or harness—not whether setup is easy.
2026-07-15
BEIR
- Tests
- Heterogeneous information-retrieval benchmark for zero-shot retrieval across many datasets and domains.
- Best for
- Retriever selection for RAG systems
- Classification
- benchmark · current · open
BRIGHT
- Tests
- Retrieval where finding relevant evidence requires multi-step reasoning.
- Best for
- Selecting retrievers for difficult research and professional search tasks.
- Classification
- benchmark · emerging · open
CRAG
- Tests
- Comprehensive RAG benchmark with factual QA and mock APIs for retrieval.
- Best for
- RAG factuality and retrieval stress tests
- Classification
- benchmark · current · open
DocVQA
- Tests
- Visual question answering over document images.
- Best for
- PDF, OCR, and document-agent evaluation
- Classification
- benchmark · current · mixed
InfiniteBench
- Tests
- Super-long-context benchmark beyond 100k tokens.
- Best for
- Context-window stress testing
- Classification
- benchmark · current · open
LongBench / LongBench v2
- Tests
- Long-context understanding and reasoning across documents and realistic multitask scenarios.
- Best for
- Long-context model screening
- Classification
- benchmark · current · open
MDPBench
- Tests
- Real-world document parsing across languages, layouts, and content structures.
- Best for
- Comparing document intelligence pipelines serving multilingual corpora.
- Classification
- benchmark · emerging · open
MTEB
- Tests
- Massive Text Embedding Benchmark for comparing embedding models across retrieval, clustering, classification, reranking, and semantic similarity tasks.
- Best for
- Embedding and retrieval model selection
- Classification
- benchmark · current · open
olmOCR-bench
- Tests
- OCR fidelity across diverse PDF pages using thousands of document-level unit tests.
- Best for
- Choosing extraction components before evaluating downstream document RAG.
- Classification
- benchmark · emerging · open
OmniDocBench
- Tests
- Document parsing benchmark for OCR, layout, table, formula, and reading-order extraction.
- Best for
- Document parsing for RAG pipelines
- Classification
- benchmark · current · open
ParseBench
- Tests
- Document parser accuracy on enterprise layouts and structured content.
- Best for
- Selecting a parser before measuring retrieval and grounded-answer quality.
- Classification
- benchmark · emerging · open
RAGBench
- Tests
- Explainable RAG benchmark across documents, retrieval, generation, and attribution.
- Best for
- RAG system evaluation
- Classification
- benchmark · current · open