Overview
The benchmark is immediately useful for testing multi-turn RAG end-to-end; retrieval and metric gaps are well supported by experiments and human annotation.
Citations2
Evidence Strength0.80
Confidence0.80
Risk Signals8
Trust Signals
Findings with numeric evidence: 8/8
Findings with evidence refs: 8/8
Results with explicit delta: 4/6
Reproducibility
Status: Code + data available
Open source: Partial
At A Glance
Cost impact: 40%
Production readiness: 60%
Novelty: 50%
Why It Matters For Business
Multi-turn customer or assistant flows need better retrievers and grounded generators; MTRAG shows current systems miss later-turn context and unanswerables, causing wrong or misleading answers that harm trust.
Who Should Care
Summary TLDR
MTRAG is a human-created benchmark of 110 multi-turn conversations (842 tasks) across four domains designed to evaluate full retrieval-augmented generation (RAG) pipelines. Conversations average 7.7 turns and 16.9 unique relevant passages. The authors evaluate lexical, dense and sparse retrievers and nine LLMs under three retrieval settings (perfect reference, reference+RAG, full RAG). Key findings: query rewriting improves retrieval; retrieval and generation both degrade on later turns and on unanswerable/non-standalone questions; current automatic metrics partially correlate with humans; synthetic conversations (MTRAG-S) are shorter and simpler than human data. The benchmark and code are公開
Problem Statement
Current RAG benchmarks focus on single-turn cases or fix retrieval. Real conversations require repeated retrieval, handle non-standalone turns and unanswerable queries, and need evaluation of both retriever and generator together. We need a multi-turn, end-to-end human benchmark to surface these gaps.
Main Contribution
A human-generated, end-to-end multi-turn RAG benchmark (MTRAG): 110 conversations, 842 tasks across 4 domains.
Evaluation of lexical, dense and sparse retrievers and 9 LLMs under three retrieval settings (Reference, Reference+RAG, Full RAG).
Key Findings
Benchmark scale and structure
Query rewriting improves retrieval
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Elser Recall@5 (query rewrite) | 0.58 (avg) | Elser Last-turn ~0.47 | ≈+0.11 | All domains, Table 3/15 | Table 3 and Table 15 report Elser R@5 with and without query rewrite. | Table 3 |
| Retrieval drop (first vs later turns) | R@5: 0.89 (first turn) vs 0.47 (later turns) | — | −0.42 | Elser, Table 4 (By Turn) | Table 4 shows large R@5 gap between Turn 1 and subsequent turns. | Table 4 |
What To Try In 7 Days
Add a query-rewrite step that converts the current user turn into a standalone query before retrieval.
Evaluate your system on a small slice of MTRAG (download from GitHub) to surface later-turn failures.
Condition your generator to prefer honest refusals (explicit 'I don't know') when retrieved passages lack evidence.
Agent Features
Memory
Tool Use
Reproducibility
Risks & Boundaries
Limitations
Relatively small human dataset (110 conversations), so rare failure modes may be underrepresented.
Creation used Elser during annotation, which may bias passages toward that retriever.
When Not To Use
As the sole evaluation for extractive-only QA or single-turn tasks.
To judge multilingual or adversarial-turn robustness (not included).
Failure Modes
Automatic metrics can favor machine-generated text and misalign with humans (low correlations for some metrics).
Retrievers return stale or repeated passages across turns if full conversation is naively used.