Overview
Production Readiness
0.6
Novelty Score
0.5
Cost Impact Score
0.4
Citation Count
2
Why It Matters For Business
Multi-turn customer or assistant flows need better retrievers and grounded generators; MTRAG shows current systems miss later-turn context and unanswerables, causing wrong or misleading answers that harm trust.
Summary TLDR
MTRAG is a human-created benchmark of 110 multi-turn conversations (842 tasks) across four domains designed to evaluate full retrieval-augmented generation (RAG) pipelines. Conversations average 7.7 turns and 16.9 unique relevant passages. The authors evaluate lexical, dense and sparse retrievers and nine LLMs under three retrieval settings (perfect reference, reference+RAG, full RAG). Key findings: query rewriting improves retrieval; retrieval and generation both degrade on later turns and on unanswerable/non-standalone questions; current automatic metrics partially correlate with humans; synthetic conversations (MTRAG-S) are shorter and simpler than human data. The benchmark and code are公開
Problem Statement
Current RAG benchmarks focus on single-turn cases or fix retrieval. Real conversations require repeated retrieval, handle non-standalone turns and unanswerable queries, and need evaluation of both retriever and generator together. We need a multi-turn, end-to-end human benchmark to surface these gaps.
Main Contribution
A human-generated, end-to-end multi-turn RAG benchmark (MTRAG): 110 conversations, 842 tasks across 4 domains.
Evaluation of lexical, dense and sparse retrievers and 9 LLMs under three retrieval settings (Reference, Reference+RAG, Full RAG).
Analysis of automatic evaluation methods and a synthetic companion benchmark (MTRAG-S) to study scaling and metric gaps.
Key Findings
Benchmark scale and structure
Query rewriting improves retrieval
Retrieval is much weaker on later turns
Generators lag reference quality and struggle with unanswerables
Human preference favors annotated reference answers
Automatic metrics partially align with humans
Synthetic conversations differ from human data
Annotators heavily repaired LLM outputs during creation
Results
Elser Recall@5 (query rewrite)
Retrieval drop (first vs later turns)
Accuracy
Generation gap vs reference (RB_alg)
Human win-rate (reference vs models)
Annotator repairs during creation
Who Should Care
What To Try In 7 Days
Add a query-rewrite step that converts the current user turn into a standalone query before retrieval.
Evaluate your system on a small slice of MTRAG (download from GitHub) to surface later-turn failures.
Condition your generator to prefer honest refusals (explicit 'I don't know') when retrieved passages lack evidence.
Agent Features
Memory
- short-term conversational context
- retrieval memory (passages)
Tool Use
- retrieval
- query_rewrite
- passage editing
Reproducibility
Code Available
Data Available
Open Source Status
- partial
Risks & Boundaries
Limitations
- Relatively small human dataset (110 conversations), so rare failure modes may be underrepresented.
- Creation used Elser during annotation, which may bias passages toward that retriever.
- Synthetic companion (MTRAG-S) differs in turn length and passage diversity and may not fully substitute human data.
When Not To Use
- As the sole evaluation for extractive-only QA or single-turn tasks.
- To judge multilingual or adversarial-turn robustness (not included).
Failure Modes
- Automatic metrics can favor machine-generated text and misalign with humans (low correlations for some metrics).
- Retrievers return stale or repeated passages across turns if full conversation is naively used.
- Generators hallucinate on unanswerable questions and in later turns.
Core Entities
Models
- Llama 3.1 8B
- Llama 3.1 70B
- Llama 3.1 405B
- Mixtral 8x22B
- GPT-4o
- GPT-4o-mini
- Command R+ (104B)
- Qwen 2.5 (7B)
- Qwen 2.5 (72B)
Metrics
- RB_alg
- RB_llm
- RL_F
- Recall@k
- nDCG@k
- Rouge-L
- Bert-K-Prec
Datasets
- CLAPNQ
- FiQA
- Govt
- Cloud
- MTRAG-S
Benchmarks
- MTRAG