Overview
The study is an applied evaluation using public models/datasets with clear ablations and released code; results reflect in-domain Wikipedia setups and zero-shot behavior, so expect moderate transfer costs to other domains.
Citations2
Evidence Strength0.80
Confidence0.82
Risk Signals9
Trust Signals
Findings with numeric evidence: 5/5
Findings with evidence refs: 5/5
Results with explicit delta: 4/4
Reproducibility
Status: Code + data available
Open source: Partial
At A Glance
Cost impact: 60%
Production readiness: 70%
Novelty: 40%
Why It Matters For Business
Multilingual RAG lets products answer factual questions in many languages by combining strong multilingual retrieval and tuned prompts, expanding reach and reducing wrong answers for non-English users.
Who Should Care
Summary TLDR
This paper builds and open-sources a zero-shot multilingual retrieval-augmented generation (mRAG) pipeline and measures which components matter. Using strong dense retrieval (BGE-m3), a multilingual generator (Command-R-35B), language-aware prompts, and multilingual Wikipedia as the datastore gives much higher answer recall than no retrieval across 13 languages. Key practical needs: translate system prompts into the user language and instruct the model to reply in that language; prefer multilingual-by-design generators; use character n-gram recall to handle transliteration differences. Main failure modes: code-switching, wrong document reading, irrelevant retrieval, and dataset label issues.
Problem Statement
RAG is well studied in English but not for multilingual users. We need a practical, zero-shot mRAG baseline and clarity on which components and prompt changes are required for good cross-lingual QA performance.
Main Contribution
A public zero-shot multilingual RAG (mRAG) baseline built on BERGEN and released at https://github.com/naver/bergen.
An empirical study of mRAG on 13 languages using MKQA and XOR-TyDi QA, isolating effects of retriever, reranker, generator, prompt language, and retrieval language.
Key Findings
RAG substantially increases answer recall vs no retrieval on evaluated QA sets.
Prompt language strongly controls output language and answer recall.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| MKQA character 3-gram recall (English) | no retrieval 58.4 -> retrieval from English 70.2 | no retrieval | +11.8 | MKQA (English) | Table 1 reports character 3-gram recall across retrieval options. | Table 1 |
| MKQA character 3-gram recall (Arabic) | no retrieval 26.4 -> retrieval from English 45.9 | no retrieval | +19.5 | MKQA (Arabic) | Table 1 shows large gains for Arabic when retrieval is used. | Table 1 |
What To Try In 7 Days
Add a dense multilingual retriever (BGE-m3) to your QA pipeline and compare recall@5 on a small test set.
Translate the system prompt into user languages and add an explicit "reply in <user language>" instruction; measure output language rate.
Use character 3-gram or fuzzy matching to evaluate multilingual answers and catch transliteration variants.
Agent Features
Memory
Frameworks
Architectures
Optimization Features
Token Efficiency
System Optimization
Inference Optimization
Reproducibility
Code URLs
Data URLs
Risks & Boundaries
Limitations
Experiments are zero-shot and evaluated on Wikipedia-based QA; domain shift to other data is untested.
Used a single strong retriever (BGE-m3) trained on Wikipedia-like data, so cross-domain retrieval limits remain unknown.
When Not To Use
If your task uses non-Wikipedia domains without retriever retraining, performance may drop.
When you cannot maintain translated prompts or lack language expertise for prompt checks.
Failure Modes
Code-switching: generator inserts English named entities in non-Latin outputs.
Irrelevant retrieval: retrieved passages may not contain the answer.