Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection
Retrieval Augmented Generation (RAG)Dense RetrievalBasic RAGContext Selection

Practical recipe and baseline for multilingual RAG across 13 languages

July 1, 20247 min

Overview

Production Readiness

0.7

Novelty Score

0.4

Cost Impact Score

0.6

Citation Count

2

Authors

Nadezhda Chirkova, David Rau, Hervé Déjean, Thibault Formal, Stéphane Clinchant, Vassilina Nikoulina

Links

Abstract / PDF

Why It Matters For Business

Multilingual RAG lets products answer factual questions in many languages by combining strong multilingual retrieval and tuned prompts, expanding reach and reducing wrong answers for non-English users.

Summary TLDR

This paper builds and open-sources a zero-shot multilingual retrieval-augmented generation (mRAG) pipeline and measures which components matter. Using strong dense retrieval (BGE-m3), a multilingual generator (Command-R-35B), language-aware prompts, and multilingual Wikipedia as the datastore gives much higher answer recall than no retrieval across 13 languages. Key practical needs: translate system prompts into the user language and instruct the model to reply in that language; prefer multilingual-by-design generators; use character n-gram recall to handle transliteration differences. Main failure modes: code-switching, wrong document reading, irrelevant retrieval, and dataset label issues.

Problem Statement

RAG is well studied in English but not for multilingual users. We need a practical, zero-shot mRAG baseline and clarity on which components and prompt changes are required for good cross-lingual QA performance.

Main Contribution

A public zero-shot multilingual RAG (mRAG) baseline built on BERGEN and released at https://github.com/naver/bergen.

An empirical study of mRAG on 13 languages using MKQA and XOR-TyDi QA, isolating effects of retriever, reranker, generator, prompt language, and retrieval language.

Key Findings

RAG substantially increases answer recall vs no retrieval on evaluated QA sets.

NumbersMKQA English recall 58.4 -> 70.2; Arabic 26.4 -> 45.9 (Table 1).

Prompt language strongly controls output language and answer recall.

NumbersTranslating prompt + instructing to reply in user language raises Correct Language Rate (CLR) to >95% in most cases (see

Multilingual-by-design generators outperform English-centric ones for non-English outputs.

NumbersCommand-R-35B reaches near-100% CLR and higher recall across languages versus LLaMA-2-7B and SOLAR on non-English (Table

Dense multilingual retriever BGE-m3 is robust cross-lingually and beats translated-query sparse baselines.

NumbersRetrieval recall@5: BGE-m3 61.5 (ko) vs SPLADE+QT 60.9; English recall@5 88.5 (Table 7).

Evaluation needs adjustment for transliterations and code-switching.

NumbersPaper proposes character 3-gram recall to capture transliteration matches (illustrated in Table 3).

Results

MKQA character 3-gram recall (English)

Valueno retrieval 58.4 -> retrieval from English 70.2

Baselineno retrieval

MKQA character 3-gram recall (Arabic)

Valueno retrieval 26.4 -> retrieval from English 45.9

Baselineno retrieval

Correct Language Rate (CLR) with translated prompts

ValueCLR >95% in most languages with translated system prompt

Baselinedefault prompt in English

Retrieval recall@5 (cross-lingual)

ValueBGE-m3 recall@5: ko 61.5, fr 78.4, en 88.5

BaselineSPLADE+QT or SPLADE

Who Should Care

Product ManagerMl EngineerEngineering LeadCto

What To Try In 7 Days

Add a dense multilingual retriever (BGE-m3) to your QA pipeline and compare recall@5 on a small test set.

Translate the system prompt into user languages and add an explicit "reply in <user language>" instruction; measure output language rate.

Use character 3-gram or fuzzy matching to evaluate multilingual answers and catch transliteration variants.

Agent Features

Memory

  • retrieval memory (external datastore)

Frameworks

  • BERGEN

Architectures

  • decoder-only Transformer

Optimization Features

Token Efficiency

  • limit generation to 128 tokens

System Optimization

  • retrieve 50 passages, rerank top-5 into LLM context

Inference Optimization

  • int4 quantization for models (experiments run quantized)
  • greedy decoding

Reproducibility

Code Urls

Data Urls

Code Available

Data Available

Open Source Status

  • partial

Risks & Boundaries

Limitations

  • Experiments are zero-shot and evaluated on Wikipedia-based QA; domain shift to other data is untested.
  • Used a single strong retriever (BGE-m3) trained on Wikipedia-like data, so cross-domain retrieval limits remain unknown.
  • Multilingual prompts require translation quality control and per-language management in practice.

When Not To Use

  • If your task uses non-Wikipedia domains without retriever retraining, performance may drop.
  • When you cannot maintain translated prompts or lack language expertise for prompt checks.

Failure Modes

  • Code-switching: generator inserts English named entities in non-Latin outputs.
  • Irrelevant retrieval: retrieved passages may not contain the answer.
  • Wrong reading: model misinterprets retrieved documents despite correct retrieval.
  • Label and dataset issues: ambiguous or outdated questions affect measured performance.

Core Entities

Models

  • BGE-m3
  • BGE-reranker-v2-m3
  • Command-R-35B
  • Mixtral-8x7B
  • SOLAR-10.7B
  • LLaMA-2-7B-chat
  • SPLADE
  • NLLB-600M

Metrics

  • character 3-gram recall
  • Correct Language Rate (CLR)
  • Retrieval recall@5

Datasets

  • MKQA
  • XOR-TyDi QA
  • TyDi QA
  • KILT Wikipedia
  • Natural Questions (NQ)

Benchmarks

  • KILT