Overview
The experiments are extensive across models, two context scales, and multiple tasks; results are reproducible using the released dataset and code, but applicability depends on your model fleet and domain.
Citations1
Evidence Strength0.80
Confidence0.80
Risk Signals11
Trust Signals
Findings with numeric evidence: 6/7
Findings with evidence refs: 7/7
Results with explicit delta: 6/7
Reproducibility
Status: Code + data available
Open source: Partial
At A Glance
Cost impact: 60%
Production readiness: 70%
Novelty: 60%
Why It Matters For Business
Choose RAG or LC based on model size, document length, and task. This reduces cost and error: RAG protects smaller models from long-context failure and reduces hallucinations; LC is better for synthesis and reasoning on strong long-context models.
Who Should Care
Summary TLDR
LaRA is a 2,326-case benchmark that compares Retrieval-Augmented Generation (RAG) and long-context (LC) LLMs across novels, academic papers, and financial statements for four QA tasks (location, reasoning, comparison, hallucination detection). There is no universal winner: small/weak models often benefit more from RAG, large models with strong long-context ability often do better with LC, RAG is best at hallucination detection, and LC wins at comparison and reasoning for strong models. Use task, model size, context length, and chunking to route queries in practice.
Problem Statement
As LLMs get much longer context windows, practitioners must choose between (A) feeding the full long text to an LLM (LC) or (B) using retrieval to inject selected chunks (RAG). Existing comparisons are inconclusive and suffer from benchmark flaws (short contexts, leakage, truncation). LaRA aims to fix those flaws and give practical guidance on when to use RAG vs LC.
Main Contribution
LaRA benchmark: 2,326 test cases, 3 context types, 4 QA tasks reflecting real queries
Systematic comparison of 11 LLMs (7 open-source, 4 proprietary) across 32k and 128k context windows
Key Findings
No universal winner — best choice depends on model size, context length, task, and chunks.
LaRA dataset: 2,326 test cases spanning novels, papers, financial statements and four QA types.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Dataset size | 2326 test cases across 3 context types & 4 tasks | — | — | LaRA | Abstract; Table 3 | Abstract; Table 3 |
| Average LC vs RAG gap (32k) | LC +2.40% accuracy | RAG | +2.40% | All models, 32k contexts | Table 2 (Avg GAP) | Table 2 |
What To Try In 7 Days
Run a small split test: route comparison/reasoning queries to your largest LC-capable model and hallucination-sensitive queries to RAG.
If using small models, prototype RAG with 600-token chunks, overlap 100, and tune number of chunks; measure accuracy and cost.
Validate an LLM-based judge (e.g., GPT-4o) on 100 samples vs humans to cheaply scale evaluation and compute Cohen's Kappa.
Optimization Features
Token Efficiency
System Optimization
Inference Optimization
Reproducibility
Code URLs
Data URLs
Risks & Boundaries
Limitations
Contexts limited to novels, academic papers, and US financial statements; other domains may behave differently.
RAG config fixed (600-token chunks, overlap 100, 5 chunks/doc) — other configs might change results.
When Not To Use
If your application uses very different document types (e.g., codebases, logs) without validation on similar data.
If you cannot control retrieval quality (no reliable embeddings or index), RAG may fail to retrieve needed chunks.
Failure Modes
LC hallucination when full context adds noisy, irrelevant information.
RAG misses required chunks for comparison tasks if retrieval does not surface all relevant pieces.