Overview
The dataset and factorial method are ready for applied comparisons; results are statistically tested and reproducible, but scope is limited to one jurisdiction and a 100-question sample.
Citations0
Evidence Strength0.85
Confidence0.90
Risk Signals10
Trust Signals
Findings with numeric evidence: 3/3
Findings with evidence refs: 3/3
Results with explicit delta: 5/5
Reproducibility
Status: Code + data available
Open source: Partial
At A Glance
Cost impact: 60%
Production readiness: 70%
Novelty: 60%
Why It Matters For Business
For legal RAG products, improving domain-specific retrieval usually yields far larger gains than swapping LLMs; invest in retrieval quality and dataset-aligned embedders to improve correctness, groundedness, and reduce hallucinations.
Who Should Care
Summary TLDR
Legal RAG Bench is a new public dataset and evaluation method for end-to-end legal retrieval-augmented generation (RAG). It pairs 4,876 passages from the Victorian Criminal Charge Book with 100 expert-crafted, hard legal questions and long answers. A full factorial study of three embedders and two LLMs shows embedding choice dominates performance: a domain-tuned embedder (Kanon 2) raises correctness, groundedness, and retrieval accuracy far more than switching LLMs. Many apparent hallucinations trace back to retrieval failures. The authors release data and code for reproducibility.
Problem Statement
Existing legal benchmarks either ignore end-to-end RAG, have weak labels, or evaluate only one component (retrieval or generation). This misleads builders and hides real failure modes (e.g., retrieval-induced hallucinations). Practitioners need a high-quality, end-to-end legal RAG benchmark with clear error attribution.
Main Contribution
Legal RAG Bench dataset: 4,876 passages + 100 expert-crafted Q‑A‑evidence triplets from the Victorian Criminal Charge Book.
An end-to-end evaluation methodology: full factorial comparisons of embedders and LLMs with a hierarchical error taxonomy (hallucination, retrieval error, reasoning error).
Key Findings
Embedding model choice is the dominant factor in end-to-end legal RAG performance.
Kanon 2 Embedder substantially improves scores versus general-purpose embedders.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Kanon 2 Embedder — correctness | 94% | Text Embedding 3 Large (76.5%) | +17.5 pp | Legal RAG Bench (100 questions) | Table 2 per-embedder averages | Table 2 |
| Kanon 2 Embedder — groundedness | 96% | Text Embedding 3 Large (91.5%) | +4.5 pp | Legal RAG Bench | Table 2 per-embedder averages | Table 2 |
What To Try In 7 Days
Run Legal RAG Bench retrieval-only checks: measure retrieval accuracy on your corpus.
Compare two embedders (domain-tuned vs general) using the benchmark's retrieval metric.
If retrieval is weak, prioritize retriever tuning or a domain embedder before changing the LLM.
Reproducibility
Risks & Boundaries
Limitations
Dataset covers Victorian criminal law only; results may not transfer to other jurisdictions or civil law domains.
Small question set (100) limits coverage of legal variability and rare edge cases.
When Not To Use
As the only evaluation for civil-law or non‑Victorian corpora without retuning the embedder.
For narrow closed‑ended classification tasks where automatic labels are available and simpler metrics suffice.
Failure Modes
Retrieval failures producing ungrounded or hallucinated answers.
LLM-as-judge bias if judge is not independently validated.