Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGPrompt OptimizationAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection

Mask-and-retrieve tests show many benchmarks can leak into LLM training

LLM EvaluationBenchmark LeakageBenchmark ConstructionAnalysis
November 16, 20237 min

Overview

Decision SnapshotNeeds Validation

The paper gives practical, low‑barrier checks (retrieval and masking) with concrete numeric signals, but some results depend on manual review and closed‑model probing.

Citations8

Evidence Strength0.70

Confidence0.80

Risk Signals10

Trust Signals

Findings with numeric evidence: 4/4

Findings with evidence refs: 4/4

Results with explicit delta: 2/4

Reproducibility

Status: Partial assets available

Open source: Partial

At A Glance

Cost impact: 60%

Production readiness: 50%

Novelty: 60%

Authors

Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, Arman Cohan

Links

Abstract / PDF

Why It Matters For Business

If test examples leak into model training, reported model gains may be inflated; flagging and removing leaked examples preserves honest evaluation and prevents bad product decisions.

Who Should Care

ML EngineerData ScientistProduct ManagerCTO

Summary TLDR

The paper presents two practical ways to detect benchmark data leaking into LLM training. First, an IR pipeline (Pyserini + BM25) searches open pretraining corpora (The Pile, C4) for overlaps. Second, TS-Guessing masks a key word or a wrong multiple-choice option and asks models to fill it. Strong closed-source models often reproduce masked benchmark content (e.g., ChatGPT/GPT-4 hit ~52–57% exact matches on masked MMLU items), and fine-tuning with a test set drives that to ~100% — a clear sign of contamination risk.

Problem Statement

Benchmark examples that appear in model training data can inflate reported accuracy. Existing n‑gram overlap checks need full training corpora and still miss many leaks. We need lightweight, model‑agnostic probes and retrieval checks to flag probable contamination for both open and closed models.

Main Contribution

A retrieval pipeline (Pyserini + BM25) to search open pretraining corpora (The Pile, C4) for benchmark overlap.

TS-Guessing: a masking protocol that asks LLMs to fill a masked keyword or a masked wrong multiple‑choice option to reveal memorized test items.

Key Findings

Closed-source LLMs often reproduce masked wrong options in MMLU.

NumbersChatGPT EM 52%, GPT-4 EM 57% on MMLU (Table 3)

Practical UseIf you see >50% EM on masked MMLU items, suspect training‑data leakage; do not treat reported accuracy as fully generalizable.

Evidence RefTable 3; Section 4.2.2

Fine-tuning on the test set makes masked exact-match near perfect.

NumbersContaminated ChatGPT EM ≈100% after fine‑tuning (Figure 4, §4.3)

Practical UsePerfect or near‑perfect reproduction of masked test content strongly indicates the model saw the exact examples in training.

Evidence RefFigure 4; Section 4.3

Results

MetricValueBaselineDeltaSplit / DatasetEvidenceEvidence Ref
Exact Match (EM) — TS-Guessing on MMLUChatGPT 52%, GPT-4 57%——MMLU (filtered test items)Table 3 reports EM for masked wrong option taskTable 3; Section 4.2.2
Exact Match (EM) after deliberate contamination≈100% EMChatGPT pre-finetune EM 52%+~48 ppMMLU (contamination probe)Fine-tuned ChatGPT reproduces masked items nearly perfectlyFigure 4; Section 4.3

What To Try In 7 Days

Run TS-Guessing on your model: mask keywords and wrong options and track EM.

Index public corpora (C4/The Pile) with Pyserini and run question+label queries for overlap.

Add a small human check (50–100 examples) where automated metrics disagree with semantic judgment.

Reproducibility

Code AvailableNo
Data AvailableYes
Open Source StatusPartial
LicenseUnknown

Risks & Boundaries

Limitations

Retrieval uses only BM25 and may miss semantic matches (§7).

Indexing and retrieval are slow and resource heavy (≈2–3 minutes per point).

When Not To Use

When full training-data access is available — prefer corpus-level n‑gram checks.

On very short or indexical questions where masking cannot isolate a keyword.

Failure Modes

False positives: model correctly guesses masked content by chance or world knowledge.

False negatives: contamination exists but model fails to reproduce masked text.

Core Entities

Models

ChatGPT (GPT-3.5-turbo)GPT-4Claude-instant-1-100kClaude-2LLaMa 2-13BMistral-7B

Metrics

Exact Match (EM)Rouge-L F1BM25SacreBLEUBLEURTGPTScore

Datasets

MMLUTruthfulQAHellaSwagWinoGrandeGSM8KOpenbookQAPIQAThe PileC4

Benchmarks

MMLUTruthfulQAHellaSwagWinoGrandeGSM8KOpenbookQAPIQA

Context Entities

Datasets

The PileC4

You May Also Want to Read

A weekly-updated, contamination-free medical benchmark plus automated rubrics that align better with physicians than LLM-as-a-judge

0.35
0.70
0.50
0

Static medical benchmarks can give falsely high performance because models may have seen test cases during training. LiveMedBench detects contamination and tests real-time generalization. If you deploy LLMs for clinical tasks, evaluate on time-separated, evidence-checked benchmarks and prefer retrieval-enabled systems.

Key finding

LiveMedBench snapshot size and scope

Numbers: 2,756 cases; 16,702 rubric criteria; 38 specialties

When synthetic training data and LLM evaluators are related, evaluators unfairly favor the student models

0.60
0.60
0.50
2

Automatic leaderboards and internal evaluations can overstate model quality when the same or related LLMs generate training data and judge models; this risks bad product decisions and misallocated resources.

Key finding

Preference leakage creates measurable bias in LLM judges.

Numbers: PLS averages up to 23.6% (Mistral with GPT-4o & Gemini, Table 1)

Auto-update benchmarks with two LLM-driven strategies to reduce leakage and tune difficulty

0.60
0.50
0.60
0

Automated updates let teams keep benchmarks fresh without heavy human labor, reduce false confidence from leaked test data, and tune question difficulty to better compare model releases.

Key finding

Mimicked datasets produce consistent evaluation scores across multiple regenerations.

Numbers: std dev 0–3% across four mimicked runs (zero-shot scores)

Ko-H5 and an open Korean LLM leaderboard: private tests, new Korean tasks, and when benchmarks stop helping

0.60
0.50
0.50
1

Private test sets and language-tailored benchmarks give more reliable model comparisons and prevent overfitting to public datasets, so product teams can trust model selection for Korean deployments.

Key finding

Private Ko-H5 test sets have very low overlap with popular Korean training corpora.

Numbers: Overlap < 1% across tasks (Table 2)

TreeEval: benchmark-free LLM evaluation via LLM examiner and tree planning

0.60
0.70
0.60
0

TreeEval reduces evaluation cost and leak risk by generating session-unique questions and using adaptive deepening, so teams can compare models faster and with fewer paid API calls while lowering the chance of benchmark contamination.

Key finding

TreeEval rankings strongly match AlpacaEval2.0 rankings.

Numbers: Spearman ρ=0.83; Kendall τ=0.73 (Table 1)