Overview
This survey consolidates many prior studies and tools; useful for engineering decisions but not an empirical benchmark itself.
Citations5
Evidence Strength0.70
Confidence0.85
Risk Signals8
Trust Signals
Findings with numeric evidence: 0/5
Findings with evidence refs: 5/5
Results with explicit delta: 0/0
Reproducibility
Status: No open assets linked
Open source: Partial
At A Glance
Cost impact: 60%
Production readiness: 70%
Novelty: 50%
Why It Matters For Business
Contaminated evaluations can create false confidence about model quality and lead to bad product choices; verifying contamination protects model selection and user trust.
Who Should Care
Summary TLDR
This survey collects and organizes research on data contamination—when evaluation examples appear in LLM training data—and explains why it inflates reported performance. It defines contamination across model lifecycle stages and benchmark types, reviews detection methods (white-/gray-/black-box), and surveys mitigation strategies: update benchmarks, rewrite examples, use prevention controls, dynamic evaluation, and LLM-as-judge. The paper catalogs benchmarks and tools and lists open challenges like robust detection and unlearning.
Problem Statement
Evaluation scores for LLMs can be artificially high when test examples overlap with training data. This ‘data contamination’ problem has multiple forms, occurs at different lifecycle stages, and undermines the trustworthiness of model comparisons and research conclusions.
Main Contribution
Unified definition and taxonomy of data contamination by phase (pretraining, finetuning, post-deploy) and by benchmark type (text, text+label, augmentation, benchmark-level).
Survey of contamination-free evaluation strategies: data updates, data rewriting, prevention controls, dynamic benchmarks, and LLM-as-a-judge.
Key Findings
Data contamination is common and, at scale, effectively inevitable.
Larger models tend to show stronger contamination effects (they memorize more).
What To Try In 7 Days
Run n-gram overlap and embedding-similarity checks between your test sets and known public corpora.
If you host models that expose logits, run a MinK% or PaCoST-style check for suspect examples.
Switch a small benchmark to dynamic sampling or paraphrase key items to see if reported gains persist.
Reproducibility
Risks & Boundaries
Limitations
May not capture newly emerging contamination mechanisms or the very latest models.
Focuses on LLM-specific contamination; related topics (MIA, unlearning, memorization) are not exhaustively covered.
When Not To Use
When you need a new empirical contamination metric—this is a literature survey not a new detection algorithm.
When you require finalized standards or policy—recommendations are high-level and research-oriented.
Failure Modes
Black-box heuristics can be fooled by paraphrasing or adversarial augmentation.
Gray-box thresholds (MinK%) are sensitive to K and threshold choices and can produce false positives or negatives.