Overview
The method is practical and training-free: formatting and adding search snippets to prompts yields large evaluated gains, but depends on search quality, prompt context limits, and regular dataset updates.
Citations6
Evidence Strength0.85
Confidence0.85
Risk Signals12
Trust Signals
Findings with numeric evidence: 5/5
Findings with evidence refs: 5/5
Results with explicit delta: 3/4
Reproducibility
Status: Partial assets available
Open source: Partial
At A Glance
Cost impact: 60%
Production readiness: 70%
Novelty: 50%
Why It Matters For Business
You can cut hallucinations and quickly update factual behavior by piping search snippets into prompts instead of retraining models—a low-cost way to keep LLM-powered features current.
Who Should Care
Summary TLDR
The paper introduces FRESHQA, a 600-question benchmark for time-sensitive and false-premise queries, and a prompting method called FRESHPROMPT that injects Google Search snippets into few-shot prompts. Human evaluation (50K+ judgments) uses two modes: RELAXED (main answer correct) and STRICT (no hallucination). FRESHPROMPT (no training) gives large accuracy gains: GPT-4 + FRESHPROMPT improves absolute accuracy by 47.0 percentage points (STRICT) and 31.4 points (RELAXED) on FRESHQA. The number and order of retrieved evidences strongly affect results. Dataset released on GitHub.
Problem Statement
Pretrained LLMs encode stale facts and hallucinate on time-sensitive or false-premise questions. The paper asks: how badly do models fail on up-to-date queries, and can search-engine evidence injected via prompting (no training) improve factuality?
Main Contribution
FRESHQA: a dynamic 600-question benchmark for time-sensitive and false-premise open-ended QA with human-verified answers and regular updates.
Large-scale human evaluation (50K+ judgments) using two modes: RELAXED (main answer correct) and STRICT (no hallucination).
Key Findings
Pretrained LLMs without web evidence perform poorly on up-to-date QA.
FRESHPROMPT yields large absolute accuracy gains for GPT-4 on FRESHQA.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Accuracy | 75.6% (GPT-4 + FRESHPROMPT) | 28.6% (GPT-4) | +47.0 pp | FRESHQA test (all) | Table 1 STRICT overall rows | Table 1 |
| Accuracy | 77.8% (GPT-4 + FRESHPROMPT) | 46.4% (GPT-4) | +31.4 pp | FRESHQA test (all) | Table 5 RELAXED overall rows | Table 5 |
What To Try In 7 Days
Run a small set of time-sensitive queries (FRESHQA dev samples) to measure your model's baseline.
Implement FRESHPROMPT: fetch top ~10 search snippets, format source/date/snippet, add 5 concise demonstrations, and prompt your LLM.
A/B test premise-check instructions and vary number/order of snippets; prefer search/time ordering and 10–15 evidences for best accuracy.
Reproducibility
Risks & Boundaries
Limitations
FRESHQA must be updated regularly; answers can become stale between updates.
FRESHPROMPT was benchmarked with Google Search; other search engines may lack answer-box or metadata and change results.
When Not To Use
When the system cannot access web search results (air-gapped environments).
When you can fine-tune a model frequently and prefer parametric updates.
Failure Modes
Search snippets can be misleading or contradict each other, causing wrong grounding.
Premise-check instructions can reduce accuracy on valid-premise queries for some models.