Overview
Production Readiness
0.7
Novelty Score
0.5
Cost Impact Score
0.6
Citation Count
6
Why It Matters For Business
You can cut hallucinations and quickly update factual behavior by piping search snippets into prompts instead of retraining models—a low-cost way to keep LLM-powered features current.
Summary TLDR
The paper introduces FRESHQA, a 600-question benchmark for time-sensitive and false-premise queries, and a prompting method called FRESHPROMPT that injects Google Search snippets into few-shot prompts. Human evaluation (50K+ judgments) uses two modes: RELAXED (main answer correct) and STRICT (no hallucination). FRESHPROMPT (no training) gives large accuracy gains: GPT-4 + FRESHPROMPT improves absolute accuracy by 47.0 percentage points (STRICT) and 31.4 points (RELAXED) on FRESHQA. The number and order of retrieved evidences strongly affect results. Dataset released on GitHub.
Problem Statement
Pretrained LLMs encode stale facts and hallucinate on time-sensitive or false-premise questions. The paper asks: how badly do models fail on up-to-date queries, and can search-engine evidence injected via prompting (no training) improve factuality?
Main Contribution
FRESHQA: a dynamic 600-question benchmark for time-sensitive and false-premise open-ended QA with human-verified answers and regular updates.
Large-scale human evaluation (50K+ judgments) using two modes: RELAXED (main answer correct) and STRICT (no hallucination).
FRESHPROMPT: a simple few-shot prompting recipe that formats and injects Google Search evidences into the prompt and substantially improves factuality without training.
Key Findings
Pretrained LLMs without web evidence perform poorly on up-to-date QA.
FRESHPROMPT yields large absolute accuracy gains for GPT-4 on FRESHQA.
More and better-ordered evidences increase correctness.
A premise-check instruction helps detect false premises but can hurt overall accuracy for some models.
Chain-of-thought (COT) often increases hallucination risk.
Results
Accuracy
Accuracy
Accuracy
Google Search baseline (STRICT)
Who Should Care
What To Try In 7 Days
Run a small set of time-sensitive queries (FRESHQA dev samples) to measure your model's baseline.
Implement FRESHPROMPT: fetch top ~10 search snippets, format source/date/snippet, add 5 concise demonstrations, and prompt your LLM.
A/B test premise-check instructions and vary number/order of snippets; prefer search/time ordering and 10–15 evidences for best accuracy.
Reproducibility
Data Available
Open Source Status
- partial
Risks & Boundaries
Limitations
- FRESHQA must be updated regularly; answers can become stale between updates.
- FRESHPROMPT was benchmarked with Google Search; other search engines may lack answer-box or metadata and change results.
- Method uses one search query per question; decomposition or multiple queries could improve results.
- Study focuses on English short-form QA; effects on multilingual or long-form QA are not evaluated.
- Relies on in-context learning and the model's context window; very long evidence sets may be truncated.
When Not To Use
- When the system cannot access web search results (air-gapped environments).
- When you can fine-tune a model frequently and prefer parametric updates.
- When queries are extremely long-form and exceed prompt context capacity.
Failure Modes
- Search snippets can be misleading or contradict each other, causing wrong grounding.
- Premise-check instructions can reduce accuracy on valid-premise queries for some models.
- Model may refuse to answer or state cutoff if it ignores retrieved evidences.
- Context-window truncation can drop the most relevant evidence if prompt formatting is poor.
Core Entities
Models
- GPT-4
- GPT-3.5
- CHATGPT
- CODEX
- FLAN-PALM
- PALM
- PALMCHILLA
- FLAN-T5
- T5
Metrics
- Accuracy
- STRICT
- RELAXED
Datasets
- FRESHQA
- FRESHEVAL
Benchmarks
- FRESHQA
Context Entities
Models
- Gpt-3.5-turbo
- gpt-4
- text-davinci-003
- code-davinci-002
Metrics
- inter-rater agreement
Datasets
- TRIVIAQA (used for demonstrations)