Overview
KGR is practical: it runs with prompts and a public KG, shows consistent F1 gains on evaluated QA benchmarks, but relies on good entity linking and triple filtering which need engineering work.
Citations10
Evidence Strength0.80
Confidence0.80
Risk Signals11
Trust Signals
Findings with numeric evidence: 4/4
Findings with evidence refs: 4/4
Results with explicit delta: 4/4
Reproducibility
Status: Partial assets available
Open source: Partial
At A Glance
Cost impact: 50%
Production readiness: 60%
Novelty: 60%
Why It Matters For Business
KGR can reduce factual errors in model outputs, especially for multi-step reasoning tasks, lowering risk in customer-facing answers and automated reporting without retraining large models.
Who Should Care
Summary TLDR
The paper introduces KGR, an automated loop that (1) extracts atomic factual claims from an LLM's draft answer, (2) finds related facts in a knowledge graph (Wikidata), (3) verifies claims, and (4) asks the LLM to retrofit its answer. KGR runs the whole cycle with the LLM (few-shot prompts) and chunked KG triples. On three QA benchmarks (SimpleQuestion, Mintaka, HotpotQA) and three LLMs (ChatGPT, text-davinci-003, Vicuna 13B), KGR improves factual scores—especially on complex, multi-hop problems—by systematically checking facts used during reasoning rather than only query-related facts.
Problem Statement
Large language models often state false facts during multi-step reasoning. Previous KG-augmentation only retrieves facts tied to entities in the user query, so it misses false intermediate facts that appear in the model's reasoning. The paper asks: can we automatically extract the model's internal factual claims, verify them against a knowledge graph, and edit responses to reduce hallucinations?
Main Contribution
KGR: a 5-step, LLM-driven pipeline (claim extraction, entity detection, KG retrieval, fact selection, claim verification, retrofitting) that checks and revises model-generated facts.
An implementation that uses only LLM prompts plus Wikidata (no extra supervised models) and supports iterative multi-turn retrofitting.
Key Findings
KGR raises ChatGPT F1 on Mintaka (complex reasoning) by about 6.2 points over question-relevant KG retrieval (QKR).
KGR yields large gains for text-davinci-003 on open-domain multi-hop HotpotQA: F1 +15.3 points over QKR.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| ChatGPT Mintaka F1 (KGR vs QKR) | 60.8 (KGR) | 54.6 (QKR) | +6.2 | Mintaka (complex reasoning) | Table 1 reports EM/F1 for ChatGPT across methods | Table 1 |
| text-davinci-003 HotpotQA F1 (KGR vs QKR) | 47.2 (KGR) | 31.9 (QKR) | +15.3 | HotpotQA (open-domain multi-hop) | Large F1 jump when retrofitting model answers with Wikidata evidence | Table 1 |
What To Try In 7 Days
Run KGR-style retrofitting on a small sample of your LLM outputs using Wikidata to measure F1 or precision gains.
Add a claim-extraction prompt to your pipeline and log extracted claims to quantify where the model hallucinates.
Test chunk size and retrieved-triple limits to find a cost-accuracy sweet spot for fact selection.
Reproducibility
Data URLs
Risks & Boundaries
Limitations
Relies on KG coverage: facts not in Wikidata remain unverifiable.
Entity detection and fact selection are error-prone and drive most failures.
When Not To Use
Low-latency applications where extra KG checks break SLAs.
Domains lacking a structured KG or with mostly private facts.
Failure Modes
Wrong or overly broad entity detection returns irrelevant triples and prevents correct verification.
Fact selection includes noisy triples, causing incorrect verification signals.