Overview
The paper gives clear execution-based evidence that KG context improves LLM QA, but it uses one model (GPT-4), zero-shot prompts, a small synthetic dataset and a 13-table subset, so results are promising but preliminary.
Citations7
Evidence Strength0.60
Confidence0.85
Risk Signals12
Trust Signals
Findings with numeric evidence: 3/5
Findings with evidence refs: 5/5
Results with explicit delta: 5/6
Reproducibility
Status: Code + data available
Open source: Partial
At A Glance
Cost impact: 50%
Production readiness: 30%
Novelty: 45%
Why It Matters For Business
Adding a knowledge graph layer (ontology + mappings) substantially improves LLM answer accuracy on enterprise SQL: expect major gains for normalized schemas and KPI-style questions.
Who Should Care
Summary TLDR
The paper builds a small enterprise Text-to-SQL benchmark (insurance domain, 13-table subset of the OMG P&C model, 43 questions, ontology + mappings) and tests GPT-4 with simple zero-shot prompts. Asking GPT-4 directly against the SQL gave 16.7% average accuracy. Asking GPT-4 to generate SPARQL over a knowledge-graph view of the same data gave 54.2% accuracy (a 37.5 percentage-point improvement). The gain is largest when schema complexity grows; SQL dropped to 0% for high-schema questions. Caveats: single model (GPT-4), zero-shot only, synthetic small dataset, and a subset of the full schema.
Problem Statement
Existing Text-to-SQL benchmarks do not match enterprise realities: large, normalized schemas, business metrics/KPIs, and a separate business context layer (ontology, mappings) are missing. The paper asks how well LLMs answer enterprise questions on SQL and whether adding a knowledge graph (KG) context helps.
Main Contribution
A reproducible enterprise-style benchmark: subset of OMG Property & Casualty schema, 43 natural-language questions, and a context layer (OWL ontology + R2RML mappings).
A controlled experiment comparing GPT-4 zero-shot SQL generation vs. SPARQL over a KG, with execution-based accuracy scoring.
Key Findings
Knowledge-graph context raised GPT-4 execution accuracy from 16.7% to 54.2%.
For low-question/low-schema tasks, SPARQL hit 71.1% vs SQL 25.5%.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Accuracy | 16.7% | — | — | All 43 questions | Table 1; Section 5.1 | Table 1 |
| Accuracy | 54.2% | SQL 16.7% | +37.5 percentage points | All 43 questions | Table 1; Section 5.1 | Table 1 |
What To Try In 7 Days
Run the provided benchmark repo against a small slice of your schema to measure current LLM accuracy.
Create an OWL ontology and simple R2RML mappings for your top 5 business concepts and test SPARQL vs SQL prompts.
Add post-checks for missing columns and identifier-to-label mapping to catch partial answers.
Reproducibility
Data URLs
Risks & Boundaries
Limitations
Single model (GPT-4) and single prompting style (zero-shot) tested
Subset of full OMG P&C schema (13 of 199 tables) and small synthetic data per table
When Not To Use
When you need guaranteed, auditable correctness without human verification
When your schema and business semantics are not modeled or mapped to a KG
Failure Modes
Column name hallucinations (SQL produced non-existent columns)
Value hallucinations used as filters