Overview
Methods are validated on three benchmarks including a large real warehouse; results consistently improve recall and downstream SQL, but rely on external LLM APIs and hand-tuned edge weights.
Citations0
Evidence Strength0.80
Confidence0.80
Risk Signals9
Trust Signals
Findings with numeric evidence: 4/4
Findings with evidence refs: 4/4
Results with explicit delta: 3/3
Reproducibility
Status: Code + data available
Open source: Yes
At A Glance
Cost impact: 60%
Production readiness: 60%
Novelty: 65%
Why It Matters For Business
If your product queries very large databases, CRUSH reduces token costs and increases correct SQL generation by selecting a smaller, higher-quality schema subset to send to an LLM.
Who Should Care
Summary TLDR
CRUSH4SQL uses an LLM to 'hallucinate' a tiny, general-purpose schema from a user question, then uses those hallucinated names as probes to retrieve a compact, high-recall subset of a very large database schema. A small combinatorial optimizer then selects a connected subset for downstream Text-to-SQL. On three new large-schema benchmarks (SpiderUnion 4.5k cols, BirdUnion 798 cols, SocialDB ~18k cols), CRUSH raises retrieval recall and improves SQL accuracy versus single-vector dense retrieval and token-level baselines.
Problem Statement
Modern Text-to-SQL models need a small high-quality subset of a client's schema to fit prompts and control cost. Dense nearest-neighbor retrieval over individual columns is brittle on very large schemas (thousands of columns). We need a way to retrieve sets of schema elements that collectively cover the question while staying small and affordable.
Main Contribution
CRUSH method: prompt an LLM to hallucinate a minimal schema for the question, use those hallucinated items as probes for dense retrieval, then run a collective subset-selection objective that maximizes coverage and schema connectivity.
Three large-schema benchmarks: SpiderUnion (4,502 columns), BirdUnion (798 columns), and SocialDB (≈18k columns) for schema-subsetting evaluation.
Key Findings
CRUSH improves column recall at moderate budget on SpiderUnion.
CRUSH improves table/column recall on a real large warehouse.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| column recall (r@10) on SpiderUnion | 0.83 (CRUSH) vs 0.77 (best baseline) | Single DPR (best) | +0.06 | SpiderUnion | Table 2 shows r@10 values | Table 2 |
| table recall (r@10) on SocialDB | 0.58 (CRUSH) vs 0.49 (best baseline) | Single DPR (OpenAI) | +0.09 | SocialDB | Table 2 shows r@10 values | Table 2 |
What To Try In 7 Days
Run CRUSH probe pipeline: few-shot GPT-3 hallucination → SGPT embeddings → candidate retrieval → greedy subset selection.
Compare r@10 and EM/EX using current dense-retrieval baseline on a few production queries.
Tune budget B (10–30 columns) and check SQL accuracy; beware larger budgets can hurt accuracy.
Optimization Features
Token Efficiency
Reproducibility
Code URLs
Data URLs
Risks & Boundaries
Limitations
Hallucination is currently unguided by any compressed client schema and may miss client-specific vocabulary.
Edge/connection weights e(d,d') are hardwired constants rather than learned.
When Not To Use
If the database schema is small enough to fit in-context, simple inclusion may be cheaper and simpler.
When policy forbids sending any question text to external LLM services.
Failure Modes
LLM hallucination can produce probes that miss domain-specific names, causing recall loss.
Hallucinated probes can push retrieval toward common generic columns (Name, id) unless entropy discounting is used.