Overview
The benchmark is well-scoped and publicly released, with clear metrics and multiple models evaluated. Evidence is strong for measured failure modes, but the study is limited to Spider-dev-based examples and specific PLM generation choices.
Citations8
Evidence Strength0.80
Confidence0.86
Risk Signals11
Trust Signals
Findings with numeric evidence: 6/6
Findings with evidence refs: 6/6
Results with explicit delta: 4/4
Reproducibility
Status: Code + data available
Open source: Partial
At A Glance
Cost impact: 40%
Production readiness: 40%
Novelty: 60%
Why It Matters For Business
Text-to-SQL systems that appear accurate in lab tests can silently fail in real use when users phrase questions differently or when schemas store data in alternate formats. That leads to wrong query results and bad UX. Dr.Spider helps find these blind spots before deployment.
Who Should Care
Summary TLDR
The authors build Dr.Spider, a diagnostic benchmark of 17 targeted perturbations (on databases, user questions, and SQL) based on Spider. Dr.Spider contains ~15K pre/post perturbation example pairs. Evaluation of state-of-the-art text-to-SQL systems shows large, repeatable drops: ~14% absolute performance loss overall and a ~50% regression on the hardest perturbation. The paper also analyzes which architecture choices (model size, decoder style, entity linking) help or hurt robustness.
Problem Statement
Text-to-SQL models often succeed on standard test sets but break when inputs change in realistic, task-specific ways. Existing robustness tests are narrow (single phenomena) or handcrafted. Practitioners need a systematic way to measure how changes to database schema, question wording, or small SQL edits affect real system outputs.
Main Contribution
Dr.Spider: a public robustness benchmark built on Spider with 17 perturbation types across DB, natural language question (NLQ), and SQL; ~15K paired examples.
A scalable expert-crowd-AI pipeline: crowdsourced paraphrases + OPT-66B generation + NLI filtering + expert review to create 9 task-specific NLQ perturbation categories.
Key Findings
State-of-the-art text-to-SQL models suffer meaningful accuracy drops on Dr.Spider.
DBcontent-equivalence is the single most damaging perturbation class.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Accuracy | 76.6% -> 65.9% | 76.6% (pre-perturbation on Spider-dev) | -10.7pp | Dr.Spider (macro-average across perturbations) | Table 3 All row for PICARD (EX) | Table 3 |
| Relative robustness (DBcontent-equivalence, PICARD) | 49.3% (relative robustness accuracy) | 88.7% pre | ≈50.7% regression | Dr.Spider DBcontent-equivalence | Table 14 and discussion in Section 5.1 | Table 14 |
What To Try In 7 Days
Run Dr.Spider on your current text-to-SQL model to identify immediate failure classes.
Add simple data augmentation examples for the top 1–2 failing perturbations (value format and DBcontent-equivalence).
Enable and validate an entity-linking or value-normalization step, then re-run targeted tests from Dr.Spider to measure gains/losses.
Reproducibility
Risks & Boundaries
Limitations
Built from Spider development set only; blind spots may exist for other domains or proprietary schemas.
NLQ perturbations depend on OPT-66B generation and filtering choices; different PLMs or prompts may yield different paraphrase distributions.
When Not To Use
As the only robustness test for domain-specific databases not covered by Spider.
To claim full robustness guarantees — Dr.Spider diagnoses common failure modes but is not exhaustive.
Failure Modes
Models overfit to string matching between NLQ tokens and DB content; value-format changes break predictions.
Models can be brittle to alternate schema representations (e.g., compound ↔ split columns, booleans ↔ text).