Overview
Production Readiness
0.6
Novelty Score
0.5
Cost Impact Score
0.6
Citation Count
0
Why It Matters For Business
Feeding a small set of ranked table cells to an LLM yields more accurate and preferred conversational answers and cuts errors from wrong-source retrievals; that improves user trust while keeping LLM API calls limited.
Summary TLDR
cTBLS is a three-step system that (1) uses a dual-encoder dense retriever to find the right table, (2) ranks table cells with a RoBERTa-based state tracker, and (3) prompts GPT-3.5 with the top-k cells to generate grounded replies. On the HYBRIDIALOGUE dataset, dense table retrieval raises table MRR@10 from 0.491 to 0.846 and Top-1 accuracy from 0.345 to 0.777. Coarse cell ranking places the correct cell Top-3 in ~78% of follow-ups. Using Top-3 cells with GPT-3.5 improves ROUGE and human judgments: coherence preferred 84%, fluency 82.7%, and informativeness 50% vs 12.4% for prior SoTA.
Problem Statement
Open conversational LLMs hallucinate when asked about facts in tables. Prior systems either use keyword matching for table retrieval or heavy decoder fine-tuning. The problem: reliably find the right table and the right cells across many tables, then feed concise, relevant table facts to an LLM to produce accurate, conversational replies while keeping API calls limited.
Main Contribution
A practical three-step pipeline (Dense Table Retrieval, coarse+fine cell ranking, LLM prompting) to ground conversational responses on tables.
Dense Table Retrieval (dual-encoder) that substantially outperforms BM25 for selecting the correct table on HYBRIDIALOGUE.
A shared encoder-based coarse+fine state tracking approach that ranks cells and reduces hallucination when used to prompt GPT-3.5 with top-k cells.
Key Findings
Dense Table Retrieval (DTR) improves table retrieval vs BM25.
Coarse state tracker finds the correct cell often in top-k.
Prompting GPT-3.5 with top-3 ranked cells improves automatic and human scores.
Results
MRR@10 (table retrieval)
Accuracy
Cell ranking Top-k
ROUGE-1 Precision (response generation)
Human preference - Coherence
Human preference - Fluency
Human Informativeness (semantic equivalence to ground truth)
Who Should Care
What To Try In 7 Days
Index tabular sources with a dual-encoder dense retriever (RoBERTa embeddings).
Implement a coarse cell ranker and include top-3 cells in LLM prompts.
Measure table MRR, Top-1 accuracy, and simple human judgments on a small eval set.
Optimization Features
Token Efficiency
- removed stopwords from knowledge in prompts to fit token limits
Training Optimization
- contrastive fine-tuning for table retriever
- triplet-margin fine-tuning for cell ranker
Inference Optimization
- pre-compute table embeddings for fast nearest-neighbor retrieval
- limit LLM prompts to top-3 cells to respect token limits
Reproducibility
Code Urls
Data Urls
- HYBRIDIALOGUE (Nakamura et al., 2022)
Code Available
Data Available
Open Source Status
- partial
Risks & Boundaries
Limitations
- OpenAI API token limits force using at most top-3 cells in prompts.
- cTBLS struggles with whole-table queries and aggregation (counting, comparisons).
- Final generation depends on a closed LLM (GPT-3.5) that cannot be fine-tuned cheaply.
When Not To Use
- When you require table-wide aggregation or precise numerical computation.
- When you cannot include external LLM API calls or need fully open-source stacks without adaptation.
- When token budgets or latency prohibit adding multiple knowledge snippets to prompts.
Failure Modes
- Incorrect cell ranking -> LLM hallucinates or answers wrong.
- Long or complex numeric reasoning leads to counting errors or wrong comparisons.
- Token truncation of knowledge causes missing context and degraded responses.
Core Entities
Models
- RoBERTa-base
- GPT-3.5 (text-davinci-003)
- DialoGPT-medium
- SentenceBERT
- TaPas
Metrics
- MRR@10
- Accuracy
- ROUGE-1
- ROUGE-2
- ROUGE-L
- Coherence
- Fluency
- Informativeness
Datasets
- HYBRIDIALOGUE
- OTT-QA (source of multi-hop questions referenced)