Overview
Production Readiness
0.4
Novelty Score
0.6
Cost Impact Score
0.5
Citation Count
1
Why It Matters For Business
You can improve black-box LLM few-shot extraction by choosing demos that match the model's own output style; this yields production-quality gains without fine-tuning and reduces annotation cost for Open IE.
Summary TLDR
The authors show that few-shot extraction with a black-box LLM (ChatGPT) improves when the demonstration set matches the LLM's own syntactic tendencies. They estimate a syntactic discrepancy by having the LLM paraphrase sentences and measuring Hierarchically Weighted Syntactic (HWS) distance. Then they build demonstrations dominated by low-discrepancy examples (plus a few variants, 8:2 ratio), sampling demos with probability proportional to the inverse discrepancy. On the CaRB Open IE benchmark, 6-shot ChatGPT reaches 55.3 F1 (slightly above supervised OpenIE6 55.2). The same pipeline, with a content-based discrepancy, improves few-shot relation and event extraction. Choosing the right dis-<
Problem Statement
Large LLMs can do few-shot information extraction, but performance varies widely because test sentences can differ in syntax or content from what the model naturally produces. We need a practical way to pick demonstrations that make an LLM reason consistently for a target dataset.
Main Contribution
A simple estimation of distributional discrepancy for a black-box LLM: prompt the model to paraphrase and measure syntactic distance (HWS).
A demo-construction mechanism: sample demonstrations so most are low-discrepancy (8:2 low/high ratio) using inverse-normalized discrepancy as sampling probability.
Empirical validation on Open IE (CaRB) and transfer to Relation Extraction (TACRED) and Event Extraction (ACE05) by swapping discrepancy metrics.
Key Findings
Syntactic mismatch predicts extraction errors for ChatGPT.
6-shot ChatGPT with selected demonstrations reached 55.3 F1 on CaRB.
Demo selection generalizes to ontology-based IE tasks when using a task-appropriate discrepancy.
Results
CaRB F1 (Open IE)
CaRB F1 (few-shot trend)
Effect of candidate set size (3-shot)
TACRED Micro F1 (30-shot)
ACE05 F1 (30-shot)
Who Should Care
What To Try In 7 Days
Measure discrepancy: have your LLM paraphrase ~200 examples and compute a simple syntactic or content distance.
Build demos: pick 8 low-discrepancy examples and 2 diverse variants per query; use inverse-discrepancy sampling.
A/B test on your dataset: compare random demos vs discrepancy-selected demos and track F1/precision/recall.
Reproducibility
Data Urls
- CaRB (public benchmark)
- ROBUST (public OIE candidate set)
- TACRED (public)
- ACE05 (public)
Data Available
Open Source Status
- partial
Risks & Boundaries
Limitations
- Choosing the right discrepancy metric is not automatic and needs task-specific design.
- Method needs a sizable candidate pool and repeated LLM calls, which costs money and time.
- For ontology-heavy tasks (RE/EE), few-shot ChatGPT still trails supervised SOTA substantially.
When Not To Use
- You lack API access or budget for many LLM calls.
- You cannot build a candidate set of demonstrations similar to your target data.
- Your task needs strict label ontologies that LLM few-shot cannot reliably learn.
Failure Modes
- Manually biased or adversarial demonstrations can degrade or bias outputs.
- Wrong choice of discrepancy metric leads to poor demonstration selection.
- Method improves LLMs but may still underperform task-specific supervised models.
Core Entities
Models
- ChatGPT (turbo-3.5)
- OpenIE6
- OpenIE4
- SpanOIE
- RnnOIE
- KLG
- SaliencyED
Metrics
- F1
- Micro F1
- Precision
- Recall
- HWS distance
Datasets
- CaRB
- ROBUST
- TACRED
- ACE05
Benchmarks
- CaRB
- TACRED
- ACE05