Overview
The pipeline is simple to run with ChatGPT and shows consistent multi-model gains on SuperGLUE; cost depends on LM API use and human review budget for validation.
Citations3
Evidence Strength0.80
Confidence0.80
Risk Signals9
Trust Signals
Findings with numeric evidence: 5/5
Findings with evidence refs: 5/5
Results with explicit delta: 5/5
Reproducibility
Status: Partial assets available
Open source: Partial
At A Glance
Cost impact: 45%
Production readiness: 60%
Novelty: 55%
Why It Matters For Business
TarGEN can create labeled training data from task descriptions without human seeds, reducing annotation cost and enabling model training for niche or proprietary tasks where examples don't exist.
Who Should Care
Summary TLDR
TarGEN is a four-step, seedless prompting pipeline that uses large LMs (ChatGPT and others) to synthesize labeled datasets from task descriptions. It adds a single-step self-correction pass to relabel noisy outputs. On eight SuperGLUE tasks, models fine-tuned on TarGEN's synthetic data match or outperform models trained on original data (typical gains 1–5% accuracy). Synthetic data shows higher lexical and semantic diversity and a broader range of difficulty while exhibiting similar named-entity bias to originals.
Problem Statement
High-quality benchmarks are costly to create and many tasks have no labeled seed examples. Existing LM-driven data generation often depends on example seeds and produces low diversity. The paper asks: can a seedless, multi-step prompting pipeline plus an LM-based self-correction step synthesize labeled datasets that train competitive models?
Main Contribution
TarGEN: a 4-step, seedless prompting pipeline (contexts → instance seeds → label-constrained generation → self-correction) for targeted dataset generation.
Self-correction: one meta-prompt evaluation pass with an LM to relabel noisy instances and reduce mislabels.
Key Findings
Models trained on TarGEN synthetic SuperGLUE match or improve over original-data-trained models.
Instruction tuning on synthetic data further improves performance.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Accuracy | Og avg → Syn avg: +~1.1–2.1 pp across model families | Original (Og) | ≈ +1–3 pp | Average across eight SuperGLUE tasks (Table 6) | Table 6 shows Og vs Syn and Og-I vs Syn-I averages | Table 6 |
| Instruction tuning improvement | Flan T5: +3.42 pp; Pythia GPT: +3.24 pp | non-instruction tuned variants | +3.24 to +3.42 pp | Single-task settings (Table 3, §4.1) | §4.1, Table 3 | Table 3 |
What To Try In 7 Days
Run TarGEN on one low-data classification or NLI task you care about using ChatGPT and the provided prompts.
Add the one-step LM self-correction pass to relabel noisy outputs before fine-tuning.
Fine-tune a small target model (e.g., T5-large or Llama2-7B) on the synthetic set and compare against any available real data baseline.
Agent Features
Planning
Tool Use
Frameworks
Optimization Features
Training Optimization
Reproducibility
Risks & Boundaries
Limitations
TarGEN depends on a closed-source LM (ChatGPT) in experiments; generalization to other LMs was tested but not extensively quantified.
Some datasets were truncated due to generation cost (ReCoRD, BoolQ), so scale effects are not fully measured.
When Not To Use
When legal or safety-critical labels require expert human annotation.
When you cannot afford LM API costs for large-scale generation and self-correction.
Failure Modes
LLM hallucination producing unrealistic instances that pass automatic checks but mislead models.
Self-correction relying on the same LM could reinforce shared model biases instead of correcting them.