Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGPrompt OptimizationAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection

Have a strong LLM critique and rewrite your instruction data, then retrain — improves instruction-following.

Training Data and MethodsInstruction TuningLLM-as-a-JudgeData-efficient TrainingMethod
October 18, 20236 min

Overview

Decision SnapshotNeeds Validation

The paper shows consistent automatic-evaluation gains and data-quality improvements. Results rely on LLM judges and automated benchmarks, so try a small pilot before full adoption.

Citations1

Evidence Strength0.70

Confidence0.86

Risk Signals9

Trust Signals

Findings with numeric evidence: 3/3

Findings with evidence refs: 3/3

Results with explicit delta: 3/5

Reproducibility

Status: Code + data available

Open source: Yes

At A Glance

Cost impact: 50%

Production readiness: 60%

Novelty: 60%

Authors

Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Heng Huang, Jiuxiang Gu, Tianyi Zhou

Links

Abstract / PDF / Code / Data

Why It Matters For Business

You can raise instruction-following quality without larger models by spending on oracle-LM calls to rewrite training data, which often costs less than collecting new human-labeled data and improves model utility quickly.

Who Should Care

Product ManagerML EngineerCTOData ScientistEngineering LeadFounder

Summary TLDR

Reflection-tuning uses an oracle LLM (e.g., ChatGPT/GPT-4) to critique and rewrite each instruction-response pair along defined criteria, producing a "recycled" dataset. Models fine-tuned on recycled data (Recycled Alpaca/WizardLM) show big gains on automatic instruction-following benchmarks (AlpacaEval, Open LLM Leaderboard) and improved data quality metrics (lower perplexity, higher coherence). The method is a data-centric, post-hoc pipeline that requires API access to a strong LLM and modest retraining of base models.

Problem Statement

Instruction-tuning quality depends heavily on the quality of instruction-response pairs. Low-quality or inconsistent examples harm model behavior. The paper asks: can an oracle LLM automatically inspect and improve existing instruction data to make instruction-tuning more effective?

Main Contribution

Reflection-Tuning pipeline that uses an oracle LLM to critique and rewrite both instructions and responses under explicit criteria.

Recycled instruction-response datasets for Alpaca and WizardLM and open release of code, data, and models.

Key Findings

Recycled Alpaca 7B beats many open-source 7B models on AlpacaEval.

NumbersRecycled Alpaca 7B win rate 76.99% vs Alpaca 7B 26.46%

Practical UseIf you have a base instruction dataset, running reflection-tuning can substantially raise automatic judged win rates without changing model size.

Evidence RefTable 1 (AlpacaEval leaderboard)

Recycled WizardLM 7B is top among 7B open models on AlpacaEval.

NumbersRecycled WizardLM 7B win rate 78.88% on AlpacaEval

Practical UseEven already-refined datasets benefit: apply reflection to boost quality further before fine-tuning.

Evidence RefTable 1 (AlpacaEval leaderboard)

Results

MetricValueBaselineDeltaSplit / DatasetEvidenceEvidence Ref
AlpacaEval win rateRecycled WizardLM 7B: 78.88%WizardLM (original)—AlpacaEval leaderboard (GPT-4 judge)Table 1, AlpacaEvalTable 1
AlpacaEval win rateRecycled Alpaca 7B: 76.99%Alpaca 7B: 26.46%+50.53ppAlpacaEval leaderboard (GPT-4 judge)Table 1, AlpacaEvalTable 1

What To Try In 7 Days

Pick a 1k–5k subset of your instruction data and run the reflection pipeline with ChatGPT/GPT-4.

Measure changes in perplexity and coherence before and after rewriting using Sentence-BERT and your base model.

Retrain a small 7B model for a few epochs and compare via AlpacaEval or pairwise GPT-4 judgments.

Optimization Features

Training Optimization
Data-efficient Training

Reproducibility

Code AvailableYes
Data AvailableYes
Open Source StatusYes
LicenseUnknown

Code URLs

https://github.com/MingLiiii/Reflection_Tuning

Data URLs

https://github.com/MingLiiii/Reflection_Tuning

Risks & Boundaries

Limitations

Relies on a strong oracle LLM; improvements depend on that model's preferences and biases.

Evaluation uses GPT-4/ChatGPT as judge, which may reflect judge-specific tastes rather than humans.

When Not To Use

If you lack access or budget for reliable oracle-LM API calls.

When you require diverse human voice or highly domain-specific labeling that an oracle may not replicate.

Failure Modes

Overfitting to judge style: model may align to oracle/LMM preferences rather than real users.

Loss of diversity: rewriting can make data more uniform and reduce edge-case behaviors.

Core Entities

Models

Llama2-7bRecycled Alpaca 7BRecycled WizardLM 7BRecycled Alpaca 13B

Metrics

Win rateAverage scorePerplexityCoherence (Sentence-BERT)IFD score (Instruction-Following Difficulty)

Datasets

Alpaca (52k)WizardLM (subset 70k)AlpacaEvalHuggingface Open LLM Leaderboard

Benchmarks

AlpacaEvalHuggingface Open LLM LeaderboardARCHellaSwagMMLUTruthfulQAVicuna test set

Context Entities

Models

GPT-4ChatGPTXwin-LMVicuna 7B

Metrics

Win rateStandard errorAverage response length

Datasets

AlpacaFarm (evaluation)Davinci003 responses (benchmarks)

Benchmarks

AlpacaEval leaderboardHuggingface Open LLM Leaderboard

You May Also Want to Read

Automatically pick high-quality instruction examples to finetune LLMs and cut training cost

0.60
0.60
0.70
2

Score and pick a few thousand high-quality instruction examples to cut finetuning time and GPU cost while keeping or improving human-facing behavior.

Key finding

A small selected subset (≈2,532 examples) produced a competitive finetuned model.

Numbers: Selected subset = 2,532 examples (≈2.5% of 100k)

Survey of financial LLMs: techniques, benchmarks, and practical gaps

0.50
0.40
0.60
14

FinLLMs help automate common finance language tasks but are uneven: use task-finetuned PLMs for classification/NER to cut cost; reserve large LLMs for complex QA or exploratory uses with human checks.

Key finding

For sentiment analysis, mixed-domain PLMs achieved top scores, while instruction-finetuned LLMs matched but cost more.

Numbers: FLANG-ELECTRA F1=92%; FinMA-30B/GPT-4 F1≈87% (5-shot)

A practical recipe that turns a 3B open base model into competitive instruction- and preference-aligned chat models using QLoRA, synthetic-m

0.30
0.50
0.70
0

You can obtain a commercially usable, preference-aligned 3B chat model with modest compute and open licensing by combining synthetic instruction data, QLoRA adapters, and DPO — useful where cost, transparency, and permissive reuse matter.

Key finding

DPO checkpoint improved LM‑Eval average over base model

Numbers: avg 0.5926 vs base 0.5704 (Table 6)

Let LLMs label and correct themselves: filter unknowns, prefer better answers, and reduce hallucinations

0.50
0.60
0.70
1

You can reduce dependence on human labels and third-party data by having models generate and clean their own training pairs; selective filtering keeps corrections focused and preserves existing capabilities, lowering labeling cost and privacy exposure.

Key finding

Self-training with filtering increases truthfulness on Wiki-Test (GPT-4 judged pairwise).

Numbers: Wiki-Test wins: 1.1B 54% win vs 16% lose; 7B 40.4% win vs 9.9% lose; 13B 36.9% win vs 12% lose

Pick 5–15% of instruction data using gradient signal-to-noise from a LoRA ensemble to match or beat full-data fine-tuning

0.60
0.50
0.60
0

Cut fine-tuning cost by selecting a small, high-value subset (5–15%) that preserves or improves model quality and reduces training time.

Key finding

GRADFILTERING-selected 5–15% subsets match or outperform Random and Superfiltering in most judged cases.

Numbers: 19/24 LLM-as-a-judge cases