Overview
Production Readiness
0.6
Novelty Score
0.6
Cost Impact Score
0.3
Citation Count
3
Why It Matters For Business
Alignment choices change who a model helps: biased SFT/PT can reduce utility for non‑US dialects, misrepresent global opinions, and harm product adoption in key markets.
Summary TLDR
This paper measures how standard alignment steps (supervised fine-tuning and preference tuning like RLHF/DPO) change model behavior across three global axes: English dialects, other languages, and opinions about countries. Key findings: alignment raises US English gains more than other dialects (disparity grew from ~1% pre-alignment to up to 17.1% post-alignment), modestly improves multilingual QA when SFT contains even small multilingual mixes (Tülu SFT is 13.1% non-English and boosts many languages), and tends to increase similarity to U.S. opinions vs China/Jordan/Nigeria (relative agreement shifts ~2–5%). Open-source reward models (Starling RM) mirror US preferences (Spearman 0.926/0.849
Problem Statement
Alignment choices (who annotates, which SFT data, which preference data) can unintentionally favor some dialects, languages, and national opinions, creating gaps in how helpful or agreeable an assistant is for different global users.
Main Contribution
Measured alignment effects along three global axes: English dialects, other languages, and country opinions.
Showed supervised fine-tuning and preference tuning often increase US English performance more than Indian/Nigerian English, widening dialect gaps.
Found small amounts of multilingual SFT data can improve many non-English tasks, and reward models can reflect US preferences even when those preferences do not transfer to the tuned language model.
Key Findings
Alignment raises English dialect performance unevenly, favoring US English.
Small multilingual SFT mixes can meaningfully boost multilingual QA.
Alignment increases model agreement with US public opinion relative to some other countries.
Starling reward model strongly reflects US citizen preferences but this doesn't fully transfer to the LM.
Reward models had little effect on out-of-distribution country-opinion preferences of LMs.
Some languages (Bengali) worsened after alignment despite overall multilingual gains.
Results
Accuracy
SFT
TyDiQA / Belebele multilingual gains
Agreement shift vs USA
Starling RM vs Gallup correlation
Who Should Care
What To Try In 7 Days
Audit your SFT and preference datasets for geographic, language, and annotator skew.
Run MD3 (or similar) dialect tests and GlobalOpinionsQA-style checks to spot gaps quickly.
Add a small, targeted multilingual SFT sample (≥5–15%) for key languages and re-evaluate QA tasks.
Reproducibility
Code Available
Data Available
Open Source Status
- partial
Risks & Boundaries
Limitations
- Cannot fully disentangle SFT vs RLHF for Llama 2 Chat because intermediate SFT checkpoint was not released.
- Experiments use released checkpoints rather than performing controlled alignment interventions.
- Evaluation covers a limited set of tasks; other downstream tasks may show different effects.
When Not To Use
- Do not use GlobalOpinionsQA or AskReddit as training targets for alignment; authors warn against optimizing models to match subjective opinion benchmarks.
- Avoid assuming reward-model biases will automatically transfer to the LM for out-of-distribution domains.
Failure Modes
- Alignment can widen performance gaps across dialects even while improving aggregate accuracy.
- Adding small multilingual SFT can help most languages but may still worsen very low-resource languages if examples are absent.
- Reward models can embed specific national biases that do not manifest predictably in the tuned LM.
Core Entities
Models
- Llama 2 7B
- Mistral v0.1 7B
- Starling LM 7B
- Starling RM 7B
- Zephyr-7B-beta
- Tülu 2 7B DPO
- Qwen1.5-7B
- Yi-6B
Metrics
- Accuracy
- CFMScore (answer equivalence)
- 1 - Jensen-Shannon divergence
- Spearman rank correlation
Datasets
- MD3 (Multidialect Dataset of Dialogues)
- TyDiQA GoldP
- Belebele
- GlobalOpinionsQA
- AskReddit country questions (554)
Benchmarks
- MD3 intent prediction
- TyDiQA extractive QA
- Belebele reading comprehension
- GlobalOpinionsQA opinion agreement