Overview
This is a literature synthesis (survey). It is practically useful for auditing and planning, but it offers limited new empirical numbers; evidence strength is moderate because claims rest on cited empirical studies.
Citations13
Evidence Strength0.70
Confidence0.85
Risk Signals8
Trust Signals
Findings with numeric evidence: 1/5
Findings with evidence refs: 5/5
Results with explicit delta: 0/2
Reproducibility
Status: No open assets linked
Open source: Partial
At A Glance
Cost impact: 60%
Production readiness: 50%
Novelty: 30%
Why It Matters For Business
Biased LLM outputs can cause legal risk, reputational harm, and unfair customer outcomes; fixing bias early reduces downstream remediation cost and regulatory exposure.
Who Should Care
Summary TLDR
This is a wide-ranging literature review that categorizes bias in large language models as intrinsic (in training data or model internals) and extrinsic (biases that show up in downstream tasks). It summarizes evaluation tools (data-, model-, output- and human-level methods), catalogs pre-/intra-/post-model mitigation techniques, and highlights ethical/legal harms in real applications. The paper does not present new experiments; it synthesizes prior work and points to open measurement and mitigation gaps.
Problem Statement
LLMs learn and sometimes amplify societal biases present in massive text corpora. These biases appear inside models and in downstream tasks, harming marginalized groups in settings like hiring, healthcare, and moderation. Practitioners lack a compact map of where bias originates, how to measure it at each stage, and which mitigation options trade off fairness, accuracy, and cost.
Main Contribution
Systematic taxonomy: divides bias into intrinsic (training/data/model) and extrinsic (downstream task) forms.
Organized evaluation methods: groups tools into data-level, model-level, output-level, human-in-the-loop, and domain-specific approaches.
Key Findings
Toxicity can emerge quickly from benign prompts in generative LLMs.
Bias shows up at multiple stages: in datasets, model internals, and final outputs.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| toxicity degeneration (generative) | models reach toxicity >0.5 in under 100 generations on RealToxicityPrompts | — | — | RealToxicityPrompts | Gehman et al., 2020 measured rapid toxic degeneration in several LMs | Gehman et al., 2020 |
| higher ASR error by speaker race | higher word error rates for Black speakers vs White speakers (reported) | — | — | ASR evaluation (Koenecke et al., 2020) | Koenecke et al. report racial disparities in ASR WER | Koenecke et al., 2020 |
What To Try In 7 Days
Run a quick dataset audit: measure demographic coverage and basic skew metrics.
Add output filters: integrate an off-the-shelf toxicity/moderation endpoint into the inference pipeline.
Run counterfactual prompts on key user flows to surface obvious disparities (swap gender/location names).
Reproducibility
Risks & Boundaries
Limitations
No new experiments: conclusions synthesize prior studies rather than reporting fresh quantitative benchmarks.
Many cited results lack consistent metrics, making head-to-head method comparisons difficult.
When Not To Use
Not a source for choosing a single 'best' debiasing algorithm—use task-specific benchmarks instead.
Not a replacement for domain-specific human evaluation in high-stakes systems like healthcare or hiring.
Failure Modes
Over-correcting on one metric (e.g., equalizing TPR) that degrades overall utility or creates new harms.
Evaluation blind spots: benchmarks and judges carry their own biases, so passing tests can give false safety confidence.