Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection
LLM EvaluationBenchmark RobustnessStress TestingLLM-as-a-Judge

LLM evaluations miss important variability: greedy often beats sampling, but best-of-N can unlock smaller models

July 15, 20247 min

Overview

Production Readiness

0.6

Novelty Score

0.6

Cost Impact Score

0.7

Citation Count

7

Authors

Yifan Song, Guoyin Wang, Sujian Li, Bill Yuchen Lin

Links

Abstract / PDF

Why It Matters For Business

Single-output LLM benchmarks can hide real-world variability. Testing multiple samples, greedy vs sampling, and best-of-N selection reveals reliability and can let smaller cheaper models match higher-cost models.

Summary TLDR

Most LLM evaluations use one output per prompt and ignore non-determinism (randomness in sampling). This paper measures how greedy decoding (always pick highest-probability token) compares to sampling across seven benchmarks and many models. Key findings: greedy usually performs better than average sampling except on one open-ended benchmark (AlpacaEval); benchmarks with constrained answer spaces (MMLU, MixEval) are stable under sampling; reasoning and code tasks (GSM8K, HumanEval) show large sampling variance; alignment reduces sampling variance; best-of-N selection (using reward models or oracle) can make smaller models match or beat larger/proprietary models. The authors release code and/

Problem Statement

Current LLM comparisons usually run one generation per example. That ignores non-determinism and hides variability in real use. We need to know when greedy vs sampling matters, which benchmarks are stable, how model size and alignment affect variance, and how much performance is recoverable by selecting among multiple samples.

Main Contribution

Systematic large-scale comparison of greedy decoding vs sampling across 7 benchmarks and multiple open and closed models.

Characterization of which benchmarks are stable (MMLU, MixEval) and which show high sampling variance (GSM8K, HumanEval).

Analysis of scaling and alignment effects: no clear reduction in sampling variance with size; alignment can lower variance.

Demonstration that best-of-N selection (with reward models or oracle) recovers large gains and lets smaller models match bigger ones.

Key Findings

Greedy decoding usually outperforms average sampling across most evaluated benchmarks.

NumbersMultiple models: typical sampling std 0.3–2.5 and ∆ up to 27.5 points (Table 2)

Some reasoning and code tasks have very high sampling variance.

NumbersGSM8K/HumanEval best-worst gaps exceed 10 points; e.g., Qwen2-7B GSM8K Greedy 83.5 vs Sample 72.0 (∆11.5)

Benchmarks with constrained outputs are stable under sampling.

NumbersMMLU-Redux standard deviations ~0.3–0.7 across samplings (Table 2)

Alignment techniques tend to reduce sampling variance but do not always improve mean performance.

NumbersPost-alignment: observed decreases in std on AlpacaEval, MMLU, GSM8K, HumanEval (Fig.1)

Best-of-N selection recovers substantial performance; oracle best-of-N can let smaller models beat GPT-4-Turbo on several tasks.

NumbersOracle Best-of-N: smaller Llama-3-8B can outperform GPT-4-Turbo on MMLU, GSM8K, HumanEval (Fig.4)

High sampling temperature and extreme repetition penalties harm reasoning and code performance.

NumbersTemperature 1.5 degrades GSM8K/HumanEval; repetition penalty 1.2 helps AlpacaEval but hurts GSM8K (Sec.4.3–4.4)

Results

GSM8K (Qwen2-7B-Instruct)

ValueGreedy 83.5 vs Sample 72.0

HumanEval (Llama-3-8B-Instruct)

ValueGreedy 58.5 vs Sample 31.8

MMLU-Redux (GPT-4-Turbo)

ValueGreedy 82.6 vs Sample 82.4 (std 0.43)

WildBench WB-Score (Llama-3-8B)

ValueGreedy 29.6 vs Sample 26.2 (std 1.65)

Who Should Care

CtoProduct ManagerMl EngineerEngineering LeadData Scientist

What To Try In 7 Days

Run 16–128 samples on a held-out subset to measure sampling std on your tasks.

Compare greedy decoding vs average sampling; use greedy for deterministic tasks (math/code).

Try best-of-N with an off-the-shelf reward model for critical prompts to boost accuracy cheaply.

Reproducibility

Code Urls

Data Urls

Code Available

Data Available

Open Source Status

  • partial

Risks & Boundaries

Limitations

  • Experiments cover selected benchmarks and open-weight models; not exhaustive of all tasks or closed models.
  • Best-of-N results depend on off-the-shelf reward models; reward model gap to oracle remains large.
  • Sampling budgets (N) differ per benchmark; results scale with N and compute.

When Not To Use

  • Do not generalize sampling harms to tasks not evaluated here.
  • Do not assume alignment always raises mean score; effects vary by method and task.

Failure Modes

  • Sampling can produce many low-quality answers for reasoning/code leading to misleading average scores.
  • Reward-model selection can still lag oracle and may prefer shorter/edited answers favored by judges.

Core Entities

Models

  • Llama-3-8B-Instruct
  • Yi-1.5-6B-Chat
  • Yi-1.5-9B-Chat
  • Yi-1.5-34B-Chat
  • Qwen2-7B-Instruct
  • Qwen2-1.5B-Instruct
  • Qwen2-0.5B-Instruct
  • Mistral-7B-Instruct-v0.2
  • GPT-4-Turbo

Metrics

  • Length-controlled win rate (LC)
  • Win rate (WR)
  • WB-Score
  • Accuracy
  • Exact Match (EM)
  • Pass@1

Datasets

  • AlpacaEval 2
  • Arena-Hard
  • WildBench v2
  • MixEval
  • MMLU-Redux
  • GSM8K
  • HumanEval

Benchmarks

  • AlpacaEval 2
  • Arena-Hard
  • WildBench v2
  • MixEval
  • MMLU-Redux
  • GSM8K
  • HumanEval