Overview
The paper provides multi-benchmark, multi-model empirical evidence showing consistent gaps and large variances on reasoning/code tasks; conclusions are well supported for the evaluated setups.
Citations7
Evidence Strength0.80
Confidence0.85
Risk Signals7
Trust Signals
Findings with numeric evidence: 6/6
Findings with evidence refs: 6/6
Results with explicit delta: 4/4
Reproducibility
Status: Code + data available
Open source: Partial
At A Glance
Cost impact: 70%
Production readiness: 60%
Novelty: 60%
Why It Matters For Business
Single-output LLM benchmarks can hide real-world variability. Testing multiple samples, greedy vs sampling, and best-of-N selection reveals reliability and can let smaller cheaper models match higher-cost models.
Who Should Care
Summary TLDR
Most LLM evaluations use one output per prompt and ignore non-determinism (randomness in sampling). This paper measures how greedy decoding (always pick highest-probability token) compares to sampling across seven benchmarks and many models. Key findings: greedy usually performs better than average sampling except on one open-ended benchmark (AlpacaEval); benchmarks with constrained answer spaces (MMLU, MixEval) are stable under sampling; reasoning and code tasks (GSM8K, HumanEval) show large sampling variance; alignment reduces sampling variance; best-of-N selection (using reward models or oracle) can make smaller models match or beat larger/proprietary models. The authors release code and/
Problem Statement
Current LLM comparisons usually run one generation per example. That ignores non-determinism and hides variability in real use. We need to know when greedy vs sampling matters, which benchmarks are stable, how model size and alignment affect variance, and how much performance is recoverable by selecting among multiple samples.
Main Contribution
Systematic large-scale comparison of greedy decoding vs sampling across 7 benchmarks and multiple open and closed models.
Characterization of which benchmarks are stable (MMLU, MixEval) and which show high sampling variance (GSM8K, HumanEval).
Key Findings
Greedy decoding usually outperforms average sampling across most evaluated benchmarks.
Some reasoning and code tasks have very high sampling variance.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| GSM8K (Qwen2-7B-Instruct) | Greedy 83.5 vs Sample 72.0 | — | 11.5 | GSM8K (N=128) | Table 2 shows Greedy 83.5, Sample 72.0 for Qwen2-7B | Table 2 |
| HumanEval (Llama-3-8B-Instruct) | Greedy 58.5 vs Sample 31.8 | — | 26.7 | HumanEval (N=128) | Table 2 reports 58.5 greedy vs 31.8 sampling for Llama-3-8B | Table 2 |
What To Try In 7 Days
Run 16–128 samples on a held-out subset to measure sampling std on your tasks.
Compare greedy decoding vs average sampling; use greedy for deterministic tasks (math/code).
Try best-of-N with an off-the-shelf reward model for critical prompts to boost accuracy cheaply.
Reproducibility
Risks & Boundaries
Limitations
Experiments cover selected benchmarks and open-weight models; not exhaustive of all tasks or closed models.
Best-of-N results depend on off-the-shelf reward models; reward model gap to oracle remains large.
When Not To Use
Do not generalize sampling harms to tasks not evaluated here.
Do not assume alignment always raises mean score; effects vary by method and task.
Failure Modes
Sampling can produce many low-quality answers for reasoning/code leading to misleading average scores.
Reward-model selection can still lag oracle and may prefer shorter/edited answers favored by judges.