Small prompt or format changes can reorder LLM leaderboards by many ranks

Minor perturbations can shift model ranks by many positions on MMLU.

NumbersUp to 8 rank positions change; example Yi‑6b moved 3→9 (Table 1)

Leaderboards often disagree under small changes (Kendall kτ falls below stability threshold).

Numberskτ dropped to 0.564 under choice shuffles (kτ ≤ 0.75 indicates major disagreement)

Choice ID tokens and choice positions cause selection bias in all tested models.

NumbersReplacing A/B/C/D with rare symbols reduced accuracy and increased RStd (e.g., ∆Acc up to −11.3, ∆RStd +35, Fig.4/TableA

Scoring method strongly affects bias and accuracy.

NumbersSymbol scoring gave highest accuracy but highest RStd; cloze lowered bias but also accuracy; hybrid reduced bias while保持

Models copy answers shown in context and are misled by incorrect context.

NumbersProviding correct one‑shot/5‑shot examples raised accuracy near 90–99% for some models; providing incorrect examples cut

Small prompt text edits and some few‑shot variations have little effect on rankings.

Numberskτ > 0.9 for prompt instruction edits and some few‑shot tweaks

Max rank displacement

ValueUp to 8 positions (MMLU)

Ranking agreement (Kendall kτ)

Valuekτ = 0.564 after random choice shuffles

Baselinekτ = 1.0 (original)

Accuracy

Value∆Acc down to −11.3, ∆RStd up to +35

BaselineSymbols A/B/C/D

Effect of scoring method

ValueSymbol scoring: highest accuracy but highest bias; Cloze: lowest bias but lower accuracy; Hybrid: reduced bias vs symbol

BaselineSymbol scoring

In‑context cheating (one‑shot / five‑shot with correct example)

ValueModel accuracies often rose to ~90–99% with correct examples (e.g., Mistral‑7B 97–99%)

Baselinezero/few‑shot baseline

Prompt instruction edits

Valuekτ > 0.9 (minimal ranking change)

Baselineoriginal prompt