Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection
LLM EvaluationAutomatic EvaluationBenchmark ConstructionSparsity

Measure many LLMs with only a few test items by learning weighted anchors

February 8, 20267 min

Overview

Production Readiness

0.7

Novelty Score

0.6

Cost Impact Score

0.8

Citation Count

0

Authors

Taolin Zhang, Hang Guo, Wang Lu, Tao Dai, Shu-Tao Xia, Jindong Wang

Links

Abstract / PDF

Why It Matters For Business

You can cut LLM evaluation costs by an order of magnitude while keeping accurate performance estimates and rankings, enabling faster model selection and cheaper continuous benchmarking.

Summary TLDR

SparseEval picks a small set of representative test items (anchors), learns weighted aggregation via a small MLP, and iteratively refines anchors using two importance scores. On standard LLM benchmarks the method estimates full-dataset model performance with far fewer items (often ~100) while keeping mean absolute error near or below 1% and Kendall's τ typically above 0.9. Code is released.

Problem Statement

Full evaluation of many LLMs is costly because running every model on large test sets requires heavy inference compute. The paper asks: can we evaluate many models accurately using only a small, learned subset of test items?

Main Contribution

Formalize efficient evaluation as a sparse optimization problem over the model×item score matrix.

SparseEval: a practical pipeline that (1) initializes anchors, (2) learns anchor weights with a small MLP via gradient descent, and (3) refines anchors using Anchor Importance Score (AIS) and Candidate Importance Score (CIS).

Empirical evidence across six benchmarks (ARC, GSM8K, HellaSwag, MMLU, TruthfulQA, Winogrande) showing low MAE and high Kendall's τ with far fewer anchors.

Open-source code and use of a large evaluation corpus (5000+ models) for robustness tests.

Key Findings

SparseEval reduces required test items to about 100 while keeping prediction error low.

NumbersMAE often 0.8%–1.6% with 100 anchors (Table 1).

SparseEval improves ranking agreements versus baselines.

NumbersKendall's τ frequently >0.90 at 40–100 anchors (Table 1).

Anchor refinement (AIS/CIS) consistently improves anchors over k-means or random initialization.

NumbersRefinement reduces MAE by up to ~1.2% compared to k-means/random in small-anchor regimes (Table 2).

SparseEval trains and runs faster than gp-IRT in experiments.

Numbersgp-IRT training took ~16 minutes on MMLU; SparseEval reports much lower training overhead (text/figures).

SparseEval generalizes better to unseen model families in hold-out tests.

NumbersAverage MAE on DeepSeek families: TailoredBench 4.216%, gp-IRT 2.657%, SparseEval 2.09% (Table 3).

Results

MAE (mean absolute error)

Value0.827% (best reported with 100 anchors on an evaluated dataset)

BaselineAnchor Points / gp-IRT / TailoredBench

Kendall's τ (rank correlation)

Value0.90+ (commonly reported at 40–100 anchors)

Baselinebaselines often 0.70–0.88

Generalization (hold-out average MAE)

Value2.09% average MAE on DeepSeek hold-outs

Baselinegp-IRT 2.657% avg, TailoredBench 4.216%

Who Should Care

Ml EngineerProduct ManagerEngineering LeadData ScientistCto

What To Try In 7 Days

Run SparseEval repo on one internal benchmark: pick 100 anchors, train the 4-layer MLP proxy, compare MAE and τ to full evaluation.

Compare k-means vs random anchor initialization on a small validation slice to pick the better start.

Run 5–10 anchor-refinement steps (AIS/CIS) and measure how MAE and τ change versus simple clustering.

Optimization Features

Infra Optimization

  • reduces inference compute proportional to number of anchors

Model Optimization

  • MLP-based weight learning for anchors

Training Optimization

  • gradient descent on proxy MLP (end-to-end for anchor weights)

Inference Optimization

  • evaluate models only on chosen anchors to reduce inference runs

Reproducibility

License

  • various dataset licenses noted in Appendix (MIT, CC BY 4.0, CC BY-SA 4.0, Apache

Code Urls

Data Urls

Code Available

Data Available

Open Source Status

  • partial

Risks & Boundaries

Limitations

  • Requires sufficient training models and prediction data to learn weights via gradient descent; performance drops with very small training sets.
  • Anchor refinement depends on a proxy model; poor proxy choices could reduce gains.
  • May be less effective on datasets with weak item clustering or when items are not redundant.

When Not To Use

  • When you have very few models or training scores (gradient-based weight learning may fail).
  • When precise per-item error analysis is required (SparseEval approximates global scores and rankings).
  • When dataset items are intentionally diverse with no redundancy.

Failure Modes

  • Bad validation split causes poor anchor initialization and biased estimates.
  • Anchor set overfits the training model family and mispredicts novel architectures if hold-out families diverge.
  • Proxy MLP underfits or overfits, producing wrong AIS/CIS and degrading refinement.

Core Entities

Models

  • SparseEval (MLP-based proxy)
  • gp-IRT
  • TailoredBench
  • Anchor Points
  • TinyBenchmark

Metrics

  • MAE (mean absolute error)
  • Kendall's τ (rank correlation)

Datasets

  • ARC
  • GSM8K
  • HellaSwag
  • MMLU
  • TruthfulQA
  • Winogrande
  • Open-LLM Leaderboard (scores)

Benchmarks

  • LLM evaluation benchmarks (above datasets)