Overview
Method is practical and open-sourced; experiments across six public benchmarks and 5000+ models support claims, but performance depends on having enough training models and representative validation splits.
Citations0
Evidence Strength0.70
Confidence0.86
Risk Signals9
Trust Signals
Findings with numeric evidence: 5/5
Findings with evidence refs: 5/5
Results with explicit delta: 3/3
Reproducibility
Status: Code + data available
Open source: Partial
License: various dataset licenses noted in Appendix (MIT, CC BY 4.0, CC BY-SA 4.0, Apache
At A Glance
Cost impact: 80%
Production readiness: 70%
Novelty: 60%
Why It Matters For Business
You can cut LLM evaluation costs by an order of magnitude while keeping accurate performance estimates and rankings, enabling faster model selection and cheaper continuous benchmarking.
Who Should Care
Summary TLDR
SparseEval picks a small set of representative test items (anchors), learns weighted aggregation via a small MLP, and iteratively refines anchors using two importance scores. On standard LLM benchmarks the method estimates full-dataset model performance with far fewer items (often ~100) while keeping mean absolute error near or below 1% and Kendall's τ typically above 0.9. Code is released.
Problem Statement
Full evaluation of many LLMs is costly because running every model on large test sets requires heavy inference compute. The paper asks: can we evaluate many models accurately using only a small, learned subset of test items?
Main Contribution
Formalize efficient evaluation as a sparse optimization problem over the model×item score matrix.
SparseEval: a practical pipeline that (1) initializes anchors, (2) learns anchor weights with a small MLP via gradient descent, and (3) refines anchors using Anchor Importance Score (AIS) and Candidate Importance Score (CIS).
Key Findings
SparseEval reduces required test items to about 100 while keeping prediction error low.
SparseEval improves ranking agreements versus baselines.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| MAE (mean absolute error) | 0.827% (best reported with 100 anchors on an evaluated dataset) | Anchor Points / gp-IRT / TailoredBench | up to ~2% lower than baselines in Table 1 | various LLM benchmarks (see Table 1) | SparseEval MAE values in Table 1 show 0.827% and many ≈1% with 100 anchors | Table 1 |
| Kendall's τ (rank correlation) | 0.90+ (commonly reported at 40–100 anchors) | baselines often 0.70–0.88 | improvements up to ~0.1 in τ | various LLM benchmarks (see Table 1) | Table 1 reports τ frequently above 0.90 for SparseEval | Table 1 |
What To Try In 7 Days
Run SparseEval repo on one internal benchmark: pick 100 anchors, train the 4-layer MLP proxy, compare MAE and τ to full evaluation.
Compare k-means vs random anchor initialization on a small validation slice to pick the better start.
Run 5–10 anchor-refinement steps (AIS/CIS) and measure how MAE and τ change versus simple clustering.
Optimization Features
Infra Optimization
Model Optimization
Training Optimization
Inference Optimization
Reproducibility
Risks & Boundaries
Limitations
Requires sufficient training models and prediction data to learn weights via gradient descent; performance drops with very small training sets.
Anchor refinement depends on a proxy model; poor proxy choices could reduce gains.
When Not To Use
When you have very few models or training scores (gradient-based weight learning may fail).
When precise per-item error analysis is required (SparseEval approximates global scores and rankings).
Failure Modes
Bad validation split causes poor anchor initialization and biased estimates.
Anchor set overfits the training model family and mispredicts novel architectures if hold-out families diverge.