Overview
Strong empirical evidence on multiple GPT-family models and datasets; code released. Some deployment caveats remain (hardware support for sparse kernels, semi-structured accuracy trade-offs).
Citations69
Evidence Strength0.90
Confidence0.88
Risk Signals10
Trust Signals
Findings with numeric evidence: 4/4
Findings with evidence refs: 4/4
Results with explicit delta: 3/5
Reproducibility
Status: Code + data available
Open source: Partial
At A Glance
Cost impact: 90%
Production readiness: 80%
Novelty: 70%
Why It Matters For Business
SparseGPT can cut model memory and inference compute roughly in half for massive GPT models, enabling cheaper hosting and faster inference without retraining. Joint sparsity+quantization can match lower-bit storage with better accuracy than pure quantization.
Who Should Care
Summary TLDR
SparseGPT is a fast post-training pruning method that can make very large GPT-family models (e.g., OPT-175B, BLOOM-176B) 50–60% sparse in one shot, without finetuning, while keeping perplexity and zero-shot accuracy nearly intact. The method reuses layer Hessian information to perform efficient weight reconstruction, supports hardware-friendly 2:4 and 4:8 patterns, and can be combined with weight quantization (e.g., joint 50% sparsity + 4-bit) for further memory savings. The authors provide code and report runs on a single A100 GPU (≈4 hours for 175B).
Problem Statement
Large GPT-family models are expensive to store and serve. Existing accurate pruning methods need expensive retraining or do not scale to 10–100B+ parameters. We need a fast, accurate post-training pruning method that works at GPT scale without retraining.
Main Contribution
SparseGPT: a one-shot, post-training pruning algorithm that scales to 10–100+B parameter GPT models without finetuning.
An efficient approximate reconstruction solver that reuses a sequence of inverse Hessians across columns to reduce computation and memory.
Key Findings
Large GPT models can be pruned to 50–60% unstructured sparsity in one shot with little accuracy loss.
SparseGPT runs quickly on a single GPU for the largest open models.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Perplexity (OPT-175B dense) | 8.35 | — | — | raw-WikiText2 | Dense OPT-175B perplexity reported as 8.35 | Table in Section 4 / Figure 1 |
| Perplexity (OPT-175B, 50% sparsity + 4-bit joint) | 8.29 | Dense (8.35) | -0.06 | raw-WikiText2 | 50% + 4-bit PPL 8.29 reported and compared to 3-bit baseline | Section 4 (Joint Sparsification & Quantization); Figure 6 |
What To Try In 7 Days
Run SparseGPT on a large model you already use and profile memory and latency (use 128 calibration segments).
Try joint 50% sparsity + 4-bit quantization and compare to your current quantized model for accuracy vs storage.
If targeting GPU speedups, test semi-structured 2:4/4:8 patterns and measure real end-to-end latency with your inference stack.
Optimization Features
Infra Optimization
Model Optimization
System Optimization
Training Optimization
Inference Optimization
Reproducibility
Data URLs
Risks & Boundaries
Limitations
Semi-structured patterns (2:4,4:8) reduce accuracy more on smaller models than on the largest ones.
Real end-to-end speedups depend on hardware and inference software; reported layer speedups may not equal full-system gains.
When Not To Use
If you need strictly lossless accuracy for your task (no tolerance for any PPL or task drop).
If your deployment stack lacks optimized sparse kernels or does not support 2:4-style acceleration.
Failure Modes
Simple magnitude pruning collapses accuracy at moderate sparsities for GPT-scale models (observed >30% collapse).
Joint semi-structured sparsity may harm smaller models more than large ones; poor layer selection for partial n:m leads to accuracy loss.