Overview
The method is an engineering-first approach: correct-by-construction transform, fused kernel to hide expansion, and broad multi-GPU benchmarks; expect reliable speedups for compute-bound prefill and large models but check driver maturity on target GPU.
Citations0
Evidence Strength0.80
Confidence0.85
Risk Signals10
Trust Signals
Findings with numeric evidence: 5/5
Findings with evidence refs: 5/5
Results with explicit delta: 2/5
Reproducibility
Status: Partial assets available
Open source: Partial
At A Glance
Cost impact: 65%
Production readiness: 75%
Novelty: 55%
Why It Matters For Business
SlideSparse lets teams deploy accuracy-preserving sparsity patterns (e.g., 6:8) and gain real GPU acceleration on existing hardware, reducing latency and compute cost without retraining.
Who Should Care
Summary TLDR
SlideSparse is a systems technique that makes milder structured sparsity patterns (the (2N-2):2N family like 6:8) run on existing NVIDIA Sparse Tensor Cores by decomposing each sparse block into overlapping 2:4 windows. It fuses the required activation rearrangement into per-token quantization, adding near-zero overhead. On real models and GPUs (A100/H100/B200/RTX) SlideSparse preserves accuracy (6:8 ~51.6% vs 54.0% dense on Qwen3) while delivering kernel and end-to-end speedups near the theoretical N/(N-1) bound (6:8 → ~1.33× end-to-end on A100 INT8). Code and conversion tools are provided.
Problem Statement
Hardware supports only rigid 2:4 (50%) N:M sparsity, which often breaks LLM accuracy. Practitioners prefer milder structured sparsity (e.g., 6:8, 25% pruning) that keeps accuracy but currently gets no hardware acceleration. The gap forces a trade-off: either accept big accuracy loss for 2× speed or keep accuracy with no speedup.
Main Contribution
Empirical gap: show 2:4 often collapses LLM reasoning accuracy while 6:8 preserves near-dense accuracy (Qwen3: 2:4 15.3% vs dense 54.0%; 6:8 51.6%).
Sliding Window Decomposition: a provably lossless transform that maps any (2N-2):2N block into N-1 overlapping 2:4 windows with optimal expansion γ=(2N-2)/N.
Key Findings
Milder structured sparsity (6:8) preserves reasoning accuracy while 2:4 destroys it.
SlideSparse attains theoretical speedup limits for (2N-2):2N on Sparse Tensor Cores.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Accuracy | dense 54.0%, 6:8 51.6%, 2:4 15.3% | dense | 6:8 −2.4 pp; 2:4 −38.7 pp | reasoning benchmarks (aggregate) | §2, Figure 2 | Figure 2 |
| End-to-end speedup (Qwen2.5-7B, A100, INT8, prefill) | 1.33× | dense cuBLASLt | — | prefill (M=8192) | §5.3, Abstract | §5.3, Figure 8 |
What To Try In 7 Days
Convert one large model checkpoint to (2N-2):2N masks (e.g., 6:8) using the offline packer and test accuracy vs dense.
Enable SlideSparse backend in vLLM and run end-to-end prefill on a representative workload (M≥4096) to measure throughput change.
Profile quant+slide overhead vs GEMM on your target GPU to confirm break-even M and expected speedup.
Optimization Features
Infra Optimization
Model Optimization
System Optimization
Training Optimization
Inference Optimization
Reproducibility
Code URLs
Risks & Boundaries
Limitations
Evaluated on post-hoc magnitude-pruned models; sparse-aware training might be needed for higher sparsity.
Small M (batch×seq) workloads often do not benefit because sparse kernel overhead dominates (M < ~256).
When Not To Use
Very low M workloads (small batches or single-token short contexts) where GEMM is not compute-bound.
On GPU platforms where cuSPARSELt or drivers lack stable sparse precision support for your precision.
Failure Modes
Driver or library bugs (cuSPARSELt/cuBLASLt/Triton) can produce large performance variance or errors.
If quantization/packing fails, the fused kernel can raise illegal-address or index overflow on extreme shapes.