Overview
Results cover multiple model sizes, include kernel-level benchmarks, and show consistent PPL/accuracy and speed/memory improvements on RTX 3090; caveats remain for smaller models and alternative architectures.
Citations3
Evidence Strength0.90
Confidence0.90
Risk Signals9
Trust Signals
Findings with numeric evidence: 4/4
Findings with evidence refs: 4/4
Results with explicit delta: 2/5
Reproducibility
Status: Partial assets available
Open source: Partial
At A Glance
Cost impact: 80%
Production readiness: 80%
Novelty: 70%
Why It Matters For Business
QuaRot makes production LLM inference much cheaper and memory-light by enabling true end-to-end 4-bit execution and large KV cache compression, so hosting large models on cheaper GPUs or smaller clusters becomes practical.
Who Should Care
Summary TLDR
QuaRot uses randomized Hadamard rotations fused into model weights to remove large activation outliers, enabling end-to-end 4-bit inference (weights, activations, KV cache) without keeping special high-precision channels. On LLAMA2-70B QuaRot gives at most +0.47 WikiText-2 perplexity, preserves ~99% of zero-shot accuracy, yields up to 3.33× prefill speedup and ~3.89× decoding memory savings on consumer GPUs. Code: github.com/spcl/QuaRot.
Problem Statement
Activation outliers (rare large values) make activation and KV-cache quantization hard. Prior fixes keep outliers in higher precision or use calibration. This prevents true end-to-end 4-bit inference with acceptable accuracy and memory savings.
Main Contribution
A practical method (QuaRot) that fuses randomized Hadamard rotations into transformer weights to remove activation outliers without changing model outputs.
An attention-aware extension that rotates keys and values so the KV cache can be quantized.
Key Findings
4-bit end-to-end quantization on LLAMA2-70B with small accuracy loss
Significant runtime and memory gains on consumer GPUs
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| WikiText-2 Perplexity (LLAMA2-70B) | 3.79 (QuaRot GPTQ INT4) | 3.32 (FP16) | +0.47 | WikiText-2, seq len 2048 | Table 1, Table 9 | Table 1 |
| Zero-shot average (LLAMA2-70B) | 75.98 (QuaRot GPTQ INT4) | 77.07 (FP16) | -1.09 | LM Eval Harness tasks (avg over PIQA, WG, HS, LA, ARC) | Table 2, Table 9 | Table 2 |
What To Try In 7 Days
Run QuaRot on a copy of your FP16 LLAMA-2/3 model using the published code and compare WikiText-2 PPL and a couple of zero-shot tasks.
Measure prefill throughput and KV memory usage on your target GPU, focusing on large batch and long context.
If you need conservative rollout, try RTN 8-bit or 6-bit first — they are near-lossless and need no calibration data.
Agent Features
Memory
Tool Use
Frameworks
Architectures
Optimization Features
Infra Optimization
Model Optimization
System Optimization
Inference Optimization
Reproducibility
Code URLs
Risks & Boundaries
Limitations
Benefits depend on fast INT4 GEMM kernels; you need CUDA/CUTLASS support or similar.
Random orthogonal transforms perform worse than Hadamard; hidden dimension constraints (power-of-two factors) simplify fast Hadamard use.
When Not To Use
If your target hardware lacks fast INT4 support or optimized kernels.
When you cannot afford any drop in small-model accuracy; RTN 4-bit can fail on small models.
Failure Modes
Round-to-nearest (RTN) at 4 bits can cause large quality drops on small models; GPTQ is safer for small sizes.
Using random orthogonal matrices instead of structured Hadamard increases perplexity.