Overview
OmniQuant is a practical PTQ upgrade: experiments show large models quantized on one GPU using public datasets, with clear gains in perplexity, accuracy, memory and throughput; some hardware and bit-format support (INT2/INT3) may still be immature.
Citations13
Evidence Strength0.85
Confidence0.85
Risk Signals10
Trust Signals
Findings with numeric evidence: 4/4
Findings with evidence refs: 4/4
Results with explicit delta: 4/4
Reproducibility
Status: Code + data available
Open source: Yes
At A Glance
Cost impact: 80%
Production readiness: 80%
Novelty: 60%
Why It Matters For Business
OmniQuant lets teams quantize large models to very low-bit formats with PTQ-level data and time budgets, cutting weight storage and often doubling throughput while keeping runtime identical to standard quantized models.
Who Should Care
Summary TLDR
OmniQuant is a post-training quantization (PTQ) pipeline that learns a small set of quantization parameters via gradient-based block-wise optimization. It adds Learnable Weight Clipping (LWC) and Learnable Equivalent Transformation (LET) to move quantization difficulty from activations to weights and to optimize clipping, while keeping full-precision weights frozen. With 128 calibration samples on a single A100 GPU, OmniQuant quantizes LLaMA/OPT/Falcon models (7B–180B) to low-bit formats (W4A4, W2A16, W6A6, W3A16) with large gains in perplexity and zero-shot accuracy, faster inference and much smaller weight storage, without adding runtime cost.
Problem Statement
Hand-crafted PTQ rules fail at very low bits (e.g., W4A4 or W2A16). Quantization-aware training (QAT) recovers accuracy but is expensive in GPU hours and data. Can we get QAT-level accuracy while keeping PTQ's time and data efficiency?
Main Contribution
A differentiable, block-wise PTQ pipeline (OmniQuant) that optimizes a small set of learnable quantization parameters instead of tuning all weights.
Learnable Weight Clipping (LWC) that adapts clipping strengths to reduce weight quantization error.
Key Findings
OmniQuant turns catastrophic W2A16 degradation into usable models.
Large average zero-shot accuracy gains at aggressive W4A4 weight-activation quantization.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Perplexity (generation) | LLaMA-13B W2A16 PPL=13.21 (OmniQuant) | GPTQ PPL=3832 (reported in text) | Huge reduction vs GPTQ | WikiText2 (reported in paper) | Intro & Table 1 | Figure 1 & Table 1 |
| Accuracy | LLaMA-7B W4A4 avg=52.65% (OmniQuant) | SmoothQuant avg=38.41% | +14.24 percentage points | Six zero-shot tasks (Table 2) | Section 4.3, Table 2 | Table 2 |
What To Try In 7 Days
Run OmniQuant on a 7B model with 128 calibration samples to measure memory and tokens/s gains.
Try W4A16g128 weight-only quantization first to get large memory savings and speedups with minimal accuracy drop.
If you must lower activation bits, run W4A4 with OmniQuant and evaluate zero-shot accuracy on key tasks before full deployment.
Agent Features
Tool Use
Architectures
Optimization Features
Token Efficiency
Infra Optimization
Model Optimization
System Optimization
Training Optimization
Inference Optimization
Reproducibility
Risks & Boundaries
Limitations
Requires per-model calibration and a small training pass (longer than pure GPTQ; shown ~5× slower than GPTQ).
Some low-bit modes (INT2/INT3) lack efficient hardware support today.
When Not To Use
When you have zero GPU time and must use fully training-free PTQ like plain GPTQ.
If your deployment hardware does not support the targeted low-bit integer formats.
Failure Modes
Over-aggressive SoftMax quantization (<=4-bit) can break generation quality.
Poor initialization or unstable gradients in LET can reduce benefit (some layers excluded in paper).