Overview
The approach is simple and tested on multiple LLaMA/OPT sizes with clear runtime and accuracy gains; results are empirical and require hardware support for mixed low-bit FP/INT for WA scenarios.
Citations5
Evidence Strength0.78
Confidence0.85
Risk Signals10
Trust Signals
Findings with numeric evidence: 5/5
Findings with evidence refs: 5/5
Results with explicit delta: 4/4
Reproducibility
Status: No open assets linked
Open source: No
At A Glance
Cost impact: 80%
Production readiness: 70%
Novelty: 60%
Why It Matters For Business
Mixing low-bit INT and FP per layer can keep model accuracy while cutting model size and quantization time; it fits current hardware that supports both INT and FP low-bit ops and reduces deployment cost.
Who Should Care
Summary TLDR
The paper compares low-bit integer (INT) and floating-point (FP) formats for LLM quantization and finds no single best format across layers. It proposes Mixture of Formats Quantization (MoFQ): pick INT or FP per layer at the same bit-width. MoFQ gives SOTA PTQ results: similar or better accuracy than prior methods for 4-bit weight-only quantization, much faster quantization time than GPTQ, and near full‑precision accuracy for 8-bit weight+activation quantization on LLaMA/OPT benchmarks. The method is simple, hardware-friendly, and works without changing bit-width.
Problem Statement
Low-bit quantization is needed to shrink LLM size and cost, but it is unclear whether integer (INT) or low-bit floating point (FP) formats work better. Tensors and layers vary in distribution, so a one-format-fits-all choice may be suboptimal. Practitioners need a fast, practical rule to pick formats that keeps accuracy and reduces inference cost.
Main Contribution
Comparative analysis of INT vs FP formats across bit widths, hardware cost, and quantization error on LLaMA/OPT tensors
MoFQ: a simple layer-wise format selector that picks INT or FP (same bit-width) using a chosen error metric
Key Findings
No single format (INT or FP) dominates across layers and bit widths.
MoFQ (per-layer INT/FP choice) matches or improves accuracy vs prior PTQ on 4-bit W-only quantization.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| WikiText-2 perplexity (LLaMA-65B, W-only 4-bit) | MoFQ4=3.78 | INT4(GPTQ)=3.85; FP16=3.53 | -0.07 vs GPTQ | WikiText-2 | Table 2 reports perplexities for LLaMA-65B. | Table 2 |
| Quantization time (LLaMA-65B, W-only 4-bit) | FP4=36s; MoFQ4=319s | INT4(GPTQ)=4684s | FP4 ≈130x faster; MoFQ4 ≈14.7x faster | runtime measurement | Table 3 runtime comparison across LLaMA sizes. | Table 3 |
What To Try In 7 Days
Run per-layer MSE-based format selection (MoFQ) on one LLM checkpoint using existing PPQ/GPTQ tools
Benchmark FP4 weight-only quantization vs INT4(GPTQ) on a single downstream task to compare runtime and accuracy
If using 4-bit FP weights in software, use the FP4 NaN/Inf reallocation trick to reduce tensor error
Optimization Features
Infra Optimization
Model Optimization
System Optimization
Inference Optimization
Reproducibility
Risks & Boundaries
Limitations
Analysis and results are empirical; no theoretical guarantees provided
MoFQ sometimes does not outperform a single FP4 or INT4—selection can be imperfect
When Not To Use
Target hardware only supports INT low-bit and not FP (WA case)
You need strict IEEE FP behavior or hardware-validated FP formats
Failure Modes
Per-layer metric (e.g., tensor MSE) may mispredict the format that best preserves end-task accuracy
FP4 redesign might be incompatible with hardware inference paths, producing incorrect runtime behavior