Overview
The idea is simple and implementable: offline permutation plus a column-permute trick avoids an AllGather. Results are measured on realistic hardware and models but apply primarily to MLP layers and GPTQ-style quantization.
Citations1
Evidence Strength0.70
Confidence0.88
Risk Signals9
Trust Signals
Findings with numeric evidence: 3/4
Findings with evidence refs: 4/4
Results with explicit delta: 4/4
Reproducibility
Status: No open assets linked
Open source: Partial
At A Glance
Cost impact: 70%
Production readiness: 70%
Novelty: 60%
Why It Matters For Business
A low-complexity, offline reorder can cut inter-GPU communication and speed up quantized LLM inference, lowering latency and increasing throughput for multi-GPU serving without changing model weights.
Who Should Care
Summary TLDR
TP-Aware Dequantization rearranges quantized weight storage so GPU-local metadata can be reused and a costly AllGather between column- and row-sharded layers is avoided. Applied to GPTQ-style 4-bit weights, the method targets MLP layers in Transformer blocks and yields up to ~1.8x latency speedups on Llama-70B and Granite-20B across A100/H100 multi-GPU nodes. The change is an offline reorder (permutation) of weight columns so dequantization and GEMM stay local, reducing global communication.
Problem Statement
GPTQ-style quantization stores per-group scales/zeros and an index mapping rows to groups. When GPTQ's activation-order reordering is used, metadata lookups become scattered and, under Tensor Parallel (TP), force extra AllGather communication between column- and row-sharded layers. This increases inference latency and reduces throughput for large LLMs in multi-GPU deployments.
Main Contribution
Identify that GPTQ act_order reordering breaks GPU data locality and causes extra AllGather in TP setups.
Propose an offline permutation-based reorder (argsort) to make group metadata consecutive and cache-friendly.
Key Findings
TP-Aware Dequantization speeds up MLP-layer inference in distributed LLMs.
Speedup grows with more tensor-parallel ranks.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Latency (Llama-70B, A100, TP=4, M=8) | Naive 0.518 ms → TP-Aware 0.285 ms | Naive Algorithm 0.518 ms | 1.82x speedup | MLP layer size (K1=8192,N1=28672,N2=8192) | Table 7 (Llama-70B TP=4 A100 M=8) | Table 7 |
| Latency (Llama-70B, A100, TP=8, M=4) | Naive 0.539 ms → TP-Aware 0.291 ms | Naive Algorithm 0.539 ms | 1.85x speedup | MLP layer size (K1=8192,N1=28672,N2=8192) | Table 11 (Llama-70B TP=8 A100 M=4) | Table 11 |
What To Try In 7 Days
If you use GPTQ quantized models with TP, test an offline argsort-based reorder of group indices and store the permutation.
For MLP layers, permute W1 columns by the downstream permutation and benchmark end-to-end latency to check AllGather removal.
Run microbenchmarks on your cluster at TP=2,4,8 to measure real speedups and pick where to deploy the change.
Optimization Features
Infra Optimization
Model Optimization
System Optimization
Inference Optimization
Reproducibility
Risks & Boundaries
Limitations
Method targets MLP layers only; attention layers need extra handling.
Gains are small for TP=1 and grow with number of TP ranks.
When Not To Use
Single-GPU inference (TP=1) where communication is absent.
When model uses different sharding/attention patterns not matching Column-TP → Row-TP sequence.
Failure Modes
Incorrectly applied permutations produce wrong alignment and wrong outputs.
Attention-layer sharding differences may reintroduce communication or require separate fixes.