Atom can multiply token throughput per GPU and shrink KV-cache memory, lowering cloud GPU costs or increasing capacity without major task-accuracy loss.
Atom increases end-to-end serving throughput up to 7.73× vs FP16 and 2.53× vs INT8 under similar latency targets.
Numbers: 7.73× vs FP16; 2.53× vs INT8 (paper Figure 10 / §5.3.2)
LLM-QAT lets you reduce model memory and improve throughput by quantizing weights and KV cache to 4 bits while keeping quality close to full precision, which can lower hosting cost and enable longer contexts on the same hardware.
Data-free generated samples fine-tuned with logits distillation outperform real-data finetuning for zero-shot tasks.
Numbers: Generated-data (hybrid sampling) avg zero-shot 63.1 vs C4 61.5 (Table 3)
Shrinking KV cache at inference increases batch size and throughput on fixed-memory GPUs without retraining, cutting hosting cost or enabling longer contexts on the same hardware.
KV cache can be several times larger than model weights and becomes the memory bottleneck.
Numbers: KV cache 2.5–5× larger than weights (Table 1; e.g., OPT-175B weights 325GB vs KV 1152GB at batch128, seq2048)
Cut KV cache memory 3–7× and preserve accuracy so you can serve much longer contexts on existing GPUs, reducing infrastructure cost or enabling new long-document features.
3-bit KV cache with 1% sparse outliers keeps perplexity near fp16 on Wikitext-2
Numbers: LLaMA-7B PPL 5.75 vs fp16 5.68 (+0.07)
You can cut inference memory and often double throughput on long prompts by fine-tuning a small module, reducing costs for batched long-context services.
The fine-tuned model can remove ~80% of prior tokens with almost no perplexity loss.
Numbers: 80.35% sparsity → −0.085 avg perplexity (context=1000) vs dense
Using real LLM traces uncovers burst-driven failures and KV-cache pressure that synthetic tests miss, letting teams fix reliability and reduce wasted GPU costs before production.
Real LLM traces are highly bursty and differ from common cloud/function workloads.
Numbers: Mean RPS: MAF 1.64 vs LLM conv 0.019, LLM API 0.21 (ChatGPT)
If you sell many small fine-tuned models (per-user or per-task), S-LoRA lets one machine host thousands of adapters, cutting GPU costs and raising throughput compared to naive merging or swapping.
S-LoRA scales to thousands of adapters on one machine.
Numbers: Served 2,000 adapters on a single A100 (80GB) in experiments
QServe turns 4-bit quantization into real GPU speedups and memory savings, cutting serving cost per token (authors report ~3× dollar cost reduction by using L40S+QServe versus A100+TensorRT-LLM).
Prior 4-bit methods incur large runtime dequantization overhead on GPUs.
Numbers: 20–90% runtime overhead reported for dequantization
Keyformer cuts memory traffic and latency for long-context generation without retraining, lowering inference cost and enabling higher throughput on existing GPU servers.
Attention concentrates on a small subset of tokens ("key tokens").
Numbers: ≈90% of attention mass on ~40% of tokens (Fig.3b)
FastGen can double effective throughput and sharply cut tail latency on long-prompt, streaming chat workloads, which lowers GPU cost per successful request and improves user-perceived responsiveness.
Effective throughput improved up to 2.3× versus vLLM under chat-style SLAs.
Numbers: up to 2.3x effective throughput (Section 4.3)
QAQ can cut the GPU memory used by KV caches by ~8–10×, enabling longer-context features or reducing GPU requirements and cost with little accuracy loss.
Key vectors are more sensitive to quantization than value vectors; uniform 2-bit quantization harms keys far more than values.
Numbers: LLaMA-2-7B experiment: 2-bit value quantization retains high accuracy; 2-bit key quantization causes large accuracy drop
QuaRot makes production LLM inference much cheaper and memory-light by enabling true end-to-end 4-bit execution and large KV cache compression, so hosting large models on cheaper GPUs or smaller clusters becomes practical.
4-bit end-to-end quantization on LLAMA2-70B with small accuracy loss
Numbers: WikiText-2 PPL +0.47 (3.32→3.79); zero-shot avg drop ~1.09 pts
MiniCache cuts KV cache memory by up to 41% and can raise throughput ~5× without retraining, enabling lower GPU costs, larger batches, and longer contexts for production LLM services.
Up to 5.02× KV cache compression when combined with 4-bit KV quantization.
Numbers: 5.02× compression (Table 1, LongBench average)
MemServe lets operators combine caching and disaggregated inference to cut end-to-end latency and tail times for many chat and long-context workloads, lowering hardware cost per request and improving user-perceived responsiveness.
Disaggregated inference plus MemPool reduces job completion time compared to colocated baseline.
Numbers: JCT improved up to 42% (P99) on ShareGPT
WKVQuant cuts decoding memory of 13B models from ~27GB to ~7GB while keeping accuracy near full-precision; this enables cheaper GPU options and larger batch/sequence support without retraining.
WKVQuant (W4KV4) maintains long-input task performance close to full precision and weight-only quantization while far outperforming weight+activation (W4A4) on long-context tasks.
Numbers: LLaMA-2-13B Longtext avg: FP16 34.12, GPTQ W4 34.06, OmniQuant W4A4 16.35, WKVQuant W4KV4 32.52
A tiny precomputed full‑precision KV prefix fixes a large fraction of quality loss from 3–4 bit quantization, enabling cheaper LLM serving while keeping near‑full performance.
Pivot tokens produce very large activation peaks and create attention sinks that dominate attention scores.
Numbers: activation peaks >1e3 at pivot channels
FlashInfer can cut inference latency and increase throughput in production LLM services, lowering GPU costs per query and improving user responsiveness.
FlashInfer reduces inter‑token latency versus a Triton backend in LLM serving
Numbers: 29–69% ITL reduction (Sec. 4, Abstract)
Cuts serving memory by up to 8x and halves latency on long inputs while keeping task quality, letting teams process far larger documents at lower GPU cost.
Compression preserves generation quality on evaluated long-context benchmarks.
Numbers: Single-Doc: Ours 34.9 vs Full-FT 34.8 (LongBench Table 1)
If many requests reuse the same system prompt, ChunkAttention cuts attention latency and KV memory dramatically, letting you serve more users from the same GPUs or reduce cloud costs.
Self-attention kernel becomes 3.2–4.8× faster on A100 when many requests share long prompt prefixes.
Numbers: kernel speedup 3.2–4.8× (ns=1024..4096)
If your product uses chat or multi-turn flows, caching KV states and overlapping cache IO with GPU work can cut latency and cloud GPU costs dramatically.
Time-to-first-token (TTFT) drops dramatically when cached KV hits occur
Numbers: TTFT reduced by up to 87% (Figure 14)
Quest reduces memory bandwidth and decode latency for very long-context LLM calls, lowering GPU cost per request and improving responsiveness for document-heavy applications.
Quest achieves large self-attention speedups by loading only top-K pages instead of the full KV cache.
Numbers: 7.03× self-attention speedup at 32K seq, token budget 2048
For multimodal services that reuse the same images or files, MPIC can cut prefill latency roughly in half and double serving throughput, lowering per-request compute and improving capacity without changing model weights.
MPIC-32 reduces Time-to-First-Token (TTFT) by up to 54.1% versus prefix caching.
Numbers: TTFT reduced up to 54.1% (Fig.9; §5.2)
DéjàVu cuts wasted GPU time, reduces memory needs, and shortens recovery after node failures—so you can serve larger LLMs cheaper and more reliably in pipeline-parallel clusters.
Disaggregation increases throughput versus a pipeline-parallel baseline
Numbers: Up to 2× throughput improvement vs FasterTransformer on OPT-66B and BLOOM-176B
LoRC halves KV cache memory in many LLaMA deployments with near-zero impact on accuracy, lowering GPU cost and enabling larger batches or longer contexts on the same hardware.
LoRC reduces KV cache size by about 55–60% while keeping average performance drop under 1% on evaluated tasks.
Numbers: 55%–60% compression; avg perf drop <1%
