Overview
PQCache is a practical, engineering-focused system: it combines a classic retrieval technique (PQ) with careful overlap and caching to trade CPU clustering for preserved model quality and low latency.
Citations0
Evidence Strength0.80
Confidence0.80
Risk Signals10
Trust Signals
Findings with numeric evidence: 5/5
Findings with evidence refs: 5/5
Results with explicit delta: 3/3
Reproducibility
Status: No open assets linked
Open source: Unknown
At A Glance
Cost impact: 70%
Production readiness: 70%
Novelty: 60%
Why It Matters For Business
PQCache reduces GPU memory needs for long-context LLMs while keeping or improving accuracy, lowering hardware cost and enabling longer-context features without costly GPU scaling.
Who Should Care
Summary TLDR
PQCache treats KVCache (keys/values used in self-attention) as an embedding retrieval problem. It compresses keys with Product Quantization (PQ) during prefilling, stores PQ codes/centroids on CPU/GPU, and at decode time uses approximate PQ search to fetch a small top-k subset of key-value pairs. This reduces GPU memory pressure while retaining or improving model quality across long-context benchmarks (e.g., +4.60% on InfiniteBench) and keeps per-token latency low via overlapping, prefetching, and a block-level GPU cache.
Problem Statement
KVCache (stored keys and values) grows linearly with sequence length and quickly exceeds GPU memory for long-context inference. Existing methods that drop or offload KVCache either hurt model quality or add unacceptable latency. A solution must keep model quality while minimizing I/O and latency.
Main Contribution
Reframe KVCache management as an approximate nearest-neighbor (ANNS) retrieval problem and apply Product Quantization (PQ) to keys
PQCache system that: PQ-compresses keys per head/layer during prefilling; keeps PQ centroids on GPU; uses PQ search to find top-k relevant tokens and fetch only their KV pairs
Key Findings
PQCache improves aggregate InfiniteBench scores versus prior selective-attention methods
On LongBench PQCache beats baselines when using fewer tokens
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| InfiniteBench average score delta vs baselines | +4.60% | existing selective-attention/offloading baselines | +4.60% | InfiniteBench (avg) | Sec. 1; Sec.4.2.2; Abstract | Tables and text |
| LongBench average score delta vs baselines | +1.74% (1/5 tokens); +3.90% (1/10 tokens) | existing methods (SPARQ, InfLLM, H2O, SnapKV, PyramidKV) | +1.74% / +3.90% | LongBench (Llama-3.1-8B) | Sec.4.2.1 Table 2 | Table 2 |
What To Try In 7 Days
Run PQCache on a dev GPU+CPU server to offload KVCache and verify end-to-end latency on your longest prompts
Tune PQ hyperparameters (m, b) with low iterations to fit CPU budget, then test selective-attention ratios (e.g., 1/5, 1/10)
Add a small block-level GPU cache (32 blocks) and measure hit-rate and per-token latency to balance cost and speed
Agent Features
Memory
Optimization Features
Token Efficiency
Infra Optimization
System Optimization
Inference Optimization
Reproducibility
Risks & Boundaries
Limitations
Performance depends on CPU clustering capacity; limited clustering iterations can reduce accuracy
PQ is an approximation: poor centroids or too few clusters hurt retrieval recall
When Not To Use
GPU-only environments without spare CPU or slow CPU where clustering would block GPU
Workloads with extremely strict single-token latency constraints that cannot tolerate any CPU round-trip
Failure Modes
PQ approximation misses important keys causing degraded generation
adaptive K-Means clipping too aggressive, producing poor centroids