Overview
Clear method and convincing small-scale experiments on GPT-2 show big bandwidth savings, but results come from 3 sample sets and require hardware kernel work for deployment.
Citations0
Evidence Strength0.70
Confidence0.75
Risk Signals10
Trust Signals
Findings with numeric evidence: 4/4
Findings with evidence refs: 4/4
Results with explicit delta: 5/5
Reproducibility
Status: No open assets linked
Open source: Unknown
At A Glance
Cost impact: 80%
Production readiness: 65%
Novelty: 60%
Why It Matters For Business
LOOKAT can cut KV-cache memory and DRAM bandwidth on edge devices by tens of times without retraining, enabling larger context or lower-cost hardware for real-time inference.
Who Should Care
Summary TLDR
LOOKAT replaces standard attention scoring with a lookup-table method built from product quantization and asymmetric distance computation (ADC). By compressing keys into small codebook indices and precomputing query×codebook dot-products, LOOKAT avoids dequantizing keys and cuts KV-cache bandwidth drastically. On GPT-2 experiments it reaches 64× compression with ~0.957 cosine output fidelity and Spearman rank correlation >0.95, without training or architecture changes. Main limits: values stay FP16 and lookup kernels need hardware support.
Problem Statement
KV-cache memory grows with sequence length and dominates edge inference memory. Standard INT4/INT8 quantization lowers storage but still needs dequantization, so bandwidth remains the bottleneck. The paper asks: can attention scoring be computed directly on compressed keys to remove the dequantization bandwidth cost?
Main Contribution
Show attention scoring is equivalent to inner-product retrieval and can use product quantization + ADC.
Introduce LOOKAT: compute attention scores from codebook indices via precomputed lookup tables, avoiding key dequantization.
Key Findings
LOOKAT achieves 64× KV-cache compression while keeping model output close to FP16.
LOOKAT preserves attention ranking structure.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| LOOKAT-2 cosine similarity (output fidelity) | 0.957 | FP16 baseline 1.000 | −0.043 vs FP16 | GPT-2, mixed text samples | Table 1 reports LOOKAT2 (64×) cosine sim 0.957 | Table 1 |
| LOOKAT-4 cosine similarity (output fidelity) | 0.950 | FP16 baseline 1.000 | −0.050 vs FP16 | GPT-2, mixed text samples | Table 1 reports LOOKAT4 (32×) cosine sim 0.950 | Table 1 |
What To Try In 7 Days
Profile your model's KV-cache bandwidth; confirm attention scoring is the bottleneck.
Implement a PQ+ADC prototype for keys (m=4) and measure per-query DRAM load reduction.
Run functional checks on a few representative prompts to compare top-k attention overlap with FP16.
Optimization Features
Token Efficiency
Infra Optimization
Model Optimization
System Optimization
Training Optimization
Inference Optimization
Reproducibility
Risks & Boundaries
Limitations
Only compresses keys; values remain FP16 so total memory savings are partial.
Codebook quality depends on calibration data and domain; results reported on three small sample types.
When Not To Use
When you need exact attention magnitudes rather than relative ordering.
If your hardware cannot support fast table lookups or optimized byte-index kernels.
Failure Modes
Long contexts degrade fidelity (cosine drops ~10% at 1024 tokens).
High KL divergence in some samples can alter attention mass distribution.