Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGPrompt OptimizationAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection

Cut KV-cache costs by predicting important tokens from RoPE frequency chunks

LLM ArchitecturesSparse TransformersSparsityMethod
February 3, 20267 min

Overview

Decision SnapshotReady For Pilot

FASA is practically ready: one-time calibration, small code change, and clear gains on long-context benchmarks make it suitable for production trials on long-input workloads.

Citations0

Evidence Strength0.80

Confidence0.85

Risk Signals9

Trust Signals

Findings with numeric evidence: 4/4

Findings with evidence refs: 4/4

Results with explicit delta: 4/4

Reproducibility

Status: Code + data available

Open source: Partial

At A Glance

Cost impact: 80%

Production readiness: 80%

Novelty: 60%

Authors

Yifei Wang, Yueqi Wang, Zhenrui Yue, Huimin Zeng, Yong Wang, Ismini Lourentzou, Zhengzhong Tu, Xiangxiang Chu, Julian McAuley

Links

Abstract / PDF / Code / Data

Why It Matters For Business

FASA cuts GPU memory needs and memory bandwidth during long-input inference with almost no accuracy loss, lowering hosting costs and enabling long-context features on smaller hardware.

Who Should Care

CTOML EngineerEngineering LeadFounderData ScientistProduct Manager

Summary TLDR

FASA is a training-free, two-stage method that predicts which past tokens matter for each query by inspecting a small set of RoPE frequency chunks (FCs). It uses those FCs to cheaply rank tokens (TIP), then runs full attention only on the top tokens (FAC). Across long-context benchmarks and long-generation tasks, FASA matches near-full KV performance while cutting memory and bandwidth: nearly 100% of full-KV on LongBench-V1 with 256 tokens, up to 8× KV memory reduction (FASA-M), and up to 2.56× end-to-end speedup in long-chain reasoning with small cache usage.

Problem Statement

Long inputs make KV caches huge and memory-bound. Existing token-eviction heuristics either lose information (static) or need costly training and still miss query-dependent importance. We need a cheap, query-aware way to keep only the tokens that actually matter during decoding.

Main Contribution

Discovered functional sparsity in RoPE: a tiny subset of frequency chunks ('dominant FCs') drives contextual attention.

Proposed FASA, a two-stage, training-free pipeline: TIP (cheap token scoring via dominant FCs) + FAC (full attention on selected tokens).

Key Findings

Dominant FCs are extremely sparse: a tiny fraction of FCs explain contextual attention.

NumbersDominant FCs ≤ 0.8% vs non-dominant ≈ 89–95% (Table 9)

Practical UseYou can inspect and store only a few FCs per head to predict which tokens matter, saving large memory without retraining.

Evidence RefTable 9 (A.4)

Dominant FC sets are stable across tasks and models.

NumbersCross-task dominant-FC overlap ≈ 70–87% across datasets (Table 10)

Practical UseRun a one-time, small offline calibration to get universal FC indices that work across tasks and models.

Evidence RefTable 10 (A.4)

Results

MetricValueBaselineDeltaSplit / DatasetEvidenceEvidence Ref
LongBench-V1 average (compared to full KV)≈100% of full-KV when keeping 256 tokensFKV (full KV)near 0% lossLongBench-V1Nearly 100% of full-KV performance at K=256Abstract; Table 2
Decoding speedup (end-to-end)2.56×FKV2.56× fasterAIME24 (long-CoT)2.56× speedup using 18.9% of the cacheAbstract; Figure 7; Table 3

What To Try In 7 Days

Run the one-time offline FC calibration on your model with a small calibration set (paper used a single sample).

Apply the provided FASA monkey-patch to FlashAttention2 and benchmark latency and KV memory on a 16k–32k workload.

If VRAM is tight, test FASA-M to offload non-dominant KV parts to CPU and measure end-to-end generation latency with prefetching enabled.

Optimization Features

Token Efficiency
query-aware top-k token selectionTIP: low-dim scoring using dominant FCs
Infra Optimization
just-in-time CPU→GPU transfers (FASA-M)sparse on-GPU key access (FASA-C)
Model Optimization
low-dimensional FC selection
System Optimization
reduced memory bandwidthintegration with FlashAttention2compatibility with PyramidKV
Inference Optimization
token selection (sparse attention)reduced KV readsGPU-CPU offload (FASA-M)

Reproducibility

Code AvailableYes
Data AvailableYes
Open Source StatusPartial
LicenseUnknown

Code URLs

https://github.com/AMAP-ML/FASA-ICLR2026

Data URLs

LongBench (public)MATH500 (public)AIME24 (public)PG-19 / WikiText / C4 (public)

Risks & Boundaries

Limitations

Relies on positional encodings that expose FC structure (RoPE-like). non-RoPE models need validation though ALiBi and Partial-RoPE showed compatibility.

FASA-M adds CPU↔GPU transfers which need careful prefetching to avoid latency regressions.

When Not To Use

Short-context workloads where KV cache is not a bottleneck.

Setups that cannot tolerate any risk of dropped rare tokens (extremely safety-critical outputs).

Failure Modes

Misidentifying important tokens under rare, atypical queries causes significant accuracy loss.

Replacing full attention with FC-proxy scores (instead of selecting tokens) yields catastrophic degradation.

Core Entities

Models

Llama-3.2-3BLlama-3.1-8BMistral-7B-v0.3Qwen2.5-7BQwen2.5-14BQwen2.5-32BR1-Distill-Llama-8BDeepSeek-R1 variants

Metrics

f1ROUGEperplexity (PPL)pass@1speedupcompression ratio

Datasets

LongBench-V1QasperGovReportNarrativeQA2WikimqaPG-19WikiTextC4MATH500AIME24

Benchmarks

LongBenchMATHAIME

You May Also Want to Read

Fine-grained Mixture-of-Experts (G=8) cuts training steps and improves accuracy at 56B scale

0.60
0.60
0.70
0

Fine-grained MoE can match or beat more expensive MoE variants while lowering training steps and inference-active compute, so you can get higher-quality models with less compute or cheaper inference at 50B-scale.

Key finding

Fine-grained MoE (G=8) lowers validation loss and raises average benchmark scores versus standard MoE at large scale.

Numbers: 56B: Avg accuracy 1xG1=57.3 -> 1xG8=59.0; Valid loss 1.811 -> 1.779

Instruction tuning unlocks Mixture-of-Experts: similar or better accuracy at ~1/3 the compute

0.70
0.50
0.80
20

Combine instruction tuning with MoE to cut runtime compute and costs: MoE models can match or beat dense baselines while using much less per-token FLOPs, so this reduces inference cost without sacrificing accuracy on many English tasks.

Key finding

Instruction tuning increases MoE gains vs dense models.

Numbers: 7.1% absolute gain on MMLU-Direct (avg) for FLAN‑MOE over dense at similar FLOPs

Practical training recipes for MoE LLMs: when to upcycle, and how to diversify experts

0.60
0.60
0.70
3

MoE models can cut compute per-token while keeping high capacity; pick upcycling when you have a good dense checkpoint and limited budget, otherwise invest in from-scratch MoE to maximize final quality.

Key finding

Upcycling vs scratch depends on training budget: with ~2× dense training budget, from-scratch MoE wins; with smaller budgets, upcycling helps.

Numbers: 100B tokens ≈ 2/3 C, 300B tokens ≈ 2 C; scratch outperforms upcycled at 300B budget

Run accuracy-preserving 6:8 sparse LLMs on current GPUs and get ~1.33× inference speed with no model changes

0.75
0.55
0.65
0

SlideSparse lets teams deploy accuracy-preserving sparsity patterns (e.g., 6:8) and gain real GPU acceleration on existing hardware, reducing latency and compute cost without retraining.

Key finding

Milder structured sparsity (6:8) preserves reasoning accuracy while 2:4 destroys it.

Numbers: Qwen3 reasoning: dense 54.0% → 6:8 51.6% vs 2:4 15.3%

Survey of techniques, hardware, and trade-offs to run LLMs directly on phones and edge devices

0.70
0.55
0.80
2

On-device LLMs cut latency, protect user data, and lower cloud bills—key benefits for mobile apps, privacy-focused services, and offline products.

Key finding

Edge AI market projected to grow nearly tenfold to $143.6B by 2032.

Numbers: 2022 $15.2B → 2032 $143.6B; CAGR 25.9%