Overview
Compress documents offline, finetune small LoRA adapters on those compressed embeddings, and at inference retrieve and prepend embeddings plus the matching adapter. This reduces token costs and latency while preserving or improving QA accuracy on evaluated datasets.
Citations1
Evidence Strength0.85
Confidence0.90
Risk Signals10
Trust Signals
Findings with numeric evidence: 5/5
Findings with evidence refs: 5/5
Results with explicit delta: 1/5
Reproducibility
Status: Code + data available
Open source: Partial
At A Glance
Cost impact: 80%
Production readiness: 75%
Novelty: 60%
Why It Matters For Business
LLoCO cuts token processing and GPU costs for long-document QA while improving accuracy and latency, letting teams serve very long documents without buying larger models or more GPUs.
Who Should Care
Summary TLDR
LLoCO compresses long documents offline into short token-embedding "cheat sheets" and then finetunes small LoRA adapters on those embeddings. At inference it retrieves compressed embeddings and the matching adapter, prepends embeddings to the LLM, and generates answers. On LLaMA2-7B this extends effective context to 128k tokens, uses ~30× fewer tokens, gives up to 7.62× inference speedup and up to 11.52× finetuning throughput on A100, and improves long-document QA on several benchmarks. Main limits: the compressor is tied to a specific LLM and adapters are per-document-group.
Problem Statement
Transformer LLMs slow down and run out of GPU memory on very long documents because self-attention and KV caches scale poorly with sequence length. This raises latency, GPU costs, and token billing for long-document QA and summarization.
Main Contribution
Introduce LLoCO: offline context compression + in-domain LoRA finetuning + retrieval of compressed embeddings at inference.
Extend LLaMA2-7B (4k) to effectively handle up to 128k tokens using compressed embeddings.
Key Findings
LLoCO raises average QA performance vs base LLaMA2-7B on evaluated long-doc tasks.
Inference token footprint is reduced by about 30× via compression.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Average QA score (selected long-doc tasks) | 30.67 (LLoCO) | 23.44 (LLaMA2-7B-4k) | +7.23 | QuALITY, Qasper, QMSum, NarrativeQA, HotpotQA (validation) | Table 1 shows LLoCO avg 30.67 vs baseline 23.44 | Table 1 |
| Compression ratio (tokens) | 30× | — | — | AutoCompressor setting (1536→50 embeddings) | AutoCompressor compresses chunks 1536→50; used across experiments | Sec 3, Table 1 |
What To Try In 7 Days
Compress a sample collection with an available compressor (AutoCompressor or ICAE) and store embeddings in your vector DB.
Finetune one small LoRA adapter on compressed embeddings for a representative document group and validate QA accuracy.
Serve queries by retrieving compressed embeddings + the matching LoRA adapter, and measure token cost and latency vs your current RAG baseline.
Optimization Features
Token Efficiency
Infra Optimization
Model Optimization
System Optimization
Training Optimization
Inference Optimization
Reproducibility
Data URLs
Risks & Boundaries
Limitations
Context encoder (AutoCompressor) is tied to a specific base LLM; a new encoder is needed per model.
Training a high-quality compressor can be costly (authors note ~15B tokens used).
When Not To Use
You need a model-agnostic compressor that works unchanged across many LLMs.
Tasks require long, detailed generated outputs rather than concise Q&A answers.
Failure Modes
Hallucination if LoRA adapter is not trained for the document distribution or compression loses critical facts.
Performance drops on out-of-distribution documents and datasets not represented in finetuning.