Overview
The paper provides a reproducible recipe, weights, and multi‑scale benchmarks; results are strong on retrieval and ultra‑long QA but depend on the provided SFT mix and retriever tuning.
Citations0
Evidence Strength0.85
Confidence0.82
Risk Signals9
Trust Signals
Findings with numeric evidence: 6/6
Findings with evidence refs: 6/6
Results with explicit delta: 4/5
Reproducibility
Status: Code + data available
Open source: Yes
At A Glance
Cost impact: 70%
Production readiness: 70%
Novelty: 65%
Why It Matters For Business
You can run an open‑source 70B model that reads 100K+ tokens and often matches or beats commercial models on retrieval and long‑document QA, reducing dependence on closed APIs and giving control over data and cost.
Who Should Care
Summary TLDR
ChatQA 2 is a Llama‑3.0 based model extended from 8K to 128K context and instruction‑tuned in three stages. The 70B model outperforms many public and commercial baselines on ultra‑long benchmarks (>100K tokens) and on RAG short‑context tasks, while being competitive on 32K tasks. The recipe (pretraining data, tuning data, retriever setup) and weights are released so practitioners can reproduce or adapt the approach.
Problem Statement
Open-source LLMs lag proprietary models for very long context and retrieval‑augmented tasks. The paper asks: can we extend an 8K Llama3 model to 128K context, tune it for instruction following and RAG, and match proprietary performance on realistic long documents?
Main Contribution
A reproducible recipe to extend Llama3‑70B from 8K to 128K context by continued pretraining and RoPE frequency scaling.
A three‑stage instruction tuning pipeline that mixes short and synthetic long SFT data to improve instruction following, RAG, and ultra‑long understanding.
Key Findings
ChatQA‑2‑70B achieves top average on four ultra‑long InfiniteBench tasks.
On short RAG conversational benchmarks (within 4K), ChatQA‑2‑70B outperforms several 128K context models.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Ultra-long average (InfiniteBench) | 41.04 | GPT‑4‑Turbo 33.16 | +7.88 | InfiniteBench (4 tasks beyond 100K) | Table 2: Llama3‑ChatQA‑2‑70B avg 41.04 vs GPT‑4‑Turbo 33.16 | Table 2 |
| Long (32K) average | 48.15 | GPT‑4‑Turbo 51.93 | -3.78 | 32K long tasks (LongBench/SCROLLS) | Table 3: Llama3‑ChatQA‑2‑70B 48.15 vs GPT‑4‑Turbo 51.93 | Table 3 |
What To Try In 7 Days
Reproduce their 128K recipe on a smaller scale (8B) using provided weights and data to validate retrieval on your docs.
Swap in a long‑context retriever (E5‑mistral or NV‑emb‑v2) and test RAG with increasing top‑k until accuracy plateaus.
For QA over very long docs, try RAG with k×chunk_size ≥12K tokens before using full long prompts.
Agent Features
Memory
Optimization Features
Token Efficiency
Infra Optimization
Model Optimization
System Optimization
Training Optimization
Inference Optimization
Reproducibility
Code URLs
Data URLs
Risks & Boundaries
Limitations
Lower summarization scores due to limited summarization SFT data.
Not as strong on some knowledge/coding benchmarks (MMLU, HumanEval) without RLHF/DPO.
When Not To Use
If you need top tier summarization quality without extra SFT data.
When coding or knowledge‑intensive benchmarks are the main goal without RLHF.
Failure Modes
Too many irrelevant retrieved chunks can degrade generation if not tuned.
Fragmentation from small chunk sizes hurts context continuity for some tasks.