Overview
Paper provides clear empirical evidence at 405M and 10B scales and public code/checkpoints; results are robust for two dataset shifts but limited to these scales and experimental choices.
Citations3
Evidence Strength0.80
Confidence0.85
Risk Signals11
Trust Signals
Findings with numeric evidence: 5/6
Findings with evidence refs: 6/6
Results with explicit delta: 5/5
Reproducibility
Status: Code + data available
Open source: Partial
At A Glance
Cost impact: 80%
Production readiness: 70%
Novelty: 45%
Why It Matters For Business
You can update large LLMs on fresh data at far lower compute cost than full re-training while keeping model quality similar, cutting operational cost and turnaround time for model updates.
Who Should Care
Summary TLDR
The paper shows that three simple, scalable tricks—learning-rate (LR) re-warming, LR re-decaying, and replaying a small fraction of past data—let you continually update decoder-only transformer LLMs (405M and 10B parameters) on hundreds of billions of new tokens while matching the performance of full re-training on pooled data. Re-warming improves adaptation but causes forgetting; adding modest replay (e.g., 1–5% for similar-language updates, ~25% for stronger shifts like adding German) recovers past performance. The authors also propose 'infinite' LR schedules that avoid re-warming and match cosine-decay performance in initial tests.
Problem Statement
Re-training large language models from scratch whenever new pre-training data arrives is costly. Naively continuing training can either not adapt to new data or erase previous knowledge (catastrophic forgetting). The paper asks: can simple, cheap continual pre-training rules match the performance of full re-training on the union of datasets?
Main Contribution
Show that LR re-warming + LR re-decaying is necessary to adapt LLMs when continuing pre-training from a low final LR.
Demonstrate that compute-equivalent replay (small fraction of past tokens) largely prevents forgetting while preserving adaptation.
Key Findings
Re-warming then re-decaying the learning rate is required to adapt well to new pre-training data.
Small replay fractions substantially reduce forgetting with little hit to adaptation.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Average final validation loss (avg of D0 and D1) | 10B: 1.89 (300B Pile→300B SP, 5% replay) | 600B union-trained: 1.87 | ≈ +0.02 | Pile ∪ SlimPajama | Table 4 average losses for 10B models | Table 4 |
| Average final validation loss (avg of D0 and D1) | 405M: 2.37 (300B Pile→300B SP, 5% replay) | 600B union-trained: 2.35 | ≈ +0.02 | Pile ∪ SlimPajama | Table 4 average losses for 405M models | Table 4 |
What To Try In 7 Days
If you have a pre-trained checkpoint that ended at a low LR, re-warm the LR and re-decay it while continuing on new data.
Add compute-equivalent replay of past data; start with 1–5% for similar-language updates, test ~25% for big shifts.
Run small-scale tests: compare final validation loss and a few downstream tasks to a union-trained baseline to pick replay percent and LR max.
Optimization Features
Token Efficiency
Training Optimization
Reproducibility
Code URLs
Data URLs
Risks & Boundaries
Limitations
Only two model sizes tested (405M, 10B); behavior at 100B+ unknown.
German validation set was not deduplicated from German training data (possible contamination).
When Not To Use
When the new data is a strict per-domain sequential stream (domain-incremental) without mixing—paper found poor results for that setting.
When the tokenizer cannot cover the new distribution (e.g., adding a very different language) without tokenization changes.
Failure Modes
LR re-warming can cause transient spikes in past-data loss and accelerate forgetting if not paired with replay.
Too much replay (large fraction like 50%) can hurt adaptation to new data.