Overview
The paper provides concrete training recipes, open code and weight releases, and benchmark numbers; results are repeatable for practitioners with TPU/GPU access.
Citations4
Evidence Strength0.80
Confidence0.90
Risk Signals12
Trust Signals
Findings with numeric evidence: 6/6
Findings with evidence refs: 6/6
Results with explicit delta: 7/7
Reproducibility
Status: Code + data available
Open source: Partial
At A Glance
Cost impact: 70%
Production readiness: 70%
Novelty: 50%
Why It Matters For Business
XGen-7B gives teams a practical, open 7B model that handles long documents (up to 8K tokens) and competitive instruction-following, lowering cost versus much larger closed models while keeping good accuracy.
Who Should Care
Summary TLDR
Salesforce trains and releases XGen-7B, a family of 7-billion-parameter models trained stage-wise to handle up to 8K input tokens. Key facts: base models saw up to 1.5 trillion training tokens (800B@2K, 400B@4K, 300B@8K). They release two instruction-tuned variants (WizardLM-based and a general public-data mix). Evaluations show XGen-7B matches or beats other open-source 7B models on standard benchmarks and shows clear gains on long-context tasks. Models and code are open-sourced.
Problem Statement
Most competitive open-source LLMs are trained with a 2K token limit. That prevents reliable handling of long documents (summaries, long code, transcripts). The paper aims to produce a practical, open 7B model that reliably uses context up to 8K tokens and remains efficient to serve.
Main Contribution
Train a 7B dense-attention LLM (XGen-7B) with stage-wise context growth: 2K → 4K → 8K tokens.
Scale total token budget to 1.5T tokens (800B, 400B, 300B splits) for the 8K model.
Key Findings
Stage-wise training yields an 8K-capable model that uses long context.
XGen-7B matches or slightly improves over LLaMA-7B on standard benchmarks.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| MMLU five-shot weighted average | 36.3% | LLaMA-7B 35.1% | +1.2 pp | MMLU five-shot | Table 4: XGen-7B 36.3 vs LLaMA-7B 35.1 | Table 4 |
| MMLU zero-shot weighted average | 32.1% | LLaMA-7B 32.0% | +0.1 pp | MMLU zero-shot | Table 5: XGen-7B 32.1 vs LLaMA-7B 32.0 | Table 5 |
What To Try In 7 Days
Run XGen-7B-8K on your long-doc tasks (documents >2K tokens) and compare summaries/QA against your 2K models.
Replace a 13–40B inference endpoint with XGen-7B-Inst for inexpensive dev/testing of assistant flows.
Fine-tune or prompt-engineer XGen-7B-Inst wizardLM on a small in-domain instruction set for rapid assistant prototyping.
Optimization Features
Token Efficiency
Model Optimization
System Optimization
Training Optimization
Inference Optimization
Reproducibility
Data URLs
Risks & Boundaries
Limitations
Models still exhibit bias, toxicity and hallucinations like other LLMs (Section 7).
Quadratic attention cost makes long-sequence training and inference expensive.
When Not To Use
When you need the highest possible single-shot assistant quality (GPT-4 still scores higher on MT-Bench).
For extreme context lengths beyond 8K tokens without further engineering.
Failure Modes
Hallucinations on factual queries despite long context.
Performance degradation if training recipe deviates from RMS-Norm and sequential attention choices.