Overview
The paper provides consistent results across three benchmarks and ablations; results look robust for QA/math tasks but rely on a teacher LLM and are evaluated on a limited set of domains.
Citations2
Evidence Strength0.70
Confidence0.85
Risk Signals9
Trust Signals
Findings with numeric evidence: 6/6
Findings with evidence refs: 6/6
Results with explicit delta: 6/6
Reproducibility
Status: Code + data available
Open source: Partial
At A Glance
Cost impact: 75%
Production readiness: 60%
Novelty: 60%
Why It Matters For Business
You can offload planning to a cheap local model and keep expensive models for final solving, cutting inference cost while keeping accuracy on reasoning tasks.
Who Should Care
Summary TLDR
The paper shows that you can distill the part of an LLM that breaks a complex question into subquestions (the decomposer) into a much smaller model while keeping most of the reasoning performance and cutting runtime cost. By contrast, distilling the solver (the part that computes or retrieves answers) hurts accuracy and generalization. The distilled decomposer generalizes across datasets and works with different solvers, enabling cost-efficient inference pipelines where a small local model plans and a large model answers.
Problem Statement
Large LLMs are costly and hard to adapt. Complex reasoning needs two skills: decomposition (split task into subquestions) and solving (use knowledge to answer). Can we compress (distill) one capability into a small model to cut inference cost while preserving accuracy? Which capability is easier to distill and more broadly useful?
Main Contribution
Show decomposition and solving can be separated into a static two-stage pipeline (decompose then solve).
Demonstrate decomposer distillation: small student models can imitate teacher-generated subquestions with low cost and strong generalization.
Key Findings
Distilling only the decomposer preserves or improves two-stage reasoning performance versus a single-stage approach on evaluated benchmarks.
A distilled decomposer matches the teacher's decomposition quality and cuts decomposition cost dramatically.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| GSM8K Exact Match (EM) | 65.13 (GPT decomposer + GPT solver, static two-stage) | 20.32 (single-stage) | +44.81 | GSM8K test | Table 1; Section 5.1 | Table 1 |
| GSM8K Exact Match (EM) | 67.78 (S_D-R decomposer + GPT solver) | 65.13 (GPT decomposer + GPT solver) | +2.65 | GSM8K test | Table 1; Section 5.2 | Table 1 |
What To Try In 7 Days
Generate decompositions from a strong LLM for one task and fine-tune a small local model to reproduce them.
Switch to a static two-stage pipeline: local decomposer → remote solver and measure token cost vs single-stage.
If you have labels, apply simple filtering (screening) of teacher examples to boost in-domain decomposer quality.
Agent Features
Planning
Frameworks
Architectures
Optimization Features
Token Efficiency
Model Optimization
Training Optimization
Inference Optimization
Reproducibility
Data URLs
Risks & Boundaries
Limitations
Evaluations limited to math and reading QA datasets; other planning tasks untested.
Approach assumes teacher can produce good decompositions.
When Not To Use
When tasks require heavy domain knowledge for solving (solver must be distilled instead).
When the teacher cannot decompose queries reliably.
Failure Modes
Distilled solver confuses subquestion answers with the main answer and degrades accuracy.
Overfitting decomposer to training domain can hurt solver performance if solver is also replaced.