Overview
Production Readiness
0.7
Novelty Score
0.7
Cost Impact Score
0.7
Citation Count
1
Why It Matters For Business
Train once and ship one compact artifact that can be reconfigured at inference for different memory or latency targets, saving repeated retraining and storage costs while maintaining task quality.
Summary TLDR
This paper presents Modular Transformers: train a set of modularized Transformer layers that each mimic different numbers of original layers, then assemble them at inference to get many compressed models from one training run. The method combines multi-grained module replacing with attention and hidden-state distillation, plus deterministic assembling strategies for either size-first or speed-first goals. On T5-base and T5-large across summarization, question generation, and translation, Modular Transformers matches or improves over prior compression methods (KD, SFT, pseudo-labeling) at common compression points, while enabling flexible compression ratios without retraining.
Problem Statement
Pre-trained seq2seq models are large and costly to store and run. Existing compression methods produce a single compact model per target size, forcing repeated training when resource targets change. The practical problem: enable one compressed artifact (trained once) that can be reconfigured at inference to meet different memory and latency budgets with small performance loss.
Main Contribution
Define modularized layers that each substitute for different numbers of consecutive original layers.
Train these layers with multi-grained module replacing plus attention and hidden-state distillation.
Provide deterministic assembling strategies (size-first and speed-first) so one trained set can produce many compressed models without retraining.
Show empirical gains on T5-base and T5-large across summarization, translation, and question generation, versus common baselines.
Key Findings
A single Modular Transformers training run supports flexible compression from small to large ratios.
At a common compression point (6→6, size-first, 1.5/2.0×), Modular Transformers slightly outperforms KD on CNN/DailyMail.
Modular Transformers consistently beats KD across varied speed-ups and shows larger gains at high compression.
Results
ROUGE-1 (CNN/DailyMail)
ROUGE-1 (CNN/DailyMail)
ROUGE-1 (CNN/DailyMail)
Flexible speed-up range
Who Should Care
What To Try In 7 Days
Apply modular-layer design to your T5 or similar seq2seq model and train modular layers with teacher frozen.
Use size-first and speed-first assembling rules to produce two candidate deployments and measure latency vs quality.
Compare assembled models to your current distillation pipeline at one or two target sizes to validate gains.
Optimization Features
Model Optimization
- multi-grained module replacing
- modularized layers that replace multiple original layers
Training Optimization
- curriculum replacing (coarse-to-fine)
- attention and representation distillation during training
Inference Optimization
- deterministic assembling strategies (size-first, speed-first)
- decoder-first, top-down, fine-to-coarse replacement
Reproducibility
Data Urls
- CNN/DailyMail
- XSum
- SQuAD
- WMT-14 En-De
Data Available
Open Source Status
- unknown
Risks & Boundaries
Limitations
- Evaluations limited to T5-base and T5-large; no results for BART or very large T5 (3B/11B).
- Modularity design tested with granularities {2,3,4,6}; very coarse modules (e.g., 12) harmed performance.
- Assembling requires careful deterministics rules; random assembling performs worse.
When Not To Use
- When you need guarantees at a single maximal accuracy point and prefer standard per-size distillation.
- If your production stack cannot accommodate a short re-assembly validation step for a new target.
- On models or architectures very different from the Transformer encoder-decoder without further testing.
Failure Modes
- Large-granularity modules can cause large performance drops if inserted incorrectly.
- Wrong assembling order (random) harms quality relative to deterministic assembling.
- Dependence on teacher model quality; poor teacher harms modularized layers.
Core Entities
Models
- T5-base
- T5-large
Metrics
- ROUGE
- BLEU
- METEOR
Datasets
- CNN/DailyMail
- XSum
- SQuAD (question generation split)
- WMT-14 En-De