Overview
Two-stage fine-tuning with fused vision features is a practical, medium-cost way to get reliable multimodal rationales on sub‑1B models; results are consistent across ablations and two benchmarks but commonsense/counting limits remain.
Citations96
Evidence Strength0.80
Confidence0.85
Risk Signals8
Trust Signals
Findings with numeric evidence: 4/4
Findings with evidence refs: 4/4
Results with explicit delta: 4/5
Reproducibility
Status: Code + data available
Open source: Partial
At A Glance
Cost impact: 45%
Production readiness: 60%
Novelty: 60%
Why It Matters For Business
You can get near-state-of-the-art multimodal reasoning with lightweight models by fine-tuning in two stages and fusing image features—this reduces hallucination and lowers compute cost versus running large multimodal LLMs.
Who Should Care
Summary TLDR
The paper introduces Multimodal-CoT: a two-stage fine-tuning method that first generates a rationale (explanation) from image+text and then predicts the answer using that rationale plus the image. For T5-based models under 1B parameters, fusing ViT image features into this two-stage pipeline raises ScienceQA accuracy from ~78.6% (two-stage baseline) to 85.3% (base) and to 90.45% (large), reduces hallucinated rationales, and speeds up convergence. Generated (pseudo) rationales from large models can replace human explanations with a modest drop.
Problem Statement
Small language models (<1B params) struggle to use chain-of-thought (CoT) reasoning: generating intermediate rationales can harm final answers because rationales are often hallucinated without visual context. The paper asks whether fusing vision features and separating rationale generation from answer inference fixes this for multimodal QA.
Main Contribution
Define Multimodal-CoT: a two-stage framework that first generates rationales from image+text and then infers answers using those rationales plus the image.
Show that fusing patch-level vision features (ViT) into T5-based models reduces hallucinated rationales and improves accuracy on ScienceQA and A-OKVQA.
Key Findings
Predicting a rationale before the answer hurts small-model accuracy in one-stage text-only training.
Adding vision features in a two-stage setup improves rationale quality and answer accuracy.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Accuracy | 90.45% | Prior published best 86.54% | +3.91pp | ScienceQA test | Table 4 shows Multimodal-CoT Large achieves 90.45% vs prior 86.54% | Table 4 |
| Accuracy | 85.31% | Two-stage baseline 78.57% | +6.74pp | ScienceQA test | Table 3 and Table 4 show base model with vision fusion at 85.31% | Table 3; Table 4 |
What To Try In 7 Days
Fine-tune a T5-base model in two stages: (1) generate rationales from image+text, (2) infer answers using rationale+image.
Fuse patch-level ViT features into the text encoder before decoding, not just image captions.
If you lack human rationales, generate pseudo-rationales with a large model (InstructBLIP/ChatGPT) and fine-tune Multimodal-CoT on them.
Optimization Features
System Optimization
Training Optimization
Inference Optimization
Reproducibility
Data URLs
Risks & Boundaries
Limitations
High failure rate on commonsense-heavy questions (map reading, counting).
Performance still far below top multimodal models with massive parameters on some out-of-domain tasks.
When Not To Use
Tasks that demand strong commonsense or symbolic counting without extra knowledge sources.
Applications that require state-of-the-art general multimodal understanding from very large models (e.g., GPT-4V/Gemini).
Failure Modes
Hallucinated rationales that mislead final answers when vision features are absent.
Correct or empty rationales that nevertheless do not change a wrong final answer.