Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGPrompt OptimizationAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection

ChartLlama: a multimodal LLM trained on GPT‑4‑synthesized chart data for chart understanding and generation

LLM EvaluationMultimodal BenchmarksBenchmark ConstructionAutomatic EvaluationSystem
November 27, 20238 min

Overview

Decision SnapshotReady For Pilot

The approach is practical and shows consistent gains on public benchmarks and novel tasks, but relies on GPT-4 synthesis, has OCR limits, and uses automatic GPT scoring which can bias results.

Citations25

Evidence Strength0.80

Confidence0.85

Risk Signals9

Trust Signals

Findings with numeric evidence: 5/5

Findings with evidence refs: 5/5

Results with explicit delta: 8/8

Reproducibility

Status: Code + data available

Open source: Partial

At A Glance

Cost impact: 50%

Production readiness: 60%

Novelty: 70%

Authors

Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, Hanwang Zhang

Links

Abstract / PDF / Code / Data

Why It Matters For Business

Companies that need automated reading, generation, or editing of charts can improve accuracy and add code-generation features by training multimodal models on synthetic, code‑paired chart datasets.

Who Should Care

ML EngineerData ScientistProduct ManagerCTO

Summary TLDR

This paper builds a pipeline that uses GPT-4 to (1) synthesize tabular data, (2) generate plotting code (Matplotlib) and images, and (3) create diverse instruction–answer pairs. The authors release a 160K instruction-tuning dataset spanning 11K chart figures and 10 chart types, and train ChartLlama (based on LLaVA-1.5). ChartLlama beats prior open models on chart Q&A, chart-to-text, and chart-extraction benchmarks and adds new tasks: detailed description, chart-to-chart, text-to-chart, and chart-editing. The dataset and code are published online.

Problem Statement

General multimodal LLMs do poorly on charts because public instruction datasets lack detailed chart data, plotting code, and diverse chart types. This hampers both chart understanding (QA, extraction, summaries) and chart generation (code and edited figures).

Main Contribution

A three-stage GPT-4 driven pipeline to synthesize tabular data, plotting code, and instruction–answer pairs for charts.

A public benchmark/dataset (≈11K chart figures, 160K instruction examples, 10 chart types, 7 task types).

Key Findings

ChartLlama improves ChartQA accuracy versus prior open models on evaluated splits.

NumbersChartQA average: ChartLlama 69.66 vs Unichart 66.24 (Table 2/5)

Practical UseTrain on chart-specific instruction data to raise QA accuracy by a few points on standard chart benchmarks.

Evidence RefTable 2, Table 5

ChartLlama raises ChartQA performance on human-authored (harder) questions.

NumbersChartQA human: ChartLlama 48.96 vs Unichart 43.92 (Table 2)

Practical UseSynthetic instruction tuning helps handling complex, reasoning-heavy chart questions.

Evidence RefTable 2

Results

MetricValueBaselineDeltaSplit / DatasetEvidenceEvidence Ref
Accuracy69.66 (ChartLlama)66.24 (Unichart)+3.42ChartQA (Table 2/5)ChartLlama average 69.66 vs Unichart 66.24Table 2, Table 5
ChartQA (human split)48.96 (ChartLlama)43.92 (Unichart)+5.04ChartQA human splitChartLlama 48.96 vs Unichart 43.92Table 2

What To Try In 7 Days

Run the public ChartLlama demo or model to test chart QA on your internal charts.

Use their GPT-4 pipeline prompts to synthesize a small corpus of company charts and paired plotting code.

Fine-tune a LLaVA-1.5 style model on a 1–10k sample of chart+code pairs and measure extraction/QA gains.

Agent Features

Tool Use
Matplotlib code generationGPT-4 for data and instruction synthesis
Frameworks
LLaVA-1.5 backboneLoRA

Optimization Features

Training Optimization
LoRATwo-stage training: projection pretrain then instruction fine-tune

Reproducibility

Code AvailableYes
Data AvailableYes
Open Source StatusPartial
LicenseUnknown

Code URLs

https://tingxueronghua.github.io/ChartLlama/

Data URLs

https://tingxueronghua.github.io/ChartLlama/

Risks & Boundaries

Limitations

Vision encoder lacks robust multilingual OCR; non-English charts perform poorly.

Heavy reliance on GPT-4 for data and evaluation may introduce synthetic biases.

When Not To Use

For charts containing non-English text until OCR is improved.

When legal/auditable provenance of training data is required.

Failure Modes

OCR misreads text in charts, causing wrong answers or hallucinations.

Generated plotting code may not execute; such samples are filtered out.

Core Entities

Models

ChartLlamaLLaVA-1.5UnichartDePlotPix2StructMatchaMiniGPT-v2InternLM-XComposer-VLQwen-VLmPLUG-Owl2Baseline*

Metrics

AccuracyBLEU-4GPT-based score (GPTScore)PrecisionF1GPT Score (chart-to-chart, text-to-chart)Success Rate (%)

Datasets

ChartLlama dataset (160K instr., 11K charts)ChartQAPlotQAChart-to-textUnichart datasetStructChart

Benchmarks

ChartQAChart-to-textChart-extractionNew tasks: detailed-description, chart-to-chart, text-to-chart, chart-editing

Context Entities

Models

GPT-4VLLaVALLaVA-1.5 (base)BLIP-2

Metrics

BLEU-4 (noted as limited)GPTScore (adapted)

Datasets

LAION-CC-SBU (subset used for projection pretrain)OKVQA, A-OKVQA, Visual Genome, RefCOCO, OCRVQA, TextCaps (used in pretraining/fine-tuning mix)

You May Also Want to Read

SimpleVQA — a 2,025-sample bilingual VQA benchmark that tests multimodal LLM factuality with atomic-fact probes

0.60
0.50
0.60
0

SimpleVQA surfaces concrete factual weaknesses in vision-capable LLMs, reducing risk if you rely on multimodal outputs for decisions; run it to find brittle visual facts before deployment.

Key finding

SimpleVQA contains 2,025 high-quality Q&A samples across 9 tasks and 9 domains.

Numbers: Dataset size = 2,025; 9 tasks; 9 domains (Table 2)

A public benchmark that tests whether multimodal LLMs can judge other model outputs across scoring, pairwise, and ranking tasks.

0.40
0.60
0.40
5

MLLMs can speed and scale human-like pairwise evaluation, but current models still fail at reliable numeric scoring and list ranking; use them to triage or pre-filter outputs, not to fully automate decisions.

Key finding

MLLMs are reliable at pairwise comparisons but not at scoring or ranking.

Numbers: Pair (no tie) GPT‑4V avg=0.773; Score Pearson GPT‑4V=0.490; Batch Levenshtein GPT‑4V=0.361

M-JudgeBench: a capability-focused multimodal judge benchmark plus Judge‑MCTS data that boosts judge model accuracy with a small synthetic-­

0.60
0.70
0.60
0

Better automated judges make preference data and alignment training more reliable; adding a modest set of structured MCTS pairs (thousands) yields large accuracy gains and reduces costly mislabeling.

Key finding

Judge models struggle to discriminate similar-length Chain‑of‑Thought pairs.

Numbers: CoT pairwise accuracy ≈50%–70% across models

CCFQA: parallel speech+text QA in 8 languages to measure cross-lingual and cross-modal factual consistency

0.60
0.70
0.50
0

If you deploy voice or multilingual assistants, factual errors rise when models face other languages or audio inputs; test with parallel speech-text data to find hidden failures.

Key finding

Dataset size and scope

Numbers: 1,800 parallel QAs × 8 languages = 14,400 samples

VALOR-EVAL: an LLM-driven open‑vocabulary benchmark that measures both hallucination and coverage across objects, attributes, and relations

0.60
0.60
0.70
0

VALOR-EVAL helps detect and quantify factual errors in vision-language outputs while measuring how much of an image models actually describe, so product teams can choose between precise vs. comprehensive models and set verification policies accordingly.

Key finding

VALOR-EVAL strongly matches human judgment on attributes and objects.

Numbers: Pearson ρ: object faithfulness 0.91, object coverage 0.89; attribute faithfulness up to 0.99