ControlBench: evaluate GPT-4, Claude 3 Opus, Gemini on 147 undergraduate control problems

Overview

Why It Matters For Business

Text LLMs can help generate control designs and explanations quickly, but they commonly make calculation and plot-reading errors, so use them for drafts and human-in-the-loop workflows, not final safety-critical designs.

Summary TLDR

The authors introduce ControlBench, a 147-question dataset of undergraduate control problems (26 with plots), and test GPT-4, Claude 3 Opus, and Gemini 1.0 Ultra. Claude 3 Opus performs best (≈58.5% accuracy, rising with a self-check prompt), GPT-4 is middling, and Gemini trails. All models struggle with reading Bode/Nyquist/root-locus plots and make arithmetic/symbolic mistakes. Simple self-check prompts and tool-based arithmetic can improve results, but these LLMs are not yet reliable for unsupervised, high-stakes control work.

Problem Statement

Can state-of-the-art LLMs solve typical undergraduate control-engineering problems? The paper builds ControlBench (147 problems, many with plots), then measures model accuracy, failure modes, and how much self-checking helps.

Main Contribution

Key Findings

Results

Who Should Care

What To Try In 7 Days

Reproducibility

Risks & Boundaries

Core Entities

Related Papers

• Large Llama models beat smaller ones on iCliniq medical QA; LLM judges help measure clinical quality
• PanCanBench: 282 real patient questions + 3,130 expert rubrics to test LLM clinical completeness and factuality
• A judge-free, n-gram benchmark that approximates GPT-4 judging for Japanese Q&A
• Professional multilingual TruthfulQA shows truth gaps across languages but smaller than expected
• MEMERAG: a native multilingual benchmark to evaluate RAG outputs and LLM-based evaluators