Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGPrompt OptimizationAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection

AI Scientists can generate ideas but routinely fail to implement and verify them

Agentic AIWorkflow AgentsCoding BenchmarksMulti-agent CoordinationAnalysis
June 2, 20257 min

Overview

Decision SnapshotNeeds Validation

Evidence is consistent across diverse benchmarks and a simulated peer review, but sample sizes and selection bias limit certainty; treat conclusions as strong for current systems but evolving with future work.

Citations1

Evidence Strength0.70

Confidence0.85

Risk Signals11

Trust Signals

Findings with numeric evidence: 3/3

Findings with evidence refs: 3/3

Results with explicit delta: 0/4

Reproducibility

Status: Partial assets available

Open source: Partial

At A Glance

Cost impact: 50%

Production readiness: 100%

Novelty: 60%

Authors

Minjun Zhu, Qiujie Xie, Yixuan Weng, Jian Wu, Zhen Lin, Linyi Yang, Yue Zhang

Links

Abstract / PDF / Code

Why It Matters For Business

AI-generated research outputs are often non-executable or non-reproducible; businesses using "AI Scientists" must invest in execution pipelines, human verification, and testing to avoid wasted spend and flawed products.

Who Should Care

CTOProduct ManagerML EngineerFounderData ScientistEngineering Lead

Summary TLDR

This position paper argues that the main limitation of current "AI Scientist" systems is implementation: models generate novel ideas but fail to execute, verify, and iterate experiments reliably. Evidence across multiple benchmarks shows execution rates often under 50% (e.g., PaperBench execution 1.8% for Claude 3.5 Sonnet, SciReplicate best 39%). A simulated peer review of 28 AI-generated papers found experimental weaknesses in 100% of samples. The authors call for focused work on planning, execution tools, verification protocols, and evaluation frameworks.

Problem Statement

AI Scientist systems can produce novel research ideas but lack the engineering and verification skills to turn ideas into correct, reproducible experiments. Benchmarks and an LLM-based peer review of 28 AI-generated papers show persistent failures in code execution, debugging, and experimental validation, limiting scientific value.

Main Contribution

Define an AI Scientist as an end-to-end system that must both generate ideas and execute verification procedures.

Aggregate benchmark evidence showing low execution/replication rates for modern LLMs on real research tasks.

Key Findings

Top LLMs fail basic experiment execution on real-paper replication tasks.

NumbersPaperBench execution 1.8% (Claude 3.5 Sonnet)

Practical UseDo not trust current AI Scientists to autonomously run and verify research code; add human-run execution checks and test harnesses.

Evidence RefSection 3.2 / Table 1 and text

Code-generation agents often produce non-functional code in research reproduction tasks.

NumbersSciReplicate best execution accuracy 39%

Practical UseExpect frequent runtime failures; integrate automated execution tests and continuous debugging loops when using generated code.

Evidence RefSection 3.2 / SciReplicate-Bench

Results

MetricValueBaselineDeltaSplit / DatasetEvidenceEvidence Ref
PaperBench execution (replication 'Execution' rubric)1.8%——PaperBenchClaude 3.5 Sonnet scored 1.8% on 'Execution' nodeSection 3.2
Accuracy39%——SciReplicate-BenchBest agent achieved 39% execution accuracy on reproducing NLP algorithmsSection 3.2

What To Try In 7 Days

Run a reproducibility smoke test for recent AI outputs: execute generated code on a sandbox.

Add unit and integration tests to any AI-generated code pipeline.

Label AI-generated reports and require human sign-off before release or publication.

Agent Features

Memory
short-term memory limitationslong-context degradation
Planning
long-horizon planning (weak)hierarchical planning suggested
Tool Use
external API callscode execution environmentslab/autonomous tools (discussed)
Frameworks
RAGMCPA2APASA
Is Agentic

Yes

Architectures
LLM-based agent pipelinemulti-agent planner-controller setups
Collaboration
multi-agent coordinationhuman-in-the-loop workflows

Optimization Features

Token Efficiency
use RAG to reduce context burden
Infra Optimization
estimate high wall-clock cost for end-to-end scientist loops
System Optimization
structured workflows and modular agents
Training Optimization
simulate environment to speed RL sampling
Inference Optimization
batch generation to improve sample throughput (discussed)

Reproducibility

Code AvailableYes
Data AvailableNo
Open Source StatusPartial
LicenseUnknown

Code URLs

https://github.com/ResearAI/Awesome-AI-Scientist

Risks & Boundaries

Limitations

Weak long-range reasoning and context retention for multi-step experiments.

Poor strategic planning for large codebases and long-horizon research.

When Not To Use

In safety-sensitive or dual-use research without strict human oversight.

As a sole replacement for human experimental verification.

Failure Modes

Generated code compiles but fails at runtime or produces wrong results.

Inability to debug or iteratively improve experiments.

Core Entities

Models

Claude 3.5 SonnetGPT-4oo1-higho3o4-miniDeepReviewer-14BDeepseek-R1

Metrics

Accuracyreplication scorepass@1success ratepeer-review rating

Benchmarks

PaperBenchSciReplicate-BenchCORE-BenchMLE-BenchML-Dev-BenchLiveCodeBenchHumanEvalScienceAgentBench

Context Entities

Models

AlphaFold (example of successful automated tool)

You May Also Want to Read

A process-aware, auditable multi-agent evaluator that produces more stable, human-aligned scores than a single LLM judge

0.60
0.60
0.40
0

AEMA gives more stable, explainable, and human-aligned evaluations for multi-step agent workflows. That yields predictable decision thresholds, fewer manual overrides, and auditable records for compliance.

Key finding

Debate in planning converged within a few rounds for repeated runs.

Numbers: 30 runs: 13/30 converge after 1 round, 13/30 after 2, 4/30 after 3

Create, customize, and run multi-step LLM agents from plain language — no code needed

0.70
0.60
0.60
0

AutoAgent lowers the engineering barrier: product teams can prototype custom assistants, retrieval workflows, and API-backed tools from plain language, cutting specialist developer time and speeding deployment.

Key finding

Strong GAIA performance — close to top commercial agents.

Numbers: GAIA avg success: AutoAgent 55.15 vs top h2oGPTe 63.64; Level1 71.7%

MLRC-Bench: a competition-based benchmark that tests if LLM agents can propose and implement novel ML research

0.20
0.40
0.35
0

If you plan to use LLM agents for real research work, expect them to assist but not replace expert researchers; measure ideas with objective tests and track cost, runtime, and code complexity.

Key finding

Best tested agent (gemini-exp-1206 under MLAB) closed only a small fraction of human gap.

Numbers: 9.3% average Relative Improvement to Human (Table 3)

A closed-loop Sensing→Regulating→Correcting system that routes LLM execution by uncertainty to cut errors and API cost

0.70
0.70
0.70
0

DenoiseFlow raises end-to-end reliability of multi-step LLM agents while cutting average API/tokens by roughly 40–56%, so you get fewer failures and lower inference bills without retraining models.

Key finding

DenoiseFlow improves average benchmark performance versus strong baselines.

Numbers: 83.3% average accuracy vs JudgeFlow 82.0% (+1.3%)

BackdoorAgent: a stage-aware framework and benchmark showing memory backdoors persist across multi-step LLM agents

0.30
0.70
0.60
0

Agent workflows can be covertly controlled without hurting visible task accuracy, so enterprises must monitor agent state and retrieval channels to avoid stealthy compromises.

Key finding

Memory-channel backdoors are the most persistent across agent families.

Numbers: Memory ASR ≈ 77.97% (aggregated, Table 5 top row)