Overview
Production Readiness
0.6
Novelty Score
0.6
Cost Impact Score
0.5
Citation Count
3
Why It Matters For Business
AIDE can automate the repetitive trial-and-error of ML engineering, producing competitive models faster and often cheaper than manual work or traditional AutoML when you have LLM API access.
Summary TLDR
AIDE is an agent that uses large language models to search directly in the space of code (Python scripts) with a tree-search workflow. It keeps a tree of code variants and applies three LLM-driven operations—draft, debug, improve—while summarizing past attempts to keep prompts small. Evaluations show AIDE outperforms traditional AutoML and other agent baselines on curated Kaggle tasks and improves medal rates on MLE-Bench. It is publicly released on GitHub but requires LLM APIs and compute. Key risks: training-data contamination, local optima from greedy search, and limits on large codebases.
Problem Statement
Building high-performance ML models is dominated by trial-and-error coding and tuning. Classic AutoML searches fixed hyperparameter spaces and needs expert design. With modern LLMs that can generate and fix code, we can instead search the space of scripts directly, but must manage prompt size and iterative history effectively.
Main Contribution
Introduce AIDE: an LLM-powered agent that frames ML engineering as code-space optimization using a solution tree.
Define practical operators: draft, debug, improve; plus a summarization operator to keep LLM prompts concise.
Show empirical wins on multiple benchmarks (Weco-Kaggle, MLE-Bench, RE-Bench) and open-source the implementation.
Key Findings
On 16 tabular Kaggle tasks (Weco-Kaggle Lite), AIDE outperformed about half of human competitors on average.
On the larger Weco-Kaggle set AIDE remains competitive but variable across tasks.
Adding AIDE to an LLM baseline substantially improved submission and medal metrics on MLE-Bench.
AIDE can iterate faster than humans and beat expert teams on some R&D tasks within limited time.
Per-task LLM cost is modest in these runs.
Results
Exceeds % of humans (Weco-Kaggle Lite)
Above Median (%) (Weco-Kaggle Lite)
Any Medal (%) (MLE-Bench, AIDE + o1-preview)
Valid submissions (o1-preview without vs with AIDE)
Gold medal rate (o1-preview without vs with AIDE)
Who Should Care
What To Try In 7 Days
Run AIDE on a small internal tabular task and compare leaderboard/validation to your current pipeline.
Replace a time-consuming debugging loop with AIDE's draft→debug→improve flow for a single project.
Measure per-task LLM cost and net time saved versus manual engineering for 3 representative problems.
Agent Features
Memory
- external solution tree storing scripts and scores
- summarization operator to compress history
Planning
- hard-coded search policy for draft/debug/improve
- atomic-change proposals to measure effect
Tool Use
- LLM APIs for code generation
- code execution environment for evaluation and submission
- Kaggle leaderboard submission flow
Frameworks
- Contrast to ReACT-style agents
- integration with AutoML baselines (H2O) for comparison
Is Agentic
true
Architectures
- LLM-based agent
- tree-search over code space
Collaboration
- single-agent workflow (no multi-agent coordination)
Optimization Features
Token Efficiency
- summarization operator reduces prompt size and history included
System Optimization
- reuse and refine promising code nodes to trade compute for performance
Reproducibility
Code Urls
Code Available
Open Source Status
- partial
Risks & Boundaries
Limitations
- Potential data contamination: LLMs may have seen competition data during training.
- Performance varies by dataset; not uniformly superior across all tasks.
- Greedy/hard-coded search policy can get stuck in local optima on complex R&D tasks.
- Struggles with large, multi-file codebases or multi-step refactorings.
When Not To Use
- When working on large multi-module codebases needing deep architecture-level redesigns.
- When strict reproducibility/privacy forbids use of external LLM APIs or possible training-data leakage.
- If a task requires coordinated multi-step planning beyond single atomic edits.
Failure Modes
- Repeating local patches without discovering new strategies (local optima).
- Generating invalid submissions due to environment mismatch or hidden dependencies.
- Overfitting to internal holdout splits if evaluation protocol differs from true test set.
Core Entities
Models
- GPT-4 Turbo
- GPT-4o
- o1-preview
- Llama 3.1
- Claude 3.5
- GPT-4
Metrics
- Exceeds % of humans
- Above Median (%)
- Valid Submission (%)
- Any Medal (%)
- Gold (%)
- pass@k
Datasets
- Weco-Kaggle (full)
- Weco-Kaggle Lite (16 tasks)
- MLE-Bench (75 Kaggle competitions)
- RE-Bench (METR R&D tasks)
- various Kaggle competitions (e.g., bike-sharing-demand)
Benchmarks
- Weco-Kaggle
- MLE-Bench
- RE-Bench