Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGPrompt OptimizationAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection

DeepFund: a live, leakage-free benchmark that runs LLM fund managers on real market data

LLM EvaluationBenchmark LeakageBenchmark ConstructionBenchmark RobustnessBenchmark
May 16, 20258 min

Overview

Decision SnapshotReady For Pilot

The paper provides a working live platform, public code, and multi-model tests, but the evaluation is short, US-only, and omits real execution frictions like fees.

Citations0

Evidence Strength0.80

Confidence0.82

Risk Signals9

Trust Signals

Findings with numeric evidence: 5/5

Findings with evidence refs: 5/5

Results with explicit delta: 0/5

Reproducibility

Status: Partial assets available

Open source: Partial

License: MIT

At A Glance

Cost impact: 60%

Production readiness: 50%

Novelty: 70%

Authors

Changlun Li, Yao Shi, Chen Wang, Qiqi Duan, Runke Ruan, Weijie Huang, Haonan Long, Lijun Huang, Nan Tang, Yuyu Luo

Links

Abstract / PDF / Code

Why It Matters For Business

Backtests can hide LLM memorization and produce over-optimistic trading results; running live, contamination-free evaluations like DeepFund reveals real performance and risk before any capital allocation.

Who Should Care

Product ManagerML EngineerCTOFounderData Scientist

Summary TLDR

DeepFund is an open, multi-agent framework that evaluates large language models (LLMs) by running them live on real market data to avoid 'time-travel' leakage from pretraining. The system connects LLM agents (planner, analysts, portfolio manager) to live feeds (e.g., Yahoo Finance, Alpha Vantage), records decisions, and reports standard financial metrics. In a 24-trading-day test on five tickers with nine LLMs, most models lost money; only Grok 3 produced a small positive return (CR = +1.1%). The code and prompts are public so teams can reproduce live evaluations and stress-test strategies without backtesting contamination.

Problem Statement

Existing financial LLM benchmarks rely on historical backtests that overlap with an LLM's pretraining data. That overlap lets models implicitly 'time travel' by regurgitating known future events, inflating performance. We need contamination-free, real-time evaluation to measure true trading ability.

Main Contribution

DeepFund: a live benchmarking platform that connects LLM-driven multi-agent fund workflows to real-time market data to avoid information leakage.

A multi-agent design (Financial Planner, Analyst Team, Portfolio Manager) that standardizes signals and decisions and records full provenance.

Key Findings

Most evaluated LLMs lost money in live trading.

NumbersCRs ranged from +1.1% to -8.1%; majority CR < 0 (Table 3)

Practical UseDo not trust backtests alone—run live or contamination-checked tests before deploying LLM-based trading.

Evidence RefTable 3, Section 4.1

Only Grok 3 achieved a small positive cumulative return.

NumbersGrok 3 CR = +1.1% (Table 3)

Practical UseSome LLMs may be marginally profitable in short windows; validate performance over longer, varied markets before scaling capital.

Evidence RefTable 3, Section 4.1

Results

MetricValueBaselineDeltaSplit / DatasetEvidenceEvidence Ref
Grok 3 cumulative return (CR)1.1%——24 trading days on AAPL/AXP/BAC/KO/CVX (Mar 17–Apr 17, 2025)Grok 3 CR = 1.1% (Table 3)Table 3
Range of model CRs+1.1% to -8.1%S&P 500 CR = -6.91%—Same trading period and tickersTable 3 lists CR per model; majority negative (Table 3)Table 3

What To Try In 7 Days

Run DeepFund or a live pilot on a small portfolio to detect leakage and test signal pipelines.

Compare model cost vs. return using short live windows (check OpenAI family cost-efficiency findings).

Add or tighten cash-reserve and diversification rules and re-run live to measure risk control impact.

Agent Features

Memory
Dual-memory: short-term FundState and long-term trading history
Planning
Financial Planner role (task allocation and budgeting)Deterministic and dynamic planner modes
Tool Use
LLM backends via provider APIsLangChain for orchestration
Frameworks
LangChain
Is Agentic

Yes

Architectures
orchestrator-worker multi-agent
Collaboration
Analyst-team concurrent analysis and portfolio manager aggregation

Reproducibility

Code AvailableYes
Data AvailableNo
Open Source StatusPartial
LicenseMIT

Code URLs

https://github.com/HKUSTDial/DeepFund

Risks & Boundaries

Limitations

Evaluations cover only US stocks and five tickers; not a full market stress test (Section 7).

Simplified trading model: no transaction fees, slippage, or exchange constraints accounted for (Section 7).

When Not To Use

When you require realistic execution costs (fees, slippage) not modeled here.

When evaluating non-US markets or large-cap institutional execution without further extensions.

Failure Modes

LLM pretraining contamination still possible if live data sources or prompts leak historical events into decisions (motivates live-only tests).

High portfolio concentration and low cash reserve can cause large drawdowns during shocks (observed in DeepSeek).

Core Entities

Models

Grok 3 mini BetaGemini 2.5 FlashClaude 3.7 SonnetLlama 4 ScoutDeepSeek-V3GPT-4.1Qwen2.5-MaxGLM-4-AirDoubao-1.5-pro

Metrics

Cumulative ReturnCumulative Return at Buy & HoldSharpe RatioMaximum DrawdownWin RateBetaAlpha

Datasets

Live market data (Yahoo Finance, Alpha Vantage)Berkshire Hathaway top-5 tickers portfolio (AAPL, AXP, BAC, KO, CVX)

Benchmarks

DeepFund live fund benchmark

Context Entities

Benchmarks

TAT-QAFinanceBenchFinBenInvestorBenchFinRL-MetaForecastBenchLiveBench

You May Also Want to Read

A weekly-updated, contamination-free medical benchmark plus automated rubrics that align better with physicians than LLM-as-a-judge

0.35
0.70
0.50
0

Static medical benchmarks can give falsely high performance because models may have seen test cases during training. LiveMedBench detects contamination and tests real-time generalization. If you deploy LLMs for clinical tasks, evaluate on time-separated, evidence-checked benchmarks and prefer retrieval-enabled systems.

Key finding

LiveMedBench snapshot size and scope

Numbers: 2,756 cases; 16,702 rubric criteria; 38 specialties

When synthetic training data and LLM evaluators are related, evaluators unfairly favor the student models

0.60
0.60
0.50
2

Automatic leaderboards and internal evaluations can overstate model quality when the same or related LLMs generate training data and judge models; this risks bad product decisions and misallocated resources.

Key finding

Preference leakage creates measurable bias in LLM judges.

Numbers: PLS averages up to 23.6% (Mistral with GPT-4o & Gemini, Table 1)

Auto-update benchmarks with two LLM-driven strategies to reduce leakage and tune difficulty

0.60
0.50
0.60
0

Automated updates let teams keep benchmarks fresh without heavy human labor, reduce false confidence from leaked test data, and tune question difficulty to better compare model releases.

Key finding

Mimicked datasets produce consistent evaluation scores across multiple regenerations.

Numbers: std dev 0–3% across four mimicked runs (zero-shot scores)

Ko-H5 and an open Korean LLM leaderboard: private tests, new Korean tasks, and when benchmarks stop helping

0.60
0.50
0.50
1

Private test sets and language-tailored benchmarks give more reliable model comparisons and prevent overfitting to public datasets, so product teams can trust model selection for Korean deployments.

Key finding

Private Ko-H5 test sets have very low overlap with popular Korean training corpora.

Numbers: Overlap < 1% across tasks (Table 2)

TreeEval: benchmark-free LLM evaluation via LLM examiner and tree planning

0.60
0.70
0.60
0

TreeEval reduces evaluation cost and leak risk by generating session-unique questions and using adaptive deepening, so teams can compare models faster and with fewer paid API calls while lowering the chance of benchmark contamination.

Key finding

TreeEval rankings strongly match AlpacaEval2.0 rankings.

Numbers: Spearman ρ=0.83; Kendall τ=0.73 (Table 1)