Overview
The benchmark is well engineered and reproducible with public code. Results are strong enough to guide pilots but not to claim agents are production-ready for general office automation.
Citations3
Evidence Strength0.80
Confidence0.87
Risk Signals8
Trust Signals
Findings with numeric evidence: 4/4
Findings with evidence refs: 4/4
Results with explicit delta: 0/4
Reproducibility
Status: Code + data available
Open source: Yes
At A Glance
Cost impact: 60%
Production readiness: 30%
Novelty: 65%
Why It Matters For Business
Agents can reliably finish some code-heavy tasks but currently fail most social and office workflows; companies should pilot agents on engineering tasks first and budget for API costs and human oversight.
Who Should Care
Summary TLDR
This paper introduces TheAgentCompany, an open, self-hosted benchmark that measures how well LLM agents perform realistic software-company work (175 tasks). Agents must browse, run code, and message simulated colleagues. The best model (Gemini-2.5-Pro) autonomously finished 30.3% of tasks and scored 39.3% with partial credit; many tasks involving social chat and complex web UIs largely fail. Code, environment, and evaluations are released for reproducible testing.
Problem Statement
We lack objective, reproducible benchmarks that test whether LLM agents can actually do real workplace tasks that require browsing, coding, terminal work, and communication. This prevents clear answers about which tasks can be automated and what caveats apply.
Main Contribution
A self-hosted, reproducible environment simulating a small software company and 175 realistic tasks.
Checkpoint-based scoring with deterministic evaluators and LLM-based fallback evaluation to give partial credit.
Key Findings
Top model autonomy is partial: Gemini-2.5-Pro completes about 30% of tasks.
Tasks are expensive in time and API cost for best-performing agents.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Full completion (best model) | 30.3% (Gemini-2.5-Pro) | — | — | Full 175-task set | Table 1 reports 30.3% success for Gemini-2.5-Pro | Table 1 |
| Partial completion score (best model) | 39.3% (Gemini-2.5-Pro) | — | — | Full 175-task set | Table 1 reports 39.3% partial score for Gemini-2.5-Pro | Table 1 |
What To Try In 7 Days
Run OpenHands + Gemini-2.5-Pro on a 10-task subset to measure real step and cost profiles using the public repo.
Benchmark an open-weight model (e.g., Llama-3.3-70B) on the same subset to compare cost vs success trade-offs.
Test agents on internal chat and document workflows to measure practical failure modes before automating them.
Agent Features
Memory
Planning
Tool Use
Frameworks
Is Agentic
Yes
Architectures
Collaboration
Optimization Features
Token Efficiency
Reproducibility
Code URLs
Risks & Boundaries
Limitations
Tasks skew to well-scoped, automatically-evaluable work; creative or open-ended tasks are not covered.
No human baseline comparison was collected due to cost, so human-vs-agent gaps are not quantified.
When Not To Use
To judge agent ability on creative or high-level strategy tasks not expressible with programmatic checkpoints.
To claim job-level automation beyond the specific set of curated tasks in a software-company setting.
Failure Modes
Agents get stuck on complex web UIs (popups, dynamic elements).
Agents fail at social follow-up or multi-turn coordination with colleagues.