Overview
Production Readiness
0.6
Novelty Score
0.6
Cost Impact Score
0.7
Citation Count
2
Why It Matters For Business
A single evaluation protocol reduces integration cost, reveals whether you should invest in a better LLM or in agent engineering, and helps pick cost-performance tradeoffs for production.
Summary TLDR
The paper introduces the Unified Protocol and Exgentic, a mediation layer and evaluation harness that let any general agent run on many existing agent benchmarks without per-benchmark rewrites. The authors run 90 agent–model configurations (5 agent styles × 3 LLMs × 6 benchmarks) and publish an Open General Agent Leaderboard. Key findings: model choice (e.g., Claude Opus 4.5) dominates agent design for success, tool shortlisting fixes tool-limit failures for some models, and there is a clear cost vs performance tradeoff (high-performing setups cost much more). Code/protocol and a public leaderboard are released at www.exgentic.ai.
Problem Statement
Current agent benchmarks tie tasks to domain-specific communication protocols and hidden assumptions, so they cannot fairly evaluate agents designed to be general. This prevents apples-to-apples comparison of general-purpose agents and slows progress toward agents that work across many environments.
Main Contribution
Unified Protocol: a canonical task/context/action format that mediates between agents and benchmarks without changing either.
Exgentic: an orchestration framework and adaptor library to run any supported agent on any supported benchmark reproducibly and at scale.
Open General Agent Leaderboard: the first public benchmark suite comparing five agent implementations across six diverse environments (90 configurations) and accompanying analysis and code.
Key Findings
Model choice explains far more performance variance than agent architecture.
Claude Opus 4.5 outperforms Gemini 3 and GPT 5.2 on the evaluated benchmarks.
Top public configuration achieved 0.73 mean success but at a high per-task cost.
Tool shortlisting substantially helps tool-limited models.
Failed runs typically use more interactions, increasing cost.
General agents can match or exceed specialized baselines on many benchmarks.
Results
Top configuration mean success
Model mean success (Claude Opus 4.5)
Model mean success (GPT 5.2)
Model vs agent variance explained
Cost per task (top config)
Cost-efficiency best example
Who Should Care
What To Try In 7 Days
Run Exgentic on one representative benchmark with your agent and two backbone models to measure model vs agent impact.
Add tool shortlisting if your model hits tool-count limits; retest for reliability and cost.
Monitor average steps and failure-run length to reduce wasted cost via early-stopping or schema guards.
Agent Features
Memory
- explicit memory present in Claude Code
- other agents rely on LLM context or implicit memory
Planning
- explicit planning present in Claude Code
- implicit/reactive planning in ReAct-style agents
Tool Use
- tool-calling APIs
- MCP tool calls
- Python-execution (code-gen) wrappers
Frameworks
- MCP
- OpenAI SDK
- LiteLLM
- Exgentic adaptors
Is Agentic
true
Architectures
- ReAct
- ReAct Short
- Smolagent (code-gen)
- OpenAI Solo (MCP)
- Claude Code
Optimization Features
Infra Optimization
- sandboxed execution per session to avoid cross-talk
System Optimization
- parallelism and caching in Exgentic
- isolated reproducible runs
Inference Optimization
- tool shortlisting to reduce tool-search overhead
- select lower-cost LLMs for cost-sensitive deployments
Reproducibility
Code Urls
Data Urls
- Benchmarks referenced (SWE-Bench Verified, BrowseComp+, AppWorld, τ2-Bench) via cited papers
Code Available
Data Available
Open Source Status
- partial
Risks & Boundaries
Limitations
- Exgentic currently supports text-based interactions only, not visual or web UIs.
- Experiments cover a subset of agents and closed-source models; results may not generalize to all LLMs.
- Evaluation is costly; sample sizes per benchmark are modest (100 instances), widening per-benchmark uncertainty.
When Not To Use
- For tasks requiring GUI or visual interactions until Exgentic adds non-text protocols.
- If you require exhaustive evaluation across many niche models beyond the supported set without extending adaptors.
Failure Modes
- Adaptor mappings might omit implicit benchmark assumptions and change agent behavior.
- Large tool sets can break models with tool-count limits unless shortlisting is used.
- High-cost configurations can fail slowly and incur large monetary losses on repeated failures.
Core Entities
Models
- Claude Opus 4.5
- Gemini 3
- GPT 5.2
Metrics
- Success Rate
- Cost per Task
- Average Steps
- Cost-Efficiency (score / cost)
Datasets
- SWE-Bench Verified
- BrowseComp+
- AppWorld
- τ2-Bench (Airline, Retail, Telecom)
Benchmarks
- SWE-Bench Verified
- BrowseComp+
- AppWorld
- τ2-Bench