Overview
The approach is practical and low-cost; evidence comes from multiple datasets and ablations, but results depend on LLM quality and benchmark sanitization.
Citations13
Evidence Strength0.80
Confidence0.90
Risk Signals8
Trust Signals
Findings with numeric evidence: 4/4
Findings with evidence refs: 4/4
Results with explicit delta: 0/6
Reproducibility
Status: Code + data available
Open source: Yes
At A Glance
Cost impact: 80%
Production readiness: 70%
Novelty: 60%
Why It Matters For Business
A focused, non-agentic pipeline cuts cost and engineering overhead while matching or exceeding many open-source agentic systems on repo-level bug fixes.
Who Should Care
Summary TLDR
The paper shows a lightweight, non-agentic pipeline (AGENTLESS) for fixing real GitHub issues: hierarchical localization (file → function → edit), LLM-based patch sampling in a small diff format, and LLM-generated reproduction tests plus regression testing for selection. On SWE-bench Lite (300 problems) AGENTLESS fixes 96 issues (32.00%) at an average cost of $0.70, outperforming prior open-source agentic tools while being simpler and cheaper. The authors also hand-audit SWE-bench Lite, remove problematic cases, and publish a filtered set (SWE-bench LiteS).
Problem Statement
Current LLM agent frameworks are complex, costly, and fragile. The paper asks: can a simple, non-agentic pipeline (no autonomous tool use or multi-turn planning) match or beat agent-based approaches on real repo-level coding tasks?
Main Contribution
AGENTLESS: a three-phase agentless pipeline (hierarchical localization, patch sampling with simple diff edits, and validation via reproduction + regression tests).
Empirical evaluation on SWE-bench Lite showing 32.00% resolved (96/300) at $0.70 average cost, competitive with or better than open-source agents.
Key Findings
AGENTLESS resolves 96 of 300 SWE-bench Lite problems
Average inference cost per issue is low
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| %Resolved | 32.00% (96/300) | — | — | SWE-bench Lite | Table 1; Sec. 5.1 | Table 1 |
| Avg. $ Cost | $0.70 | — | — | SWE-bench Lite | Table 1; Sec. 4 | Table 1 |
What To Try In 7 Days
Run AGENTLESS-style pipeline on a small set of repo issues: localize → sample diff patches → validate with regression + generated tests.
Add a lightweight embedding retrieval step (chunk embeddings via OpenAI) and a file-skeleton prompt to reduce LLM context size.
Audit your in-house bug reports for exact-patch leaks or missing info; filter them before model evaluation.
Agent Features
Memory
Tool Use
Frameworks
Architectures
Optimization Features
Token Efficiency
Reproducibility
Risks & Boundaries
Limitations
Performance drops on problems with no location clues; agentic tools with search tools still do better there.
Generated reproduction tests are imperfect: many reproduce the bug but fewer can validate fixes.
When Not To Use
When you need agents to run complex toolchains or perform multi-step environment interactions.
When issue descriptions lack any location hints and you need aggressive repository-wide search tools.
Failure Modes
LLM is distracted by long file contents if skeleton compression is not used.
Incorrect reproduction tests can bias patch selection if regression tests are weak.