Overview
Production Readiness
0.5
Novelty Score
0.6
Cost Impact Score
0.6
Citation Count
1
Why It Matters For Business
AI coding agents can cut developer time but only if they earn developer trust through verifiable outputs, provenance, and integrated review processes.
Summary TLDR
This short opinion paper argues that deploying AI "software engineers"—LLM-based agents that write, edit, and test code—depends less on raw capability and more on trust. The authors outline technical (testing, static analysis, formal proofs, guardrails) and human (explainability, provenance, review parity) mechanisms for trust, survey early agent systems, and call for unified, explainable agents that integrate coding, testing, and review into developer workflows.
Problem Statement
LLMs can generate and edit code but industry adoption of fully autonomous AI software engineers is held back by lack of developer trust. The paper asks how LLM agents can be designed to earn the same practical, reviewable trust that human contributors have.
Main Contribution
Framing trust as the central barrier to adopting AI software engineers and separating technical vs human trust.
Describing what software-engineering LLM agents are: LLM back-ends + tool interaction + autonomy.
Surveying existing agent examples (Devin, RepairAgent, AutoCodeRover, SWE-agent, Codex agent) and contrasting their trust strategies.
Proposing concrete trust-building measures: testing, lightweight static analysis, formal proofs, guardrails, specification inference, and human-AI collaboration practices.
Key Findings
Developer trust, not raw generation skill, is the main barrier to widespread adoption of AI software engineers.
Integrating standard engineering tools (tests, linters, program analysis) into LLM agents increases technical trust.
Formal verification offers stronger guarantees than testing when agents produce verifiable specs and code.
Guardrails (input sanitization and output validation) help defend against prompt injections, malicious code, and insecure inputs.
Human factors—explainability, provenance, confidence scores, and review parity—affect developer willingness to accept AI changes.
Existing systems take different trust paths: some emphasize program analysis (AutoCodeRover), others emphasize tool interfaces (Devin, SWE-agent).
Who Should Care
What To Try In 7 Days
Run an LLM agent to generate a small feature and require it produce tests; run those tests and linters automatically.
Add provenance tags and a short rationale to AI-generated pull requests before human review.
Implement basic guardrails: input sanitizers and output validators around any code-writing agent.
Agent Features
Memory
- code search / retrieval for intent inference
Planning
- autonomous nondeterministic work-plans
- tool invocation planning
Tool Use
- file navigation
- code editing
- test execution
- static analysis
- shell commands
- web browsing
Is Agentic
true
Architectures
- LLM back-end
Collaboration
- AI-human feedback loops
- review parity
- confidence and provenance reporting
Reproducibility
Open Source Status
- unknown
Risks & Boundaries
Limitations
- Opinion piece with no empirical user studies or measurements.
- No quantitative evaluation of trust interventions or agent designs.
- Recommendations may require significant engineering to integrate into real workflows.
When Not To Use
- As sole decision-maker for safety-critical code without formal verification.
- As a replacement for human review in regulated environments.
- When provenance and audit trails cannot be recorded.
Failure Modes
- Agent hallucinations leading to incorrect code.
- Prompt-injection or malicious inputs causing unsafe outputs.
- Overwhelming developers with too much auto-generated code.
- Mismatch between generated code and team coding standards or expertise.
Core Entities
Models
- LLMs (general)
- Codex
Metrics
- correctness
- security
- performance
- maintainability
- explainability
- confidence