Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGPrompt OptimizationAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection

A meta-agent that auto-generates persona-driven adversarial tests and judges agents to find deeper failures fast

LLM EvaluationAutomatic EvaluationLLM-as-a-JudgeStress TestingSystem
August 24, 20257 min

Overview

Decision SnapshotNeeds Validation

The system is a practical developer tool: it speeds up failure discovery and supports regression testing, but it needs rubric tuning and human review for tone and stylistic judgment.

Citations0

Evidence Strength0.70

Confidence0.80

Risk Signals11

Trust Signals

Findings with numeric evidence: 3/3

Findings with evidence refs: 3/3

Results with explicit delta: 3/4

Reproducibility

Status: Partial assets available

Open source: Partial

At A Glance

Cost impact: 80%

Production readiness: 60%

Novelty: 60%

Authors

Sameer Komoravolu, Khalil Mrini

Links

Abstract / PDF / Code

Why It Matters For Business

ATA finds deeper, regression-ready failures fast and cheaply, letting teams catch hard bugs before costly human reviews and speed up release cycles.

Who Should Care

Product ManagerML EngineerEngineering LeadCTO

Summary TLDR

This paper introduces ATA, a meta-agent that automatically generates, runs, and judges adversarial conversational tests against other agents. ATA inspects agent code, asks the designer questions, mines papers/datasets, then synthesizes persona-based dialogues whose difficulty adapts using an LLM judge. On a travel planner and a Wikipedia writer, ATA found broader and deeper failures than a ten-person human team and finished in 20–30 minutes versus days. An ablation without code analysis and web search raises score variance and miscalibration, highlighting the value of evidence-grounded test generation.

Problem Statement

Agent evaluations rely on static benchmarks or small human studies that are slow, brittle, and low-coverage. Developers need an automated, repeatable way to find diverse, high-impact failures in agentic systems without heavy domain annotation.

Main Contribution

Design and open-source implementation of ATA, a meta-agent that auto-generates adversarial persona dialogues and evaluates agents end-to-end.

A weakness-planning algorithm that builds a difficulty posterior and adapts test difficulty online using judge feedback.

Key Findings

ATA finds a broader and deeper set of failures than a ten-annotator human round while matching severity on overlapping issues.

NumbersFull runs: ATA completed in 20–30 minutes vs human round taking ten days

Practical UseUse ATA to get fast, depth-focused failure discovery for CI and regression tests before costly human review.

Evidence Ref§4.4 (Cost/Time) and §4 (Holistic Comparison)

Removing code analysis and web search increases score variance and miscalibration.

NumbersScore variance σ²: ablated 7.15 vs full ATA 3.23

Practical UseInclude code- and literature-based evidence in automated testers to avoid noisy, unreliable scores.

Evidence Ref§5.2 (Score Distributions)

Results

MetricValueBaselineDeltaSplit / DatasetEvidenceEvidence Ref
End-to-end run timeATA: 20–30 minutes; Human round: 10 days——Travel + Wikipedia experiments§4.4 Cost/Time§4.4
Score variance (σ²) on human-overlapping weaknessesFull ATA: 3.23; Ablated ATA: 7.15—ablated higher by 3.92Ablation study (§5.2)§5.2 Score Distributions§5.2

What To Try In 7 Days

Run ATA on a development agent and compare its top 10 failures to recent bug reports.

Add ATA's test scenarios to CI as smoke tests to catch regressions per weakness thread.

Calibrate ATA rubrics to match one expert annotator, then run the ablation to see evidence impact.

Agent Features

Memory
global JSON-like shared stateper-thread history and difficulty posterior
Planning
weakness-planning (builds prioritized failure hypotheses)adaptive difficulty planning (difficulty posterior)
Tool Use
static code analysis via LLMweb/literature searchpersona-based prompt generationLLM judge (LAAJ)
Frameworks
open-source Agent-Testing-Agent repoLAAJ judging pipeline
Is Agentic

Yes

Architectures
meta-agent (agent that tests agents)threaded per-weakness execution
Collaboration
designer interrogation (user-in-the-loop refinement)evidence gathering from literature and datasets

Reproducibility

Code AvailableYes
Data AvailableNo
Open Source StatusPartial
LicenseUnknown

Code URLs

https://github.com/KhalilMrini/Agent-Testing-Agent

Risks & Boundaries

Limitations

Misses interpersonal and tone-related failures that humans detect well (§4.4).

Evaluation tested on two domains (travel planner, Wikipedia writer), so cross-domain generality is unproven.

When Not To Use

When human tone, emotional nuance, or interpersonal behavior is the primary concern.

When you cannot provide code access or context for evidence grounding.

Failure Modes

Judge miscalibration or bias producing over/under-scoring of real issues.

High score variance when evidence gathering is disabled (ablated pipeline).

Core Entities

Models

GPT-4.1-minio3 deep-reasoning (OpenAI o3)

Metrics

LAAJ overall score (1–10)Rubric criterion scores (1–5 per criterion)Score variance σ² (comparison: 3.23 vs 7.15)

Datasets

TRAVELPLANNER (referenced)

Benchmarks

TRAVELPLANNER

You May Also Want to Read

Make LLM judges reliable and cheaper: EIF and tuned PPI give tighter, valid evaluation with few labels

0.70
0.60
0.70
0

You can get valid, narrower confidence intervals for model evaluation while labeling far fewer examples by using EIF or tuned PPI, which saves annotation cost and gives more precise decisions about model comparisons.

Key finding

EIF-based estimators (and optimally tuned PPI++) produce substantially narrower confidence intervals than standard PPI in simulations.

Numbers: CI width 35–55% smaller vs PPI (Section 5.3; Fig.4)

Model rankings from LLM judges become less biased and get valid confidence intervals when you down-weight unreliable judges

0.70
0.60
0.60
0

If you use LLMs to evaluate other models or products, weighting judges by inferred reliability reduces bias, yields calibrated uncertainty, and saves annotation cost when judges or data are limited.

Key finding

Judge-aware aggregation produces narrower confidence intervals than unweighted aggregation on real data.

Numbers: Average CI width 0.185 vs 0.211 (13.5% narrower)

Large Llama models beat smaller ones on iCliniq medical QA; LLM judges help measure clinical quality

0.30
0.30
0.60
0

This paper shows large open models can give much better zero-shot medical answers, but parameter-efficient architectures can approach top quality. For product teams, that means you can trade compute cost for accuracy and still get usable clinical answers if you pick the right model and add quality checks.

Key finding

Llama-3.3-70B achieved the highest automatic and judge scores.

Numbers: BLEU-1 0.2207; ROUGE-1 0.2761; Judge 4.40/5

A judge-free, n-gram benchmark that approximates GPT-4 judging for Japanese Q&A

0.70
0.55
0.80
0

You can cheaply and deterministically evaluate short Japanese Q&A outputs without repeated human or expensive LLM judging, cutting evaluation cost and speeding model iteration.

Key finding

Benchmark scores correlate very highly with GPT-4o judge scores.

Numbers: Pearson r = 0.9896 (Section 5.2; Fig.4)

MCTS-Judge: Use Monte Carlo Tree Search at test time to double LLM judge accuracy on code tasks

0.70
0.70
0.70
0

MCTS-Judge can greatly improve automated code-evaluation accuracy using smaller models and fewer tokens, cutting evaluation cost and reducing dependence on handcrafted tests while providing richer explanations for CI/QA workflows.

Key finding

MCTS-Judge raised DeepSeek-Coder-V2-16B-Instruct accuracy on APPS from 41.0% to 80.0%.

Numbers: APPS: 41.0% → 80.0%