Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGPrompt OptimizationAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection

CoSQA+: a large multi-choice code-search dataset built by test-driven agent annotations (412k pairs, agent accuracy 93.9%)

LLM EvaluationBenchmark ConstructionWorkflow AgentsBenchmark
June 17, 20247 min

Overview

Decision SnapshotNeeds Validation

The pipeline is practical: agents reached 93.9% accuracy and low annotation cost ($0.00062/sample), but query ambiguity and execution constraints limit perfect automation.

Citations2

Evidence Strength0.80

Confidence0.85

Risk Signals11

Trust Signals

Findings with numeric evidence: 5/5

Findings with evidence refs: 5/5

Results with explicit delta: 4/5

Reproducibility

Status: Code + data available

Open source: Yes

At A Glance

Cost impact: 75%

Production readiness: 70%

Novelty: 65%

Authors

Jing Gong, Yanghui Wu, Linxi Liang, Yanlin Wang, Jiachi Chen, Mingwei Liu, Zibin Zheng

Links

Abstract / PDF / Code / Data

Why It Matters For Business

CoSQA+ reduces expensive human labeling by using test-driven agents to create a large, functionally verified multi-choice code-search dataset that improves retrieval performance and cuts annotation cost to roughly $0.00062 per sample.

Who Should Care

CTOML EngineerEngineering LeadProduct ManagerData Scientist

Summary TLDR

This paper builds CoSQA+, a multi-choice code-search dataset (412,080 query–code pairs) labeled by an automated test-driven agent pipeline. The pipeline generates tests, runs them in Docker, fixes dependency/runtime issues, and uses a final arbiter to decide matches. The agents achieve 93.9% accuracy versus human experts on a 1,000-sample ground truth, and generated tests execute at an 83.67% rate. Models fine-tuned on CoSQA+ (CodeBERT, UniXcoder, CodeT5+) show consistent MAP@10 and MRR gains on a CSN Python test set compared to CoSQA.

Problem Statement

Existing code-search benchmarks treat search as one-to-one and rely on humans or LLM semantics, not functional verification. Real developers often need multiple correct snippets per query (survey: 63.2% of queries). This mismatch harms evaluation and training quality and limits scalability and accuracy.

Main Contribution

CoSQA+ dataset: 412,080 agent-labeled query–code pairs and a 1,000-sample human-verified subset.

A fully automated test-driven annotation pipeline (screener, test generator, executor, bug fixer, arbiter).

Key Findings

CoSQA+ provides 412,080 labeled query–code pairs.

Numbers412,080 pairs; 132,952 unique codes

Practical UseYou can train retrieval models at scale on multi-choice data instead of single-choice pairs.

Evidence RefTable II

Test-driven agents label with high agreement to human experts.

NumbersAgent accuracy 93.9% ±0.020 vs expert ground truth

Practical UseAutomated test-based annotation can replace much manual labeling and scale cheaply.

Evidence RefTable IV

Results

MetricValueBaselineDeltaSplit / DatasetEvidenceEvidence Ref
Accuracy0.939human without tests 0.891+0.0481,000-sample ground truthAgent-based annotator accuracy vs expert majority voteTable IV
Executable test program rate83.67%——CoSQA+ pipeline outputsProportion of generated tests that run successfullyTable II

What To Try In 7 Days

Download CoSQA+ verified subset and run your model's evaluation to compare MAP@10 vs your current benchmark.

Fine-tune a small retrieval model (e.g., CodeBERT or UniXcoder) on CoSQA+ training split and measure MAP@10 and MRR on a held-out CSN Python test.

Run the open-source test-driven agent on a 100–500 sample subset of your codebase to estimate executable-test rate and annotation cost.

Agent Features

Memory
short-term execution traces and logs used per sample
Planning
multi-stage triage (screener → test generation → execution → bug fixing → arbiter)iterative dependency installation and test rerun
Tool Use
Docker for safe executionpackage managers for dependency fixesLLM calls to synthesize repair/install commands
Frameworks
DeepSeek-V3custom test generation and arbiter prompts
Is Agentic

Yes

Architectures
LLM-driven modular agent pipelineembedder ensemble for candidate selection
Collaboration
single automated pipeline with modular stages (not multi-agent negotiation)

Reproducibility

Code AvailableYes
Data AvailableYes
Open Source StatusYes
LicenseUnknown

Code URLs

https://github.com/DeepSoftwareAnalytics/CoSQA_Plus

Data URLs

https://github.com/DeepSoftwareAnalytics/CoSQA_Plus

Risks & Boundaries

Limitations

Query ambiguity: vague queries lead to multiple valid interpretations and mislabels.

Test misjudgment risk: correct code outside 20-candidate set may be missed.

When Not To Use

Datasets with closed-source or non-executable code (no runtime available).

Queries that require design rationale or non-functional aspects (style, performance trade-offs).

Failure Modes

Ambiguous natural language leads to wrong test generation.

Checker and arbiter disagreements when tests pass but intent differs.

Core Entities

Models

CodeBERTUniXcoderCodeT5+jina-embeddings-v3multilingual-e5-largeall-MiniLM-L12-v2all-mpnet-base-v2DeepSeek-V3

Metrics

MAP@10MRRNDCG@10Recall

Datasets

CoSQACoSQA+_allCoSQA+_verifiedCodeSearchNetCSN99CSN Python

Benchmarks

CoSQA+CSN Python

You May Also Want to Read

Large Llama models beat smaller ones on iCliniq medical QA; LLM judges help measure clinical quality

0.30
0.30
0.60
0

This paper shows large open models can give much better zero-shot medical answers, but parameter-efficient architectures can approach top quality. For product teams, that means you can trade compute cost for accuracy and still get usable clinical answers if you pick the right model and add quality checks.

Key finding

Llama-3.3-70B achieved the highest automatic and judge scores.

Numbers: BLEU-1 0.2207; ROUGE-1 0.2761; Judge 4.40/5

PanCanBench: 282 real patient questions + 3,130 expert rubrics to test LLM clinical completeness and factuality

0.60
0.50
0.60
0

If you plan to deploy LLMs for patient-facing oncology guidance, you must test both completeness and factual accuracy on real patient questions; rubric scores alone can be misleading and AI-only rubrics overestimate quality.

Key finding

Models' rubric-based completeness scores vary widely.

Numbers: Top score 82.3% (o3); range 46.5%–82.3%

A judge-free, n-gram benchmark that approximates GPT-4 judging for Japanese Q&A

0.70
0.55
0.80
0

You can cheaply and deterministically evaluate short Japanese Q&A outputs without repeated human or expensive LLM judging, cutting evaluation cost and speeding model iteration.

Key finding

Benchmark scores correlate very highly with GPT-4o judge scores.

Numbers: Pearson r = 0.9896 (Section 5.2; Fig.4)

Professional multilingual TruthfulQA shows truth gaps across languages but smaller than expected

0.60
0.50
0.60
0

Multilingual truth checks are necessary: models can be less accurate in low-resource languages and simple automatic scores (MC2) can mislead decisions; judge-style evaluation and MT-enabled datasets reduce cost and improve cross-lingual assessment.

Key finding

LLM-as-a-Judge correlates better with humans than MC2.

Numbers: Cohen Kappa: Judge (Gemma-2-9b-inst) vs human up to 0.75; MC2 lower

MEMERAG: a native multilingual benchmark to evaluate RAG outputs and LLM-based evaluators

0.70
0.60
0.60
0

If you deploy RAG across multiple languages, MEMERAG gives a native-language testbed to validate whether automatic evaluators match human judgements; use it before trusting LLM judges for quality gating.

Key finding

High coarse-grained inter-annotator agreement for faithfulness and relevance.

Numbers: Faithfulness Gwet's AC1 0.84–0.93; Relevance Gwet's AC1 0.95–1.00 (Table 2)