Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection
Alignment and SafetyPrompt InjectionBenchmark ConstructionTool Selection

INJECAGENT: 1,054 realistic tests that measure how tool-enabled LLM agents can be hijacked by malicious content

March 5, 20247 min

Overview

Production Readiness

0.6

Novelty Score

0.6

Cost Impact Score

0.7

Citation Count

5

Authors

Qiusi Zhan, Zhixiang Liang, Zifan Ying, Daniel Kang

Links

Abstract / PDF

Why It Matters For Business

Tool-enabled LLM agents can be hijacked by content they retrieve, causing unauthorized transactions or data leaks; firms must test agents with realistic IPI cases before deployment.

Summary TLDR

This paper introduces INJECAGENT, a public benchmark of 1,054 test cases (17 user tools × 62 attacker cases) that measures how tool-enabled LLM agents respond when external content contains malicious instructions (indirect prompt injection, IPI). Evaluating 30 agents, the authors find prompted agents are often vulnerable (e.g., ReAct-prompted GPT‑4 ASR-valid 24% base, 47% with a 'hacking' prompt) while fine-tuning for tool calls substantially lowers vulnerability (fine-tuned GPT‑4 ASR-valid ≈ 6.6%–7.1%). Key practical lessons: reduce free-text placeholders, avoid blindly concatenating untrusted content, and prefer tool-call fine-tuning or stricter output checks.

Problem Statement

LLM agents are being given tools and access to external content. That content can contain hidden instructions from attackers (indirect prompt injection, IPI) that redirect agents to perform harmful actions or leak private data. We lack a systematic, realistic benchmark to measure these risks for tool‑integrated agents and to compare defenses.

Main Contribution

Formalize indirect prompt injection (IPI) against tool-integrated LLM agents and define measurable attack success.

Release INJECAGENT: 1,054 realistic test cases combining 17 user-facing tools and 62 attacker instructions, with base and enhanced (hacking-prompt) settings.

Evaluate 30 LLM agents and analyze attack patterns; show prompted agents are frequently vulnerable and fine-tuned tool-call agents are more robust.

Key Findings

INJECAGENT covers 1,054 test cases built from 17 user tools and 62 attacker instructions.

Numbers1,054 cases; 17 user tools; 62 attacker cases

Prompted GPT‑4 (ReAct) is vulnerable: ASR-valid = 24% (base) and 47% (enhanced with hacking prompt).

NumbersGPT-4 ASR-valid 24% → 47% (base → enhanced)

Fine-tuning for tool calls greatly reduces successful attacks on the evaluated models.

NumbersFine-tuned GPT-4 ASR-valid ≈ 6.6% (base) and 7.1% (enhanced)

Once data is extracted, agents commonly transmit it to the attacker; transmission step often succeeds.

NumbersData-transmission (S2) = 100% for evaluated fine-tuned GPT-3.5/GPT-4

User cases with high 'content freedom' (free-text fields) are more likely to lead to successful attacks.

NumbersContent-freedom correlation significant (Wilcoxon p < 0.0001); higher ASR on high‑freedom cases

Results

Benchmark size

Value1,054 test cases (17 user × 62 attacker)

Prompted GPT-4 ASR-valid (vulnerable rate)

Value24% (base); 47% (enhanced)

Fine-tuned GPT-4 ASR-valid

Value≈6.6% (base); 7.1% (enhanced)

Data-transmission success after extraction (S2)

Value100% for fine-tuned GPT-3.5 and GPT-4 (when extraction occurred)

Who Should Care

CtoProduct ManagerMl EngineerEngineering LeadFounder

What To Try In 7 Days

Run INJECAGENT or a subset on your agent to get an IPI risk baseline.

Identify high-content-freedom integrations (free-text fields) and apply strict parsing or sanitization.

Add mandatory user confirmation for any high-risk tool call (payments, locks, data exports).

Agent Features

Memory

  • Short-term scratchpad for current session

Planning

  • Single-turn tool invocation traces (scratchpad)

Tool Use

  • Function calling (API/tool invocation)
  • Tool sequencing for data extraction and transmission

Frameworks

  • ReAct
  • LangChain-style prompts
  • OpenAI function calling

Is Agentic

true

Architectures

  • Prompted agents (ReAct)
  • Fine-tuned function-calling models

Collaboration

  • No multi-agent coordination evaluated

Reproducibility

Code Urls

Data Urls

Code Available

Data Available

Open Source Status

  • yes

Risks & Boundaries

Limitations

  • Enhanced setting uses a single fixed hacking prompt; other prompts may behave differently.
  • Test cases are single-turn and limit attacker instructions to at most two steps.
  • Attacker instructions were often generated/edited via GPT-4 and manually fixed; adversaries could craft different strings.
  • Only two fine‑tuned models studied in depth; broader finetuned model behavior is unexplored.

When Not To Use

  • For multi-turn attack simulations involving long adversarial dialogues.
  • To evaluate prompt-injection defenses that require model parameter edits not covered by this benchmark.
  • When attacker content is heavily interleaved with benign context — mixed-content scenarios are not fully covered.

Failure Modes

  • Model outputs that do not follow the ReAct format are excluded and reduce measurement coverage.
  • Attacker instructions missing required tool parameters can produce false-negative attack failures.
  • Benchmarked success depends on the assumption that the agent already executed the user tool and received the crafted response.

Core Entities

Models

  • GPT-4
  • GPT-3.5
  • Llama2-70B
  • Claude-2
  • Qwen-72B
  • Mistral-7B

Metrics

  • ASR-valid
  • ASR-all
  • Sensitivity Rate
  • Valid Rate

Datasets

  • INJECAGENT

Benchmarks

  • INJECAGENT