Overview
The benchmark and attacks are practical and reproducible, but some evaluations are limited (GPT-4V on subsets, CLIP-tuned attacks) so use results as targeted diagnostics rather than universal guarantees.
Citations4
Evidence Strength0.70
Confidence0.80
Risk Signals11
Trust Signals
Findings with numeric evidence: 5/5
Findings with evidence refs: 5/5
Results with explicit delta: 3/5
Reproducibility
Status: Code + data available
Open source: Partial
At A Glance
Cost impact: 60%
Production readiness: 60%
Novelty: 50%
Why It Matters For Business
If you deploy image+text models, simple visual attacks and text changes can break behavior; test both inputs and add safety-aware visual instruction tuning before release.
Who Should Care
Summary TLDR
The authors build a safety benchmark for Vision LLMs (VLLMs) covering out-of-distribution (OOD) VQA and red-teaming attacks. They release two OOD datasets (OODCV-VQA and Sketchy-VQA, each with a harder variant) and adversarial test sets for vision and language attacks. Key takeaways: VLLMs often read unusual images well (yes/no), but fail when the text is counterfactual; simple CLIP-based image perturbations can mislead many VLLMs; sketch images are hard; vision-only jailbreaking is limited in transfer; vision-language tuning can weaken LLM safety. The authors evaluate 21 models including GPT-4V and open-source VLLMs and release code/data.
Problem Statement
VLLMs are rapidly deployed for image+text tasks, but we lack a focused safety benchmark that tests both out-of-distribution visual cases and adversarial/jailbreak attacks on visual and language inputs. The paper fills this gap with new OOD datasets and redteaming attacks to measure safety weaknesses in current VLLMs.
Main Contribution
A safety benchmark with two OOD VQA datasets (OODCV-VQA, Sketchy-VQA) and harder counterfactual / rare-object variants.
Two straightforward CLIP-ViT based adversarial attacks (SIN.ATTACK and MIX.ATTACK) and transfer/jailbreak evaluations.
Key Findings
VLLMs answer OOD visual yes/no questions very well but fail when text is counterfactual.
Sketch images with minimal detail cause consistent recognition failures.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Accuracy | GPT-4V 80.61% | many open VLLMs 50–76% (varies by model) | — | OODCV-VQA (Table 5) | GPT-4V overall 80.61; top open models (InstructBLIP, CogVLM, Qwen-VL) 72–76% | Table 5 |
| Accuracy | Most VLLMs >=95% on Yes/No | — | — | OODCV-VQA Yes/No (Table 5) | Multiple models report >=95% Yes/No accuracy; GPT-4V 100% | Table 5 |
What To Try In 7 Days
Run OODCV-VQA and Sketchy-VQA on your VLLM to spot weaknesses in counting, sketches, and counterfactual text.
Apply CLIP-based SIN/MIX attacks to a small image sample to check if your system hallucinates or rejects.
Re-evaluate safety rules after any vision-language fine-tuning and add counterfactual/text-robust examples to safety data.
Agent Features
Architectures
Reproducibility
Risks & Boundaries
Limitations
GPT-4V was evaluated only on selected challenging subsets, not full benchmark.
Counterfactual questions are template-generated, which may differ from human-written counterfactuals.
When Not To Use
Not a coverage test for generative image synthesis or creative multimodal tasks.
Not a definitive proof of robustness—only diagnostic for the tested attack families and OOD types.
Failure Modes
Models may refuse to answer (rejection) which skews 'misleading' vs 'rejection' metrics.
Template-based counterfactuals can produce artificial failure modes not seen in real user input.