Overview
The study offers a clear, implementable test suite and reports statistically significant differences across models; however the small problem corpus and single-shot sampling limit generality.
Citations0
Evidence Strength0.78
Confidence0.90
Risk Signals9
Trust Signals
Findings with numeric evidence: 4/4
Findings with evidence refs: 4/4
Results with explicit delta: 0/5
Reproducibility
Status: No open assets linked
Open source: Partial
At A Glance
Cost impact: 40%
Production readiness: 60%
Novelty: 70%
Why It Matters For Business
Models that score well on standard tests may behave unpredictably when users reword requests; testing for semantic invariance prevents surprising errors in customer-facing or safety-critical agents.
Who Should Care
Summary TLDR
The paper introduces a metamorphic testing framework to measure whether LLM-based reasoning agents give consistent outputs when semantically equivalent problem statements are rewritten. Across 7 models and 19 multi-step problems, smaller models sometimes produce more stable reasoning than larger ones. Contrastive prompts (adding plausible distractors) consistently break reasoning. The work exposes robustness patterns invisible to standard accuracy benchmarks and offers a practical test-suite for evaluating agent reliability.
Problem Statement
LLM agents can change answers when the same problem is reworded. Standard accuracy benchmarks use fixed phrasing and miss this instability. The paper asks: how stable are reasoning agents under semantic-preserving rewrites, and which models or architectures are most vulnerable?
Main Contribution
A metamorphic testing framework with eight semantic-preserving transformations (identity, paraphrase, reorder facts, expand, contract, academic framing, business framing, contrastive).
A multi-model study across seven foundation models (Hermes, Qwen3, DeepSeek-R1, gpt-oss) using 19 multi-step problems in eight domains.
Key Findings
Smaller model showed highest robustness: Qwen3-30B-A3B had the best stability and trace similarity across transformations.
Contrastive (distractor) transformations consistently reduced scores across all models.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Qwen3-30B-A3B: Stability Rate | 79.6% | — | — | All transformations, 19 problems | Highest reported stability across models | Table 3 |
| Qwen3-30B-A3B: MAD | 0.049 | — | — | All transformations | Lowest mean absolute score change | Table 3 |
What To Try In 7 Days
Run the paper's 8 metamorphic transformations on your top candidate model for a small set of representative tasks.
Measure Stability Rate and MAD; shortlist models with low MAD before deployment.
Add a lightweight filter that flags contrastive or distractor-rich inputs for human review or ensemble handling.
Agent Features
Memory
Planning
Frameworks
Is Agentic
Yes
Architectures
Reproducibility
Risks & Boundaries
Limitations
Only 19 problems across eight domains; coverage is limited.
Single inference per problem-transformation pair; does not capture sampling variability.
When Not To Use
When you need absolute accuracy metrics across large public benchmarks (this complements, not replaces, such benchmarks).
When input rewrites intentionally change problem semantics (contrastive was included as a stress test only).
Failure Modes
Contrastive/distractor prompts causing major answer shifts.
Fact reordering breaking models that rely on presentation order.