Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGPrompt OptimizationAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection

ERI: 57,750 engineering instruction-response items across 9 fields to test LLM reasoning and agent tool-use

LLM EvaluationBenchmark ConstructionReasoning BenchmarksAgent PipelinesBenchmark
February 16, 20267 min

Overview

Decision SnapshotNeeds Validation

ERI is ready as a regression and diagnostic tool for engineering LLM evaluation; use it with human review and slice audits before trusting outputs for high-stakes decisions.

Citations0

Evidence Strength0.80

Confidence0.90

Risk Signals11

Trust Signals

Findings with numeric evidence: 5/5

Findings with evidence refs: 5/5

Results with explicit delta: 0/7

Reproducibility

Status: Code + data available

Open source: Partial

At A Glance

Cost impact: 70%

Production readiness: 70%

Novelty: 60%

Authors

MZ Naser, Ahmad Bani Awwad, Zoie McCreery, Radwa Eissa, Ahmad Naser, Gianluca Cusatis, Andrew Metcalf, Kapil Madathil, Jamal Abdalla, Venkatesh Kodur, Mohammad Reza Saeb

Links

Abstract / PDF / Code / Data

Why It Matters For Business

ERI lets teams measure engineering-specific failure modes (units, constraints, verification) so you can route tasks to cheaper models when safe and reserve frontier models for high-risk work.

Who Should Care

CTOProduct ManagerML EngineerEngineering LeadData Scientist

Summary TLDR

ERI is a large, taxonomy-driven benchmark built to test engineering reasoning and instruction following in LLMs and agent pipelines. It covers 9 engineering fields, 55 subdomains, 7 intent types (definition, explanation, calculation, comparison, design, troubleshooting, code), and 3 difficulty tiers, producing 57,750 structured instruction-response records. The authors release JSONL splits, taxonomy specs, validation scripts, and an evaluation harness. In a 10% stratified test, frontier LLMs (GPT-5, Claude Sonnet 4, DeepSeek V3.1) averaged >4.30/5, smaller 7–8B models averaged ~3.0 with >10% failures, and the team bounds hallucination contamination in references to ~1.7%. ERI is meant for R

Problem Statement

General benchmarks miss engineering-specific checks like units, constraints, and verification steps. ERI addresses that gap by forcing structured coverage across field, intent, and difficulty so teams can detect silent failures (e.g., violating constraints or inventing assumptions) and make reliable routing, fine-tuning, or audit decisions.

Main Contribution

A 57,750-item instruction-response dataset with per-item metadata (field, subdomain, intent, difficulty).

A taxonomy covering 9 engineering fields and 55 subdomains to guarantee slice-level coverage.

Key Findings

ERI contains 57,750 instruction-response records with explicit metadata.

Numbers57,750 records; 1,155 metadata cells (50 per cell)

Practical UseYou can evaluate or fine-tune models on controlled slices (field × intent × difficulty) rather than only broad averages.

Evidence RefAbstract; Sec 4.1; Sec 5.1

Frontier models deliver near-expert average quality on ERI.

NumbersGPT-5 mean 4.48/5 (σ=0.49); top-3 >4.30

Practical UseReserve high-risk, high-rigor engineering tasks for frontier models to reduce silent constraint violations.

Evidence RefSec 6.2, Fig.3

Results

MetricValueBaselineDeltaSplit / DatasetEvidenceEvidence Ref
Dataset size57,750 records (1,155 cells; 50 samples per cell)——fullSec 4.1; AbstractSec 4.1
Top model mean scoreGPT-5 mean 4.48/5 (σ=0.49)——10% stratified testSec 6.2, Fig.3Sec 6.2

What To Try In 7 Days

Load the ERI test split from HuggingFace and run your model on a few targeted slices (e.g., CALC, DES).

Use the provided evaluation harness to collect automatic checks and rubric scores for those slices.

Set up a simple router: route undergraduate CALC to a cheaper model and graduate/DES to a stronger model; measure cost vs. failure rate change.

Agent Features

Planning
tool planning evaluation (calculator/solver integration)
Tool Use
calculator/solver/checker integrationplanned tool-trace extension for agent evaluation
Frameworks
LLM-as-a-judgeSelf-Instruct generation
Is Agentic

Yes

Architectures
MoEmodel-as-judge panel
Collaboration
multi-judge aggregation across providers

Optimization Features

Training Optimization
SFTLoRA
Inference Optimization
routing low-rigor slices to smaller models to save cost

Reproducibility

Code AvailableYes
Data AvailableYes
Open Source StatusPartial
LicenseUnknown

Code URLs

https://github.com/mznaser-clemson/ERI-Benchmarkhttps://huggingface.co/datasets/mznaser/ERI-Benchmark

Data URLs

https://huggingface.co/datasets/mznaser/ERI-Benchmark

Risks & Boundaries

Limitations

Synthetic generation inherits generator model biases and temporal anchoring.

Not a substitute for jurisdiction-specific code compliance or stamped engineering review.

When Not To Use

Certifying safety or legal compliance

Directly automating high-stakes control without human oversight

Failure Modes

Silent constraint violations (units, bounds) despite plausible prose

Overfitting to benchmark phrasing (teaching to the test)

Core Entities

Models

GPT-5Claude Sonnet 4DeepSeek V3.1Claude Haiku 4.5GPT-4.1 MiniMistral Small 3Mistral 7BLlama 3.3 70BQwen 2.5 7BLlama 3.1 8BGPT-5.1

Metrics

mean score (1-5)failure rate (score ≤2)Spearman inter-judge correlationσ (per-item standard deviation)perfect-score rate

Datasets

ERI-Benchmark (mznaser/ERI-Benchmark on HuggingFace)

Benchmarks

MMLUBIG-benchGSM8KEngDesignSoM-1K

You May Also Want to Read

Large Llama models beat smaller ones on iCliniq medical QA; LLM judges help measure clinical quality

0.30
0.30
0.60
0

This paper shows large open models can give much better zero-shot medical answers, but parameter-efficient architectures can approach top quality. For product teams, that means you can trade compute cost for accuracy and still get usable clinical answers if you pick the right model and add quality checks.

Key finding

Llama-3.3-70B achieved the highest automatic and judge scores.

Numbers: BLEU-1 0.2207; ROUGE-1 0.2761; Judge 4.40/5

PanCanBench: 282 real patient questions + 3,130 expert rubrics to test LLM clinical completeness and factuality

0.60
0.50
0.60
0

If you plan to deploy LLMs for patient-facing oncology guidance, you must test both completeness and factual accuracy on real patient questions; rubric scores alone can be misleading and AI-only rubrics overestimate quality.

Key finding

Models' rubric-based completeness scores vary widely.

Numbers: Top score 82.3% (o3); range 46.5%–82.3%

A judge-free, n-gram benchmark that approximates GPT-4 judging for Japanese Q&A

0.70
0.55
0.80
0

You can cheaply and deterministically evaluate short Japanese Q&A outputs without repeated human or expensive LLM judging, cutting evaluation cost and speeding model iteration.

Key finding

Benchmark scores correlate very highly with GPT-4o judge scores.

Numbers: Pearson r = 0.9896 (Section 5.2; Fig.4)

Professional multilingual TruthfulQA shows truth gaps across languages but smaller than expected

0.60
0.50
0.60
0

Multilingual truth checks are necessary: models can be less accurate in low-resource languages and simple automatic scores (MC2) can mislead decisions; judge-style evaluation and MT-enabled datasets reduce cost and improve cross-lingual assessment.

Key finding

LLM-as-a-Judge correlates better with humans than MC2.

Numbers: Cohen Kappa: Judge (Gemma-2-9b-inst) vs human up to 0.75; MC2 lower

MEMERAG: a native multilingual benchmark to evaluate RAG outputs and LLM-based evaluators

0.70
0.60
0.60
0

If you deploy RAG across multiple languages, MEMERAG gives a native-language testbed to validate whether automatic evaluators match human judgements; use it before trusting LLM judges for quality gating.

Key finding

High coarse-grained inter-annotator agreement for faithfulness and relevance.

Numbers: Faithfulness Gwet's AC1 0.84–0.93; Relevance Gwet's AC1 0.95–1.00 (Table 2)