Overview
The method shows strong empirical gains on a 16k MT test set and uses standard components (LoRA, ORPO), but is validated only on MT and on preferences collected with GPT-4o, so wider production readiness requires more cross-model and task checks.
Citations0
Evidence Strength0.80
Confidence0.87
Risk Signals10
Trust Signals
Findings with numeric evidence: 4/4
Findings with evidence refs: 4/4
Results with explicit delta: 4/4
Reproducibility
Status: Partial assets available
Open source: Partial
At A Glance
Cost impact: 80%
Production readiness: 70%
Novelty: 60%
Why It Matters For Business
Compress evaluation prompts to cut LLM token bills by roughly 2.4× while keeping metric quality, making large-scale or repeated MT evaluations more affordable.
Who Should Care
Summary TLDR
PromptOptMe trains a small model to compress the inputs of LLM-based MT evaluators (like GEMBA-MQM) in two stages: supervised fine-tuning to preserve error spans, then preference optimization (ORPO) to prefer compressions whose LLM scores match uncompressed prompts. On a 16k test set the best setup cuts token use up to 2.37× (19M → ≈8M tokens) while keeping system-level accuracy and often improving segment Kendall τ on evaluated language pairs. Task-agnostic compressors (LLMLingua-2) break metric quality, showing the need to preserve error spans.
Problem Statement
LLM-based MT metrics give strong human alignment but prompts are long and expensive (≈1100–1200 tokens per example). This paper asks: can a small model compress prompt inputs to cut token costs while preserving evaluation quality?
Main Contribution
A two-stage prompt compression pipeline: supervised fine-tuning to preserve MQM error spans, then ORPO preference optimization to prefer compressions whose LLM scores match uncompressed inputs.
An MT-focused compression model (PROMPTOPTME) based on LLaMA-3.2 1B/3B that reduces token use up to 2.37× on a 16k test set while retaining or improving evaluated metric quality.
Key Findings
Up to 2.37× reduction in input tokens for MT metric evaluation on the evaluated 16k test set.
Prompt compression can preserve or improve measured metric quality on several language pairs.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Token Usage | 19M → 8.07M | 19M (GPT-4o ref) | 2.37× reduction | 16k test set (news domain) | Table 1: GPT-4o ref vs PROMPTOPTME-3B with GPT-4o lite | Section 5; Table 1 |
| Accuracy | 0.7736 (PROMPTOPTME-3B with GPT-4o lite) | 0.7789 (GPT-4o ref) | ≈ -0.0053 absolute (comparable) | 16k test set | Table 1; Section 5 | Table 1 |
What To Try In 7 Days
Measure token usage and quality of your current LLM MT metric on a small test slice.
Swap full prompt for the paper's simplified prompt and re-run a sample evaluation.
Train a small compressor (LoRA on LLaMA-3.2 3B) on MQM data and test token savings vs quality on 1k examples.
Optimization Features
Token Efficiency
Model Optimization
Training Optimization
Inference Optimization
Reproducibility
Risks & Boundaries
Limitations
Evaluated only on machine translation using MQM-annotated WMT data; generalization to other NLG tasks is untested.
Preference data and evaluation rely on GPT-4o; learned compressor may encode evaluator-specific bias.
When Not To Use
When no error-span annotations (MQM-like) are available for supervision.
When strict adherence to full MQM error-typology is required for interpretability.
Failure Modes
Task-agnostic compressors can severely reduce metric quality (near-zero segment τ).
Compressed prompts may omit fine-grained MQM categories, hurting error-category analyses.