Overview
The study uses reproducible code, repeated runs, and production-like deployments; results directly map to operational choices and show consistent large cost/latency gaps favoring encoders.
Citations0
Evidence Strength0.90
Confidence0.88
Risk Signals10
Trust Signals
Findings with numeric evidence: 5/5
Findings with evidence refs: 5/5
Results with explicit delta: 6/6
Reproducibility
Status: Code + data available
Open source: Yes
At A Glance
Cost impact: 92%
Production readiness: 88%
Novelty: 30%
Why It Matters For Business
Choosing LLM prompting for routine fixed-label classification often increases operating cost by 10–100× and raises tail latency risk; using fine-tuned encoders saves money, stabilizes SLAs, and eases governance.
Who Should Care
Summary TLDR
This paper benchmarks fine-tuned BERT-family encoders against zero- and few-shot LLM prompting (GPT-4o, Claude Sonnet 4.5) on IMDB, SST-2, AG News, and DBPedia. It measures macro-F1, end-to-end latency (p50/p95/p99), TTFT, and per-request cost. Result: encoders match or exceed LLM accuracy on these fixed-label tasks while delivering one to two orders of magnitude lower inference cost and much tighter tail latency. DistilBERT often wins the utility ranking (best trade-off). The authors release code and configs to reproduce cost/latency estimates.
Problem Statement
Model choice for fixed-label text classification is usually driven by accuracy alone. In production, latency, tail behavior, recurring inference cost, and governance (reproducibility, versioning) matter just as much. The paper asks: when does LLM prompting justify its higher operational overhead compared to fine-tuned encoders?
Main Contribution
A reproducible benchmark comparing fine-tuned BERT-family encoders and zero/few-shot LLM prompting on four standard datasets, reporting macro-F1, latency percentiles, TTFT, and per-request cost.
A decision framework: Pareto-front analysis plus a parameterized utility function that ranks models under different latency tolerances.
Key Findings
Fine-tuned encoders match or exceed LLM prompting on structured fixed-label tasks while running far cheaper and faster.
Tail latency and TTFT for LLM APIs are much higher and more variable than for encoders.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| IMDB macro-F1 | RoBERTa 94.84% ±0.12 vs Claude 4.5 FS 96.48% ±0.01 | RoBERTa encoder | LLM +1.64 F1 points | IMDB test | Table 2 F1 rows | Table 2 |
| IMDB estimated cost (USD / 1M req) | DistilBERT $12.44 vs GPT-4o ZS $842.78 | DistilBERT encoder | LLM ~67.7× higher cost | IMDB cost estimate | Table 2 cost rows | Table 2 |
What To Try In 7 Days
Run this paper's repo to measure your p50/p95 latency and cost using your pricing snapshot.
Fine-tune DistilBERT on your label set as a baseline and measure macro-F1 vs latency and cost.
Compute the provided utility function with your τ (latency tolerance) to pick a deployment candidate.
Optimization Features
Token Efficiency
Infra Optimization
measure end-to-end latency on realistic serverless infra (Cloud Run) rather than raw hardware timing
Model Optimization
System Optimization
Training Optimization
Inference Optimization
Reproducibility
Code URLs
Data URLs
Risks & Boundaries
Limitations
Benchmarks limited to four English fixed-label datasets; results may differ for open-ended, high-ambiguity tasks.
Latency and cost estimates tied to specific Cloud Run config and January 22, 2026 pricing snapshot.
When Not To Use
When the task requires open-ended generation, schema discovery, or evolving taxonomies where LLM reasoning adds unique value.
Low-volume prototypes where per-request cost is negligible and developer convenience matters more than operating cost.
Failure Modes
Adopting few-shot LLM prompting at scale causes high token bills and unpredictable tail latencies.
Relying on API LLMs creates vendor-lock and silent behavior changes from provider updates.