You can cut ensemble inference cost by roughly 4× while improving automatic quality, making LLM deployment cheaper and more scalable for high-throughput services.
MODI achieves higher automatic-quality than prior ensembling on MixInstruct.
Numbers: BARTScore: MODI −2.14 vs LLM-BLENDER −2.77 (Δ +0.63)
ABC can cut real inference costs quickly by routing easy inputs to cheap models, reducing cloud, network, and API bills while keeping quality.
ABC matches or improves accuracy over the single best model while lowering compute.
Numbers: Accuracy +1–2 percentage points on Pareto frontier (Figure 2).
You can cut model invocation spend substantially (up to ~27% on benchmarks) without retraining routing logic, onboard new models with 20–50 tests, and add routing with only milliseconds of overhead.
Routing reduces inference cost on evaluated benchmarks while keeping similar accuracy.
Numbers: Up to 27% cost reduction (e.g., MMLU) and 11% reduction on AlpacaEval GPT4/Mixtral setting
Routing tokens around low-impact layers can cut average compute and latency per request, reducing serving costs and allowing larger models for the same budget.
Per-layer residual contributions shrink as model size grows.
Numbers: OPT-125M median residual ratio ≈ 20%; OPT-66B ≈ 5.9%
A small learned router can cut GPU costs or delay scaling while keeping user-facing LLM services responsive during bursts, increasing quality-per-GPU and availability.
Learned router preserves availability at much higher arrival rates than serving only the large model.
Numbers: Remains available for >10× faster arrival rates than OPT-6.7B (stable workload).
HyFunc reduces latency and compute for live API-style agents, enabling faster, cheaper, and more responsive assistants without sacrificing accuracy.
HyFunc reduces end-to-end inference latency to 0.828 seconds per case.
Numbers: Latency = 0.828s (HyFunc ♣, Table 1)
Routing saves inference cost by sending easy queries to cheaper models. That lowers cloud bills and lets you scale reasoning services while keeping top-model accuracy.
Middle layers of a strong reasoning model carry the most signal for difficulty and correctness prediction.
If you host code-completion services, cascading can cut inference costs substantially while holding accuracy steady. It is a low-risk, black-box add-on that uses validation to pick cost-aware plans.
Cascading reduces inference cost on evaluated benchmarks, with average savings reported at 26% and up to 70% in the best case.
Numbers: avg 26% cost reduction; best-case 70% (paper abstract)
Routing can improve accuracy or cut API cost, but many published routers give similar gains; practical wins come from curated model pools and simple, cheap routers. Always test routers against your Best Single baseline and measure cost/latency together.
Models show clear complementarity: no single model dominates all tasks.
Numbers: Table11: dataset-level bests; many datasets led by different models
You can cut API and compute bills substantially by asking a small model if it 'knows' and how confident its token choice is, then only escalate uncertain queries to a bigger model.
8B→70B cascade matches 70B accuracy with much less compute
Numbers: Acc: 8B→70B 83.22% vs 70B 83.57%; Reduced CC 36.46%; PD -0.35%
