PETALS lets teams share idle consumer GPUs to run 50B+ models interactively, cutting the need for expensive multi‑GPU servers and lowering inference latency versus RAM offloading; consider privacy and trust tradeoffs.

On Ampere GPUs, INT4 computation can sharply reduce latency and cost for encoder-based workloads (search, classification, embedding). But it is risky to use for autoregressive generation (chatbots, text generation) until activation-quantization problems are solved.

Cutting cold-start time from minutes to seconds reduces user-visible latency, lowers GPU idle costs, and increases successful request completion for large models, improving SLA and capacity efficiency.

FastGen can double effective throughput and sharply cut tail latency on long-prompt, streaming chat workloads, which lowers GPU cost per successful request and improves user-perceived responsiveness.

Co-designing model and device cuts operational energy and peak power by an order of magnitude while keeping or slightly improving accuracy — lowering battery drain, thermal limits and cloud costs for on-device NLP.

Quantizing ViTs to 8-bit often preserves accuracy while halving memory and improving throughput on INT8-capable hardware, enabling real-time and edge deployment with lower cost.

MemServe lets operators combine caching and disaggregated inference to cut end-to-end latency and tail times for many chat and long-context workloads, lowering hardware cost per request and improving user-perceived responsiveness.

When you run multiple LLM-based agents in parallel, overall response time depends on the slowest chain of steps (the critical path). Training the orchestration policy to minimize that path reduces latency a lot without sacrificing accuracy, which helps interactive products and time-sensitive workflows.

If your product uses chat or multi-turn flows, caching KV states and overlapping cache IO with GPU work can cut latency and cloud GPU costs dramatically.

Aladdin can cut GPU spending by tens of percent while keeping token-level SLOs, turning large inference clusters from a fixed-cost bottleneck into a more efficient, demand-driven service.

Better online scheduling of token decoding can cut average latency and GPU-hours: simulations show ~31% lower latency versus a memory-aware baseline, implying fewer GPUs and lower energy costs.

You can deploy competitive RAG QA at lower cloud cost by using Trainium/Inferentia2 plus serving optimizations, enabling cheaper fine-tuning and elastic scaling for production assistants.

throttLL'eM can lower GPU energy costs by tens of percent while keeping latency SLOs, reducing operating cost and carbon footprint for LLM inference services.

Microserving lets ops teams reprogram serving coordination at the router level without restarting engines, cutting tail latency and compute waste on long-input workloads and enabling live tuning of prefill/decode balance.

OrbitFlow reduces token-level latency violations and raises throughput for long-context LLM services, improving user-perceived responsiveness and allowing more requests per GPU under real workloads.

TableCache cuts Text-to‑SQL response latency by precomputing and reusing table caches, improving user experience and lowering repeated GPU compute costs in applications where users query shared tables.

Minions can materially reduce serving latency and multiply throughput for conversational LLMs without retraining large models, lowering operational cost and improving user responsiveness on evaluated workloads.

Cutting model size and latency lets teams run transformers on phones and edge devices, reducing server costs and improving responsiveness.

HAQ plus CPU–GPU scheduling lets teams run large MoE models on consumer GPUs with near-full-precision accuracy, lowering GPU costs and enabling predictable, lower-latency edge services.

Streaming-dLLM cuts inference compute and latency dramatically for diffusion LLMs without retraining. That reduces cloud GPU cost and improves responsiveness for production services that use dLLMs for long or batch generation.

ThunderServe lets you run more LLM replicas for the same cloud spend by mapping compute-heavy and memory-bound phases to different GPU types and by cutting KV transfer costs—so you can increase throughput and meet SLOs without buying only high-end GPUs.

HydraServe makes serverless LLMs more reliable by cutting time-to-first-token and raising SLO attainment for bursty, long-tail workloads while keeping costs similar or lower.

OD-LLM trims LLM memory in half while keeping ranking quality and cutting inference time, making on-device personalization feasible for latency- and privacy-sensitive apps.

CITER lowers memory-transfer costs and latency during decoding by routing only the important tokens to the big model, cutting cloud/GPU bill and improving realtime responsiveness while keeping quality.