Overview
The review aggregates many mature techniques (quantization, PTQ, MoE, sharding) ready for prototypes; remaining engineering work is hardware and system integration.
Citations2
Evidence Strength0.70
Confidence0.85
Risk Signals10
Trust Signals
Findings with numeric evidence: 6/6
Findings with evidence refs: 6/6
Results with explicit delta: 1/6
Reproducibility
Status: Partial assets available
Open source: Partial
At A Glance
Cost impact: 80%
Production readiness: 70%
Novelty: 55%
Why It Matters For Business
On-device LLMs cut latency, protect user data, and lower cloud bills—key benefits for mobile apps, privacy-focused services, and offline products.
Who Should Care
Summary TLDR
This 38-page review maps the state of running large language models (LLMs) on edge devices. It organizes methods (quantization, pruning, distillation, low-rank factorization, MoE, parameter sharing), software stacks (llama.cpp, MLC-LLM, VLLM), hardware trends (GPUs, NPUs, PIM/PNM), and deployment patterns (edge-only, edge-cloud sharding). The paper compiles numbers from many recent works (e.g., AWQ 3× speedups, EdgeShard up to 50% latency drop, LLMCad 9.3× token speedups) and flags open problems: energy, continual learning, privacy, and hardware-software co-design.
Problem Statement
Cloud LLMs lead to latency, privacy risk, and recurring cloud cost. Running LLMs on phones and edge devices promises instant replies and local data control but is hard because of limited RAM, compute, energy, and thermal budgets. The review asks: which model, compression, and deployment methods make on-device LLMs practical, and what open problems remain?
Main Contribution
Taxonomy of techniques to make LLMs run on edge: compression, efficient architectures, MoE, and collaborative deployment
Survey of software frameworks and hardware options for on-device inference and tiny training
Key Findings
Edge AI market projected to grow nearly tenfold to $143.6B by 2032.
Post-training activation-aware quantization (AWQ) preserves a tiny fraction of weights and enables large speedups on mobile GPUs.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Edge AI market projection | $143.6B by 2032 | $15.2B in 2022 | ≈9.4× | — | Market.us report (2024) | Market.us (2024) |
| AWQ speedup on mobile GPUs | up to 3× | FP16 implementation | — | — | TinyChat + AWQ evaluation | Lin et al. (2024b) |
What To Try In 7 Days
Run a small on-device proof-of-concept using llama.cpp or MLC-LLM with a 1–7B model and AWQ/PTQ.
Measure TTFT and energy-per-token on target phones; compare to cloud API baseline.
Prototype a hybrid flow: local fast generator + cloud verifier to balance latency and quality.
Agent Features
Memory
Frameworks
Architectures
Collaboration
Optimization Features
Token Efficiency
Infra Optimization
Model Optimization
System Optimization
Training Optimization
Inference Optimization
Reproducibility
Risks & Boundaries
Limitations
Many reported gains depend on specific hardware and are not universally reproducible.
Quantization and pruning introduce accuracy trade-offs that vary by model and task.
When Not To Use
If device memory/compute is extremely small (microcontrollers) prefer server inference.
When strict, continually updated model knowledge is required and cloud-only models provide fresher data.
Failure Modes
Accuracy drop after aggressive quantization or pruning on certain tasks.
Battery drain and device thermal throttling during sustained inference.