Use the host CPU to offload decoding attention and KV cache so GPUs can batch larger and give higher online throughput

Overview

Production Readiness

0.7

Novelty Score

0.6

Cost Impact Score

0.8

Citation Count

1

Authors

Xuanlin Jiang, Yang Zhou, Shiyi Cao, Ion Stoica, Minlan Yu

Links

Abstract / PDF

Why It Matters For Business

NEO lets you squeeze more online throughput from existing GPU servers by using the host CPU, lowering per-token serving cost where GPU memory is the bottleneck.

Summary TLDR

NEO is a systems design that offloads only the decoding attention and corresponding KV cache to the host CPU while keeping model weights on the GPU. It uses asymmetric GPU–CPU pipelining and a load-aware scheduler to balance work across devices, increasing effective GPU batch sizes and overall token throughput without changing model accuracy or average latency. On tested workloads and hardware, NEO improves throughput from modest (≈10–30%) on high-end GPUs to multiple-times on memory-limited GPUs (up to ~7.5× on T4-like cases), with larger CPU memory bandwidth giving larger gains.

Problem Statement

GPUs have limited memory so the KV cache and per-request state restrict batch size and waste GPU compute. Existing offloading either hurts latency or overloads CPUs. How to offload parts of inference to the host CPU to increase GPU batch size and throughput for online (low-latency) serving while keeping latency and accuracy unchanged?

Main Contribution

Design of asymmetric GPU–CPU pipelining: partial offload of decoding attention+KV cache to keep both devices busy.

Load-aware scheduler that dynamically assigns requests per iteration to balance CPU/GPU busy time and maximize throughput.

Efficient CPU implementation (PACPU using ISPC) and an end-to-end prototype built on SwiftLLM with multi-GPU support.

Key Findings

NEO raises throughput significantly on memory-limited GPUs.

Numbersup to 7.5× on T4; 26% on A10G; 14% on H100 (reported maxima)

CPU memory bandwidth controls offload benefit more than CPU cores.

Numbersup to 79.3% extra throughput on A10G as CPU bandwidth grows

NEO keeps latency comparable to GPU-only systems while increasing throughput.

Numbers14.3% higher throughput on H100 at 2s latency; 563% on T4 at 1s latency

Full offloading (FastDecode+) can be worse on irregular workloads.

NumbersFastDecode+ falls below 60% of baseline throughput as output length grows

Results

relative throughput vs GPU-only

Valueup to 7.5× (T4), 26% (A10G), 14% (H100)

BaselineGPU-only inference (SwiftLLM / vLLM-like)

throughput gain with larger CPU bandwidth

Valueup to 79.3% (A10G, larger g5 instances)

Baselinesame GPU, smaller CPU bandwidth instances

throughput at fixed latency points vs vLLM

Value14.3% higher on H100 at 2s; 6.40% on A10G at 2s; 563% on T4 at 1s

BaselinevLLM

Who Should Care

CtoEngineering LeadMl EngineerData ScientistFounder

What To Try In 7 Days

Measure if GPU memory limits batch size on your workloads (track KV-cache footprint).

Run a small NEO prototype on one GPU server: enable CPU offload for decoding attention only.

Compare throughput/latency vs current GPU-only stack and try a higher-memory-bandwidth host instance if CPU-bound.

Agent Features

Memory

KV cache split into GPU-cache and CPU-cache

Frameworks

SwiftLLM-based prototype

Optimization Features

Infra Optimization

prefer hosts with higher CPU memory bandwidth for offloading
keep weights on GPU to minimize PCIe transfers

System Optimization

reduce kernel-launch overhead by replacing Triton-JIT kernels with CUDA C++
multi-GPU model sharding via Ray and NCCL

Inference Optimization

partial CPU offloading of decoding attention
asymmetric GPU–CPU pipelining
load-aware per-iteration scheduler
layer-wise KV swap to overlap transfer with compute
paged KV cache on CPU

Reproducibility

Data Urls

Data Available

Open Source Status

partial

Risks & Boundaries

Limitations

Benefits depend heavily on host CPU memory bandwidth; low-bandwidth CPUs limit gains.
Scheduler relies on offline profiling; profiling inaccuracies can cause suboptimal scheduling.
NEO does not offload model weights; it assumes GPU holds all weights.
Prototype (SwiftLLM-based) is less optimized than production vLLM for multi-GPU.

When Not To Use

If your GPU has enough memory to achieve large batch sizes and already saturates GPU compute.
When host CPU memory bandwidth is low relative to workload demand.
For extremely short-output workloads where offload overheads outweigh benefits.
When remote CPUs would be required and network transfers add latency.

Failure Modes

CPU becomes the bottleneck (memory bandwidth or compute) and reduces throughput.
Excessive PCIe transfer if offloading policy causes repeated swapping.
Suboptimal per-iteration scheduling causes pipeline bubbles and lost throughput.
Kernel launch overheads or Python GIL interactions hurt latency under load.