Overview
The design is implemented and evaluated on real traces and shows strong gains; production risk comes from TTFT estimation sensitivity and hardware differences.
Citations0
Evidence Strength0.80
Confidence0.85
Risk Signals9
Trust Signals
Findings with numeric evidence: 6/6
Findings with evidence refs: 6/6
Results with explicit delta: 6/6
Reproducibility
Status: Partial assets available
Open source: Partial
At A Glance
Cost impact: 70%
Production readiness: 70%
Novelty: 60%
Why It Matters For Business
DualMap can serve more latency-sensitive requests and lower per-request compute cost by combining cache reuse with balanced load, improving throughput and reducing tail latency under real skewed workloads.
Who Should Care
Summary TLDR
DualMap is a scheduling layer for distributed LLM serving that maps each request to two candidate instances (two independent hashes) and picks between them using SLO-aware rules. It preserves KV-cache reuse (cache affinity) while using the 'power of two choices' to balance load. Key techniques: adaptive prefix length, SLO-aware routing (prefer cache reuse until TTFT risk), hotspot-aware rebalancing (migrate within the two candidates), and dual-hash-ring scaling for low-disruption elasticity. On real traces, DualMap raises effective request capacity up to 2.25× and cuts P50/P90 TTFT substantially versus baselines.
Problem Statement
In distributed LLM serving, routing for KV-cache reuse (cache affinity) tends to concentrate popular prefixes on a few nodes and creates hotspots. Pure load-based routing scatters prefixes and forces recomputation. Existing single-mapping schedulers trade one objective for the other and cannot guarantee both low time-to-first-token (TTFT) and even load under real, skewed workloads.
Main Contribution
DualMap: a dual-mapping scheduler that assigns two prefix-bound candidate instances per request and picks one at dispatch time.
SLO-aware routing: prefer cache reuse until expected TTFT would exceed the SLO, then switch to load-aware choice.
Key Findings
DualMap increases effective request capacity up to 2.25× versus state-of-the-art schedulers on evaluated traces.
On the Tool&Agent trace DualMap raised effective request capacity by as much as 125% and goodput by 16.7–48% versus the best baseline.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Effective Request Capacity | up to 2.25× vs baselines | state-of-the-art schedulers (e.g., Mooncake/Preble) | up to 2.25× | real-world Mooncake traces | Abstract, §4.2 | Abstract; §4.2; Figures 3 |
| Goodput | +16.7%–48% (Tool&Agent); +14.3%–40% (Conversation) | best baseline per trace | range depends on workload and model | Tool&Agent and Conversation | §4.2, Figures 3a–3d | §4.2; Figures 3 |
What To Try In 7 Days
Measure prefix-sharing in your traces (shared-prefix rate) to estimate cache opportunities.
Prototype dual-hash mapping for prefixes and track cache hit rate vs queue lengths.
Add an SLO threshold: compute a pending-token TTFT threshold and switch to less-loaded candidate when exceeded.
Optimization Features
Token Efficiency
Infra Optimization
System Optimization
Inference Optimization
Reproducibility
Code URLs
Risks & Boundaries
Limitations
Relies on accurate TTFT estimation; misestimation under heavy decode bottlenecks can hurt decisions (§A.7).
Evaluations use two Mooncake traces and specific Ascend NPU hardware; results may vary on different workloads or hardware.
When Not To Use
Workloads with almost no prefix sharing (no KV-cache benefits).
Very small clusters where two-choice gains are minimal.
Failure Modes
Incorrect TTFT estimates cause wrong switches from cache-aware to load-aware routing, increasing recomputation.
Frequent migrations if thresholds are tuned too aggressively, reducing cache reuse and increasing overhead.