Scaling AI Inference with KV Cache Offloading

This white paper provides a detailed analysis of the performance impact and workload characteristics associated with key-value (KV) cache offloading in large language models (LLMs). It outlines the importance of KV cache in optimizing inference workloads, particularly in agentic AI environments where multi-turn interactions are common. The document explains the two primary stages of LLM inference: the prefill stage, where key-value pairs are generated and stored, and the decode stage, where these pairs are reused to produce subsequent tokens. The paper emphasizes the challenges of KV cache persistence in multi-node environments, highlighting how offloading to external resources can mitigate latency issues during GPU migrations. It also discusses the characteristics of I/O patterns when managing KV cache, noting the predominance of read operations during session migrations. The findings underscore the necessity of efficient KV cache management for maintaining performance and scalability in modern AI systems.