Serving LLMs at Scale: The Hidden KV Cache Advantage

KV cache is the hidden lever behind inference cost and performance. It directly impacts GPU utilization, throughput, and Time to First Token.

This session explains how KV cache works, where it resides across the memory hierarchy, how and when it spills, and how cache aware routing reshapes system design. We will also explore disaggregated prefill, why separating prefill and decode changes fleet architecture, and how these choices influence utilization at scale.

We will walk through vLLM and LMCache and do live tuning to improve performance in real time.

You will leave with a concrete performance model for serving LLMs efficiently and at scale.