Benchmarking Framework for LLM Serving Systems

This document is a technical report that presents XPerf, a benchmarking framework designed for load-testing LLM serving systems with diverse agentic AI workloads. It outlines the challenges associated with benchmarking these systems, particularly due to the nondeterministic nature of LLM outputs which can lead to unpredictable workload patterns. XPerf addresses these challenges by utilizing a fine-grained trace replay approach, enabling users to collect traces from real agentic applications and synthesize new workloads. The framework supports detailed profiling of both serving system and hardware performance metrics, allowing users to identify performance bottlenecks. The report details the features of XPerf, including application-agnostic tracing, workload synthesis, reproducible workload replay, and comprehensive performance profiling. It also discusses the empirical study conducted using eight agentic applications, demonstrating XPerf's effectiveness in accurately replaying workloads and providing valuable performance insights. The framework aims to assist developers in evaluating and optimizing LLM serving systems.