Microsecond-Scale Tail Latency Scheduling Techniques

This technical report presents Concord, a scheduling runtime designed to efficiently manage microsecond-scale tail latency in datacenter applications. The report outlines the challenges faced by existing systems in balancing tail latency and throughput, particularly under strict service level objectives (SLOs). It details how Concord approximates theoretically optimal scheduling policies to enhance application throughput while maintaining tight tail-latency SLOs. The evaluation of Concord shows significant improvements in throughput, achieving up to 83% greater throughput for Google’s LevelDB and up to 52% for microbenchmarks, without compromising tail latency. The report also discusses the mechanisms employed by Concord, including compiler-enforced cooperation and Join-Bounded Shortest Queue scheduling, which allow for efficient approximations of single queue and precise preemption. The findings indicate that Concord is deployable in public cloud environments and can adapt to future demands for microsecond-scale scheduling.