Enabling Spatially Fine-Grained DVFS in Neural Processing Units

This technical report presents the development of eNPU, a hardware and software solution designed to enable spatially fine-grained dynamic voltage and frequency scaling (DVFS) in neural processing units (NPUs) for energy-efficient large language model (LLM) serving. The report outlines the challenges associated with implementing component-level DVFS, including the need for architectural support and the complexities of compiler scheduling. eNPU addresses these challenges by refactoring the NPU core pipeline to create separate voltage and frequency domains for different components, thus allowing for optimized energy consumption. The report details the implementation of eNPU on an open-source NPU core and evaluates its performance using a production-level NPU simulator. Results indicate that eNPU can reduce energy consumption by 25.8% to 35.2% while maintaining strict service-level objective (SLO) guarantees. The findings highlight the potential for significant energy savings in NPU operations through the proposed DVFS approach.