Comprehensive Evaluation of Predictive Algorithms

This research article presents a comprehensive evaluation of predictive algorithms using statistical decision theory (SDT) as a replacement for traditional K-fold and Common Task Framework validation methods in machine learning (ML) research. The authors, Jeff Dominitz and Charles F. Manski, argue that SDT offers a formal frequentist framework for conducting out-of-sample (OOS) evaluations across various training samples and populations. They emphasize the necessity for practitioners to acknowledge the potential discrepancies between past and future data when developing predictive algorithms. The paper discusses the application of SDT in clinical decision-making, particularly in predicting patient illness. It highlights the computational challenges associated with implementing SDT while calling for an expansion of its practical applications within the fields of ML, econometrics, and statistics. The authors critique the prevalent reliance on OOS evaluation methods in ML, noting that they may not yield generalizable insights and can lead to misguided applications in critical areas such as healthcare and law enforcement.