Replacing Cross-Validation with Interrogation Methodology

This document is a technical report that introduces a novel approach for evaluating the reliability of prediction routines in machine learning, specifically addressing the limitations of traditional cross-validation methods. The authors propose an interrogation-based method that assesses model calibration without the need for validation samples. The report details the process of decomposing prediction logic into linear, nonlinear, pairwise, and high-order interaction components, allowing for a comprehensive evaluation of model performance. It outlines the methodology used in simulations to demonstrate the effectiveness of the interrogation approach, which successfully identifies optimal model calibrations. The findings indicate that this method can effectively differentiate between underfitting and overfitting in predictive models. Additionally, the report discusses the application of this methodology in real-world scenarios, such as predicting currency prices, and suggests potential extensions for further research. The overall goal is to enhance model transparency and reliability in predictive analytics.