Pseudo-online Measurement of Retrieval Recall for Job Recommendations

This document is a case study that presents a pseudo-online metric designed to measure the effectiveness of the retrieval stage in a job recommendation system at Indeed. It outlines the multi-stage process of a typical recommender system, which includes retrieval, filtering, scoring, and ordering. The study discusses the limitations of traditional offline metrics, such as NDCG and recall@k, in translating to online performance metrics like click-through and conversion rates. The authors introduce an adapted version of recall@k as a business metric for evaluating retrieval strategies. The paper details the definitions of forward and backward recall, explaining how they assess the immediate and historical effectiveness of match providers. Additionally, it describes the dataset used for measuring retrieval recall and the incorporation of delayed signals to enhance the analysis. The findings indicate that using the pseudo-online measurement provides better insights into the performance of the retrieval stage compared to conventional metrics.