Analysis of Expert Finding System Evaluations

This technical report presents an analysis of expert finding systems, focusing on the impact of automated recommendations on the evaluation of these systems. The study highlights the challenges organizations face in identifying experts due to the complexity of internal knowledge distribution. It details the process of expert retrieval, which relies on comprehensive annotations of expertise, often difficult to obtain. The report examines how system-validated annotations can lead to overestimated performance in traditional retrieval models and affect comparisons with neural methods. It proposes constraints to improve the annotation process, aiming to mitigate biases and enhance the utility of annotation suggestions. The findings are intended to inform the creation and selection of benchmarks for expert finding, ensuring that evaluations remain meaningful and representative. The analysis is grounded in the TU Expert Collection, which provides a framework for understanding the characteristics of different annotation methods and their implications for expert finding tasks.