Statistical Interpretation of Term Specificity in Retrieval

This technical report discusses the statistical interpretation of term specificity and its implications for document retrieval. It outlines the relationship between the exhaustivity of document descriptions and the specificity of index terms, suggesting that specificity should be viewed as a function of term use rather than solely its meaning. The report presents findings from experiments conducted on three test collections, demonstrating that variations in term specificity can significantly affect retrieval performance. It argues for the weighting of terms based on their collection frequency, positing that matches on less frequent, more specific terms yield greater value than those on more common terms. The report further explains the concepts of exhaustivity and specificity, detailing how they influence the effectiveness of indexing vocabularies. It emphasizes the need for a balance in term distribution to optimize retrieval outcomes, highlighting the challenges posed by frequently used terms that may lack discriminative power.