Using LLMs for Measurement in Social Science

This document is a seminar talk that discusses the application of large language models (LLMs) for measuring similarity in semantic spaces. It reports on the use of fine-tuned BERT and pre-trained instruction-tuned LLMs, such as GPT-4 and Meta Llama 3, to assess the typicality of text documents across various concepts, including political tweets and literary genres. The talk also includes a systematic comparison of the performance of recent LLMs for these tasks and outlines a strategy for selecting appropriate models based on specific research objectives and constraints. The content is derived from recent papers that explore the positioning of political texts in ideological spaces and the semantics of concepts using LLMs. The speaker, Gaël Le Mens, is a professor at Universitat Pompeu Fabra and discusses how learning agents can develop biases based on the information they encounter, which has implications for improving decision-making in organizations and AI technologies.