Evaluating WMT 2024 Metrics Shared Task Submissions

This document is a technical report that evaluates the submissions to the WMT 2024 Metrics Shared Task, focusing on the AFRIMTE challenge set designed for low-resource African languages. The report outlines the capabilities of various machine translation evaluation metrics, emphasizing the importance of language-specific adaptation and cross-lingual transfer learning. It details the performance of different metric systems, noting that larger language model sizes contribute to improved outcomes. The analysis highlights promising results for specific language pairs, including Darija-French and English-Swahili, while also identifying significant challenges for extremely low-resource languages like English-Luo. The report discusses the introduction of the AFRIMTE dataset, which addresses data scarcity and employs simplified evaluation guidelines. Furthermore, it examines the baseline metrics used in the evaluation and the submissions from participants, providing insights into the ongoing efforts to enhance machine translation metrics for African languages. The findings underscore the need for continued research and development in this area.