Explanation-Guided Metamorphic Testing of Specialized Language Models

This technical report presents an empirical study on explanation-guided metamorphic testing for task-specialized language models. The study aims to investigate whether this approach can enhance the effectiveness and validity of robustness testing compared to traditional heuristic mutation strategies. It evaluates various configurations involving attribution methods, LLM-driven mutation, and automated semantic verification across multiple datasets and model architectures. The findings reveal that explanation-guided metamorphic testing significantly outperforms heuristic strategies, generating more verified failure-inducing test cases and improving semantic validity. The study also uncovers systematic shortcut behaviors in language models, such as over-reliance on named entities and formatting cues, which can lead to inconsistent predictions. The results indicate that employing explainability-guided techniques can provide a more robust framework for testing the reliability of language models utilized in vertical AI applications, addressing the challenges associated with brittle decision boundaries and shortcut learning.