Rational Agent Framework for Visualization Evaluation

This document is a technical report that presents a rational agent framework designed for the evaluation of visualization experiments. It outlines the challenges in assessing the effectiveness of visualizations due to confounding factors in study design. The framework proposes a method for comparing the performance of behavioral agents, such as human subjects, with that of a hypothetical rational agent. This comparison aims to evaluate visualizations by bounding expected performance improvements and deconfounding errors related to information extraction and optimization. The report details how the framework can be applied both pre-experimentally to assess the design's capability to reveal significant differences between visualizations and post-experimentally to analyze behavioral performance. Additionally, it discusses the importance of establishing benchmarks for visualization performance and the potential improvements in experimental design. The report includes references to existing literature and previous studies to contextualize the proposed framework and its applications.