IEEE VIS 2024 Content: Tasks and Telephone: Understanding Barriers to Inference due to Issues in Experiment Design

Tasks and Telephone: Understanding Barriers to Inference due to Issues in Experiment Design

Abhraneel Sarma - Northwestern University, Evanston, United States

Sheng Long - Northwestern University, Evanston, United States

Michael Correll - Northeastern University, Portland, United States

Matthew Kay - Northwestern University, Chicago, United States

Download preprint PDF
Download camera-ready PDF

Room: Bayshore I

2024-10-14T12:30:00ZGMT-0600Change your timezone on the schedule page
2024-10-14T12:30:00Z
Exemplar figure, described by caption below
The 'Telephone' framework describes two possible pathways of participants’ behaviour in experiments. In the desired pathway, a user performs the experimental task using the optimal strategy, allowing the researcher to estimate a measure of visualisation effectiveness. However, this desired pathway may not always manifest in practice. What an experiment instead might be measuring is described through the alternative pathway—a user performs what they think the task is, using a strategy which they think best supports this perceived task; the experiment is actually measuring how well the visualisation supports a user in performing their perceived task using their perceived optimal strategy.
Abstract

Empirical studies in visualisation often compare visual representations to identify the most effective visualisation for a particular visual judgement or decision making task. However, the effectiveness of a visualisation may be intrinsically related to, and difficult to distinguish from, factors such as visualisation literacy. Complicating matters further, visualisation literacy itself is not a singular intrinsic quality, but can be a result of several distinct challenges that a viewer encounters when performing a task with a visualisation. In this paper, we describe how such challenges apply to experiments that we use to evaluate visualisations, and discuss a set of considerations for designing studies in the future. Finally, we argue that aspects of the study design which are often neglected or overlooked (such as the onboarding of participants, tutorials, training etc.) can have a big role in the results of a study and can potentially impact the conclusions that the researchers can draw from the study.