Figuring out how to do good researchbookmark_border

Dashboards are widely used interfaces for data analysis, combining multiple visualizations, text, and interactive controls within a single view. While dashboard authors often structure layouts to suggest a logical consumption flow, users may interpret and navigate dashboards differently depending on the interplay between design features, analytical goals, and personal preferences. In this work, we investigate how people make sense of dashboards by examining their reading orders, i.e., the sequences in which users engage with dashboard components. We conduct a mixed-methods study with 18 dashboard authors and 16 end-users, capturing how participants design for and reason through these component transitions. Through qualitative and quantitative analyses of participant-generated flows, we outline a set of factors that influence dashboard reading order, including layout, visual saliency, semantics, functional roles, interaction, and user context. We also identify emergent reading patterns and analyze them through aggregate and variability measures, revealing where users converge and diverge in their interpretations. Finally, we discuss implications and opportunities for computational approaches that aim to automatically model, guide, or serialize dashboard consumption.

Comprehending data visualizations supports informed decision-making, yet evaluation methods, such as accuracy, response time, and eye-tracking, primarily capture outcomes or overt attention rather than the cognitive processes unfolding during visualization use. Therefore, we investigate fixation-related potentials (FRPs), EEG signals time-locked to eye fixations, as implicit markers of evaluative processing that can distinguish task-relevant information processing from overt attention allocation indexed by fixation duration alone. We provide the first empirical evaluation of FRPs as relevance markers in task-driven visualization, extending their use beyond reading and visual search. In a controlled study (N=25), participants answered questions about time-series charts while we recorded responses, eye movements, and EEG. We compared fixation durations and FRP amplitudes across task-relevant and irrelevant chart areas. Relevant fixations elicited both longer durations and larger P300 amplitudes, indicating that task relevance modulates not only overt attention but also evaluative neural processing. By disambiguating decision-related processing at the single-fixation level, FRPs complement behavioral metrics and enable a time-resolved, process-level understanding of visualization use. This provides the visualization community with a complementary method for evaluating how design choices influence not just task outcomes but the cognitive processing of visual information as it unfolds.

Visualization experiments need a set of "good" stimuli that effectively address research questions and hypotheses. Creating, managing, and deploying stimuli are often challenging, as these tasks require tremendous care. Inappropriate stimuli can make the outcome invalid or uninteresting, wasting both researchers' and participants' resources. As the speed of science increases, better support for stimuli-related tasks is essential, yet we lack a closer look at how visualization researchers deal with them. To understand the experiences of visualization experimenters and guide future improvements, we interviewed 19 visualization researchers with diverse backgrounds and experiences. Our findings describe practices and challenges across the life cycle of stimuli, from exploration and selection through shipment, deployment, and analysis. For example, stimuli management and deployment require tedious manual effort, which does not scale for experiments with many levels and complex conditioning. We also discuss both concerns and optimism around AI-assisted visualization experiment design. We conclude with future research opportunities in supporting stimuli creation, automated stimuli inspection, and experimental apparatus concerns.

A perennial research question in visualization involves identifying which visual encodings for a particular dataset are most effective for users in performing a specific task. The relative effectiveness of the different encodings are commonly identified through controlled experiments. However, designing an experiment involves making many, often ad hoc, decisions about the experimental setup such as whether to include a training module, whether to provide performance-based incentives to participants, etc. Yet, there is limited guidance on how these decisions should be made, and we do not fully understand the impact of these subjective decisions on empirical results. In this paper, we investigate the impact of one such key design decision—monetary rewards. Specifically, we ask: does providing or not providing participants with performance-based financial incentives affect the results and the conclusions that we draw from visualization studies? We conducted two crowdsourced studies investigating the impact of incentives on (i) a low-level, perceptual task (perception of correlations in scatterplots or parallel coordinate plots), and (ii) a task involving reasoning (decision-making based on a weather forecast represented as intervals or density plots). In each of these studies, we manipulate both the visual representation and the presence of incentives as between-subject conditions. We expected to find no effect of incentives on the perceptual task, but to see an effect for the decision-making task. However, we found no effect on task performance in either study. While these are results of only two studies and should be replicated, they suggest that performance-based financial incentives may not always have the intended effect on participants that we presumed, and calls for a reflection of how incentivized studies should be designed. A copy of this paper and all supplemental materials are available at https://osf.io/t8eah.

The evaluation of visual encodings is grounded in empirical user studies, including controlled comparisons of alternative designs. These studies are essential for measuring whether a visualization supports accurate human judgment, and many can provide evidence about underlying perceptual and cognitive mechanisms. However, empirical evidence alone does not enable causal prediction of interpretation errors. In this sense, evaluations are often post hoc: they assess visualization efficacy after a design has been specified, rather than predicting how human attention, uncertainty, memory, and bias may lead a viewer to accurate or erroneous interpretation. This mechanistic gap restricts the field’s ability to accumulate predictive design knowledge. Even with evidence-supported cognitive frameworks for visual decision-making, we lack general means to simulate user behavior and predict errors in silico. To bridge this gap, we demonstrate a translation of a cognitive theory of visualization interpretation into executable simulation using Active Inference: a probabilistic process theory of living systems often applied to human perception, learning, and action-taking. Active Inference agents iteratively minimize the probability of surprising observations by updating internal belief states and choosing informative actions. Simulating human interpretation of data visualization in this context, we frame chart reading as a dynamic visual search, minimizing both uncertainty and cognitive effort to reach task completion. As a foundational proof-of-concept, we engineer Active Inference agents that perform a bar-chart average-estimation task under a dual-process theory of decision-making. Crucially, our architecture aims to replicate human perceptual vulnerabilities by presenting a Fast, heuristic (Type 1) agent prone to tick salience bias and a Slow, analytic (Type 2) agent more prone to working-memory decay. Both agents yield inspectable cognitive traces, including the evolution of belief uncertainty and the chosen visual fixation sequences. By distilling these hypothesized failure mechanisms into interpretable parameters, we present this architecture as a framework for formalizing hypotheses about mechanisms of visualization interpretation. These models highlight a new role for empirical studies in visualization design: to parameterize, test, refine, or falsify decision-making simulations. Fitting these models to user data could support a shift from primarily post hoc testing toward predictive in silico validation, allowing researchers to anticipate efficacy earlier in the design process.

Recent advances in large language models (LLMs) have enabled agentic systems to translate natural-language intent into executable scientific visualization (SciVis) tasks. Despite rapid progress, the community lacks a principled and reproducible benchmark for evaluating these emerging SciVis agents in realistic, multi-step analysis settings. We present SciVisAgentBench, a comprehensive and extensible benchmark for evaluating scientific data analysis and visualization agents. Our benchmark is grounded in a structured taxonomy spanning four dimensions: application domain, data type, complexity level, and visualization operation. It currently comprises 108 expert-crafted cases covering diverse SciVis scenarios. To enable reliable assessment, we introduce a multimodal outcome-centric evaluation pipeline that combines LLM-based judging with deterministic evaluators, including image-based metrics, code checkers, rule-based verifiers, and case-specific evaluators. We also conduct a validity study with 12 SciVis experts to examine the agreement between human and LLM judges. Using this framework, we evaluate representative SciVis agents and general-purpose coding agents to establish initial baselines and reveal capability gaps. SciVisAgentBench is designed as a living benchmark to support systematic comparison, diagnose failure modes, and drive progress in agentic SciVis. The benchmark is available at https://scivisagentbench.github.io/.

Continuous colormaps are widely used to visualize scalar fields, and their quality is typically evaluated using measures such as discriminative power and uniformity. Existing measures primarily characterize the intrinsic perceptual properties of the colormap itself, largely independent of the underlying data distribution. In practice, however, user perception arises not from the colormap alone but from the visualization generated by mapping data values through the colormap. The perceptual differences that users actually experience depend jointly on the colormap and the underlying data. We propose a data-aware formulation that complements existing data-independent colormap assessment approaches. Rather than analyzing the colormap in isolation, we model the color-encoded visualization as a composite mapping from the spatial domain of the data to perceptual color space. This approach yields data-aware counterparts of established measures, including discriminative power, uniformity, and smoothness, as well as additional measures such as perceptual anisotropy and degeneracy. We validate the proposed measures against both the existing data-independent framework and empirical results from perceptual studies, extend the formulation to 2D colormaps, and demonstrate its integration into colormap optimization. An interactive system is provided for exploring colormap assessment under varying data distributions. Results demonstrate that our formulation offers a principled foundation for data-aware colormap assessment and design.