Talk to my agentbookmark_border

Machine learning systems often benefit from human involvement, but human-in-the-loop support is not free: teams may need to build specialized interfaces, analysts must spend time interacting with them, and some interactions trigger costly retraining or re-optimization. The central design question is therefore not simply whether interaction helps, but when it is worth the added effort relative to other interventions, such as improving the data, revising the model or training specification, or relying on automated workflows alone. In this paper, we build upon Van Wijk's foundational economic model of visualization, extend it by explicitly incorporating machine learning components, and present a cost-benefit framework for reasoning about these choices in interactive visual machine learning systems. We identify measurable quantities that govern practical trade-offs, including analyst time, interaction frequency, training latency, model improvement, downstream operational savings, and failure-prevention value. We organize these quantities into workflow-specific models for three common intervention targets: data iteration, model or specification iteration, and evaluating and understanding model behavior. We then instantiate the framework on representative systems to show how it supports break-even analysis, latency bounds, and stopping criteria for interactive analysis. The contribution is a practical decision framework for reasoning about when interactive visual tooling is likely to be justified, which intervention can provide value, and where human effort can be allocated more effectively across IVML workflows.

Large language model (LLM) agents enable natural language interaction for scientific visualization (SciVis). Still, prior systems have essentially prioritized autonomy over human analytical control, thereby limiting transparency and human oversight. We present HiLSVA, a human-in-the-loop agentic system that supports mixed-initiative SciVis workflows. HiLSVA integrates a plan-first multi-agent architecture with explicit human oversight, stepwise provenance tracking, and learn-at-test-time adaptation from user feedback. The system supports fluid handoff between humans and agents through both natural language and direct manipulation of visualizations, while sandboxed execution ensures safe, reproducible workflows. In doing so, HiLSVA reframes agentic SciVis as a collaborative process that augments, rather than replaces, human analytical reasoning. We evaluate HiLSVA through representative case studies and a controlled user study with twelve participants of varying expertise across multiple autonomy settings. Results show that mixed-initiative interaction improves task completion, user control, and workflow transparency across different levels of user expertise, while revealing a tradeoff between execution efficiency and human oversight. These findings highlight the importance of human-centered design in agentic SciVis and guide the development of future collaborative visualization systems. We encourage readers to explore our demo video, case studies, and source code at https://hilsva.github.io/.

Large language models (LLMs) produce high-dimensional embeddings that encode rich linguistic structure. We leverage mapper graphs, a technique from topological data analysis, to examine the organization of these embedding spaces, where nodes represent clusters of embeddings and edges capture their overlap. Although mapper graphs can reveal meaningful patterns, interpreting the linguistic properties they encode remains time-consuming and labor-intensive. We introduce Explainable Mapper, a framework for semi-automatic annotation and systematic exploration of mapper graphs that enables fine-grained analysis of embedding spaces. Explainable Mapper defines a taxonomy of explorable elements, including nodes, edges, paths, components, and trajectories, and integrates two types of LLM-based agents: an Explanation Agent that generates candidate interpretations of mapper elements, and a Verification Agent that evaluates their robustness using perturbation-based strategies. Together, these agents enable semi-automated and stable interpretation of topological structures. We instantiate Explainable Mapper within an interactive visual analytics workspace that combines agent-driven analysis with user-guided exploration, supporting diverse human--AI workflows for scalable and interpretable investigation. We evaluate our approach through case studies on embeddings from multiple transformer models and datasets, as well as an expert user study. Results show that Explainable Mapper reproduces established linguistic patterns while uncovering novel insights into LLM embedding spaces. This work extends mapper-based embedding analysis by integrating LLM-assisted explanation generation with perturbation-based verification, enabling scalable and systematic interpretation of high-dimensional embedding spaces.

Data validation is a crucial step in data analytics workflows that assesses and ensures the reliability of data flowing into analytical processes. One common approach to data validation involves defining validation rules, which provide explicit constraints and conditions that data must satisfy. However, creating accurate and effective validation rules remains challenging for many practitioners. This challenge stems from the need for practitioners to understand both data structures and their domain-specific semantic relationships. Recent studies have proposed automated approaches to generate validation rules by deriving patterns from data properties. However, these approaches generate rules with limited interpretability and lack support for rule verification and modification, making the rules difficult to understand and adapt. To address these limitations in current validation rule authoring approaches, we present RuleScope, an interactive system for authoring data validation rules through semantic-aware rule generation, visualization, and refinement. RuleScope employs an LLM-based workflow to generate interpretable rules by analyzing data semantics and incorporating domain knowledge. To facilitate rule comprehension, we design a matrix-based visualization that helps users understand rules and analyze validation results. Additionally, RuleScope enables users to interactively refine rules. We evaluate the LLM-based workflow through model evaluation on datasets from different domains and assess RuleScope's usability and effectiveness through two case studies and a user study.

Real-world decision making often unfolds in fluid, uncertain, and ill-defined contexts where objectives shift, data are incomplete, and non-quantifiable factors such as social values, ethics, and institutional constraints play critical roles. Conventional AI and decision-support systems assume fixed criteria and stable data, leaving these contexts underserved. Building on an interdisciplinary definition of decision making attentive to its ill-defined forms, we introduce DARE, an explainable AI and visualization framework that complements the FAIR data principles with the DARE principles: Deliberation, Agency, Resilience, and Empathy, which emphasize dialogue, human control, adaptability, and human sensitivity in design. DARE conceptualizes decision making as an iterative alignment of human-defined criteria with algorithmic representations through which decision structure gradually emerges. We revisit existing AI paradigms through this lens and illustrate how weak supervision and concept-based modeling exemplify this process by connecting heuristic human reasoning to interpretable model concepts. Input visualization serves as the expressive layer that captures evolving, qualitative, and uncertain reasoning through interaction, allowing humans to externalize and refine decision logic before formalization. Explainability in DARE arises not from post-hoc justification but from the continuous visibility of how human and algorithmic reasoning co-develop. Uncertainty is treated as an inherent dimension of deliberation, something to represent, navigate, and learn from within the decision process, while human and algorithmic heuristics are regarded not as truths or biases but as evolving hypotheses to examine and refine through interaction. Together, these elements support human–AI decision making that remains transparent, adaptable, and grounded in human judgment across value-laden and ill-defined contexts.

Coordinating human analysts with autonomous AI agents faces the same challenges as human-to-human collaboration: sharing intermediate results, avoiding conflicts, and maintaining group awareness. Current tools rely on unstructured messaging or single-threaded chatbot interaction, which lack the structure to track evolving hypotheses or link claims to evidence. We propose agentic social data analysis, a collaboration paradigm extending social data analysis with a shared coordination feed modeled on the content timeline in social media services. We instantiate this concept in TREADSTONE, a platform where human and AI agents asynchronously post, link, and contest analytical claims via threaded messages within a shared feed. By allowing agents to proactively broadcast hypotheses and enabling users to steer the analysis through lightweight curation, Treadstone seeks to balance machine autonomy with human analytical control. A qualitative user study shows that Treadstone fosters collaboration while preserving human analytical agency, in contrast to the solitary experience of conventional chatbot interaction.

Knowledge graphs (KGs) capture rich, multi-relational data, but comparing entities, relationships, and local structures remains challenging with traditional graph visualizations. We present NEXO, a task-driven system for comparative KG exploration that combines natural language interaction with adaptive, coordinated visualizations. The system translates free-form queries into structured comparison tasks and supports value comparison, path comparison, and neighborhood exploration across one-to-one, one-to-many, and many-to-many comparison structures. Rather than exposing users to the full graph, NEXO generates task-specific views that combine structural context with concise summaries to support efficient comparison.We evaluate the system through a user study with eye tracking to assess task performance and attention allocation across coordinated views. Participants completed tasks with high accuracy and minimal query reformulation. Gaze patterns reveal task-dependent attention strategies, with users relying more on graph views for structural exploration and summary views for pattern identification and comparison. These findings demonstrate the effectiveness of combining natural language interaction with task-adaptive visualizations for comparative KG exploration.