VAxAutoSci: Visual Analytics in the Age of Autonomous Sciencebookmark_border

Autonomous data-analysis agents must decide not only what to compute but also how to represent the result. Existing Evaluations of natural-language-to-visualization systems emphasize execution, answer correctness, and visual quality, but rarely test whether the selected chart matches the analytical task. We introduce charttype bias as an evaluation lens and audit four LLM-based visualization systems on 100 Text2Vis tasks spanning 19 chart types. Three configurations using Claude Sonnet 4.6 produce bars for 60–64% of tasks, despite bars comprising only 9% of the benchmark subset. This pattern persists across three Sonnet-based workflows, whereas Qwen3-Coder-480B exhibits a less bar-dominated profile under the same one-shot protocol, suggesting that chart selection is more strongly associated with the underlying model than with the tested harnesses. Observable Claude Code traces reveal scalarcomparison collapse, a failure mode in which an agent reduces distributional, temporal, hierarchical, or compositional structure to scalar summaries and then plots those summaries as bars. This reduction can conceal variation, trends, outliers, or structural relationships even when the resulting chart is executable and numerically accurate. Adding a dedicated chart-selection stage substantially reduces false-bar outputs and improves chart-type accuracy. These findings show that execution-based evaluation can miss systematic representational failures and motivate making chart selection an explicit, auditable stage in autonomous visualization systems. Code is available here.