NL2SciVis

A benchmark for evaluating AI agents' ability to generate scientific visualizations from natural language requests. Agents must produce ParaView Python scripts that create correct visualizations.

What is NL2SciVis?

NL2SciVis evaluates agents on atomic operations - single-intent visualization tasks with strict preconditions. Agents receive natural language prompts along with dataset metadata and a pre-configured ParaView session where prerequisites (loaded dataset, active render view, valid pipeline state) are already satisfied, allowing them to focus solely on the requested operation.

Example atomic operations:

• Setup: "Create a contour of the pressure field at value 101325" (creates new Contour filter)
• Adaptation: "Set camera position to (10, 10, 10) looking at origin" (modifies existing camera)
• Reporting: "Calculate the maximum temperature in the dataset" (computes statistic)

Agents must generate working ParaView Python scripts that produce correct visualizations for these single-intent tasks.

By testing fundamental building blocks independently with strict preconditions, we can systematically identify which capabilities are robust and enable deterministic evaluation without clarifying questions.

Key Features:

• ParaView Integration: Industry-standard scientific visualization platform
• Binary Gate Evaluation: Execution + Technique gates for deterministic pass/fail assessment
• Atomic Operation Taxonomy: Systematic decomposition into Setup, Adaptation, and Reporting operations

Quick Start

For complete installation and setup instructions, see DEV.

Requirements: Python 3.12+, uv package manager, ParaView

Execution Environments: NL2SciVis supports configurable execution environments (Dask local, ALCF) via hive configuration in configs/.

Core Concepts

Atomic Operations

• Trial: Complete agent evaluation across all/selected benchmark tasks
• Task: Single atomic operation (Setup/Adaptation/Reporting) with strict preconditions
• Operation Types:
  • Setup: Load data, create initial visualizations (contour, slice, volume render, streamlines, glyphs)
  • Adaptation: Modify existing visualizations (camera adjust, colormap change, parameter tuning)
  • Reporting: Compute statistics (min/max, mean, integral, surface area, volume)