Add performance profiling infrastructure

[forsyth2]

Description

This PR introduces a self-contained performance profiling workflow for zstash, intended to be run on Perlmutter.

What's added

• conda/perf.yml — a new conda environment (zstash_perf) that bundles the standard dev dependencies plus matplotlib, pandas, and numpy for performance visualization.
• tests/performance/README.md — full setup and usage guide covering how to generate performance data (via generate_performance_data.bash) and visualize results (via visualize_performance.py), including config file templates and Perlmutter path conventions.
• tests/performance/visualize/visualize_performance.py — a matplotlib-based script that produces up to four figures: an overview of the current run, a baseline comparison, and two archive-level figures (create/update and extract). Supports config-driven filtering by HPSS mode and figure selection, and outputs to either a file path or an interactive display.
• A pointer from tests/README.md to the new performance docs.

Also fixed

• Corrected the pre-commit config filename in conda/dev.yml (.pre-commit.config.yaml → .pre-commit-config.yaml).
• Updated the CI workflow filename references in conda/dev.yml to reflect the current split into build_workflow.yml and release_workflow.yml.

Notes

Performance records are currently user-specific (stored in a personal long-term directory on Perlmutter). A centralized location can be considered if more developers start running profiling regularly.

Background

#414 introduced some early performance profiling for zstash, but these changes were not ultimately included in main. However, performance is becoming an ever more important feature of zstash, as that work showed. Other issues relating to performance: #402/#424, #249.

Objectives:

• Test zstash runtimes on various combinations of variables:
  • create, update, extract
  • extract sequential vs parallel mode (especially relevant for Add Python 3.14 support #402/Fix Python 3.14 compatibility: AbsoluteLinkError, pickling errors, config propagation, HPSS hang, and print_all_contents deadlock in extract.py and parallel.py #424)
  • --hpss as "none", an HPSS path, or a Globus path

Resolves:

• [Feature]: Add averages plot to performance profiling #449

Select one: This pull request is...

• a bug fix: increment the patch version
• a small improvement: increment the minor version
• a new feature: increment the minor version
• an incompatible (non-backwards compatible) API change: increment the major version

Small Change

• To merge, I will use "Squash and merge". That is, this change should be a single commit.
• Logic: I have visually inspected the entire pull request myself.
• Pre-commit checks: All the pre-commits checks have passed.

[forsyth2]

Action items:

• Actually run these scripts and confirm visualizations and workflows are working properly
• Run this code on main and on Add Python 3.14 support #402 and on Fix Python 3.14 compatibility: AbsoluteLinkError, pickling errors, config propagation, HPSS hang, and print_all_contents deadlock in extract.py and parallel.py #424 to compare the effects of parallelism changes needed for Python 3.14

[forsyth2]

Results from running on 2026-04-02:

Notes:

• Globus runtimes are definitely pretty variable. Notice the two wall clock times for zstash create with Globus differ substantially for instance.
• Next time we run this, we can run against these results to get some comparison between runs.
• If those look ok, we can probably merge this and try to do weekly performance testing on zstash's main branch.

Setup

cd ~/ez/zstash
git status
# On branch force-fork-parallel
# nothing to commit, working tree clean
git checkout add-performance-profiling
git log --oneline | head -n 10
# 9d6d15f Fixes made comparing pr402 and pr424
# a6cbc12 Update parameters
# 7467b68 Improve plots
# def78ac Fixes to generate plots
# ee61184 Add ability to configure hpss options
# c3ead41 Add regression testing
# 8662955 Send output to web server
# 8c3c555 Apply changes from Claude
# ee2dda7 Add performance profiling as a standard zstash test
# c14b8ee Add AGENTS.md (#423)

# Good, matches https://github.com/E3SM-Project/zstash/pull/427/commits
nersc_conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zstash-pr427-performance-profile-20260402
conda activate zstash-pr427-performance-profile-20260402
pre-commit run --all-files
python -m pip install .
cd tests/performance
emacs generate_performance_data.bash # Edit parameters
git diff # Check diff
./generate_performance_data.bash
# ~2-3 hours to run, note there is the manual step to paste an auth code

# [SUCCESS] All tests completed. Results saved to: /pscratch/sd/f/forsyth/zstash_performance/performance_20260402/results.csv
# [INFO] Now edit IO paths and run: python visualize_performance.py

emacs visualize_performance.py # Edit parameters
git diff # Check diff
pre-commit run --all-files
git add -A
python visualize_performance.py
# Figure 1 (overview) saved to: /global/cfs/cdirs/e3sm/www/forsyth/zstash_performance/performance__20260402_pr427.png
#   Accessible at: https://portal.nersc.gov/cfs/e3sm/forsyth/zstash_performance/performance__20260402_pr427.png

[forsyth2]

Rebased off the latest main.

Latest commit 1cb3eb8 is Claude's resolution of:

Looking at our 2026-04-14 profiling, the single-branch plots look ok, but the branch comparison plots have some issues:

• the "baseline" bars appear to the left of the "current" bars. It would make more sense for them to be to the right. Currently, a taller bar on the right is good, but visually it looks like an increase in runtime!
• For the "Extract: Sequential vs Parallel" plot, let's look at an example: "create: build/ update: run/" has 12 bars -- 3 groups of 4, a blue group for "No HPSS", an orange group for "Direct HPSS", and a green group for "Globus". Now let's dive into one of those groups, say the blue group. The 4 bars are "sequential current, sequential baseline, parallel current, parallel baseline". Based on the legend, we'd expect those to be differentiated by "solid blue, transparent hatched blue, solid double hatched blue, transparent blue that is somehow both double hatched for parallel and single hatched for baseline". What we actually see is "solid blue, transparent blue, solid double hatched blue, transparent double hatched blue"

For reference, previous plots:

• 2026-04-14 profiling -- this was to profile main during the testing period for E3SM Unified 1.13.0
• 2026-04-06 profiling -- this was to profile the refactoring of Refactor to resolve tars not deleting when --non-blocking is set #416.

Remaining action items:

• Visual inspection of latest commit
• Rerun plotting code on already existing profiling data
• Try to automate keeping a history -- i.e., we should have a log of previous performance data, so we can review changes over time.

[forsyth2]

Rerun plotting code on already existing profiling data

Setup

nersc_conda # Activate conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zstash-pr427-20260603
conda activate zstash-pr427-20260603
pre-commit run --all-files

# Make changes to tests/performance/visualize_performance.py

python -m pip install .
cd tests/performance/
python visualize_performance.py
# Figure 1 (overview) saved to: /global/cfs/cdirs/e3sm/www/forsyth/zstash_performance/performance_pr427_20260603.png
#   Accessible at: https://portal.nersc.gov/cfs/e3sm/forsyth/zstash_performance/performance_pr427_20260603.png
# Figure 2 (baseline comparison) saved to: /global/cfs/cdirs/e3sm/www/forsyth/zstash_performance/performance_pr427_20260603_vs_baseline.png
#   Accessible at: https://portal.nersc.gov/cfs/e3sm/forsyth/zstash_performance/performance_pr427_20260603_vs_baseline.png

Fig1: https://portal.nersc.gov/cfs/e3sm/forsyth/zstash_performance/performance_pr427_20260603.png

Fig2: https://portal.nersc.gov/cfs/e3sm/forsyth/zstash_performance/performance_pr427_20260603_vs_baseline.png

✅ "baseline" bars are now to to the left of the "current" bars
✅ the 4 bars of each hpss type in the final extract plot of Fig2 are now distinguishable.

[chengzhuzhang]

@forsyth2 would it possible to do the performance profiling for zstash check ? There have been complaints about zstash check being slow. Would be useful to profile its performance and find potential solution to speed it up.

[Copilot]

Pull request overview

Adds a new, documented performance profiling workflow under tests/performance/ to
measure and visualize zstash runtime across operation modes and HPSS settings,
and updates the dev Conda environment to support the visualization tooling.

Changes:

• Add a Perlmutter-oriented performance data generation script that records
  zstash create/update/extract runtimes into a CSV.
• Add a Python visualization script to plot performance results and optionally
  compare against a baseline run.
• Document the performance profiling workflow and link it from the main tests
  README; add plotting/data dependencies to the dev Conda environment.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
tests/README.md	Adds a pointer to the new performance profiling docs.
tests/performance/README.md	Documents how to generate and visualize performance CSVs.
tests/performance/generate_performance_data.bash	Generates performance CSV by running a matrix of zstash operations/modes.
tests/performance/visualize_performance.py	Plots performance results and optional current-vs-baseline comparisons.
conda/dev.yml	Adds matplotlib-base, pandas, and numpy to support visualization.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[forsyth2]

@chengzhuzhang

There have been complaints about zstash check being slow.

I'm assuming you mean new/recent reports? I know we resolved the high-res team's issues by making more use of the --tars option.

would it possible to do the performance profiling for zstash check ?

I think that's good idea, but can we do it as a separate pull request? I think this PR is pretty close to merging, and I'd like to have something on main. Furthermore, debugging zstash check's slowness may give indicators of how best to profile it.

For what it's worth, Claude's assessment:

---

Looking at the code, adding zstash check would be a moderately sized addition — not trivial, but not a full rewrite either. Here's my assessment:

Arguments for a separate PR:

• The current PR already has a coherent, complete scope (create/update/extract profiling with full visualization). Shipping it now lets you get feedback and merge a working system before extending it.
• check has meaningfully different semantics — it validates archive integrity rather than measuring transfer/compression performance, so it may warrant its own visualization considerations (e.g., does it even make sense to compare check across HPSS modes the same way?).
• If something goes wrong with the check implementation, it won't block merging the core profiling work.

Arguments for doing it now:

• The plumbing is all there. Adding check to the bash script is straightforward: a run_check() function, a record_result call, and hooking it into the per-test loop after extract. The Python side just needs check added to OP_ORDER/OP_TITLES/OP_DIR_COL (or a new extract-style handler if it operates on the full archive).
• You'd avoid a second round of review setup, CI runs, and README updates for what is mechanically similar work.

My recommendation: separate PR. The current PR is clean and self-contained. check deserves a moment to think through what its x-axis and grouping should be — it runs on an existing archive, so it's more like extract than create, but it doesn't use workers in the same way. Rushing that design into this PR risks muddying both the code and the review. Merge what you have, then open a focused check PR once you've decided how it fits into the visualization layout.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

[forsyth2]

ea14728 addresses #449 by adding plots of performance records over time.

A few things to address:

• Ignore outlier datapoints to keep y-axis constrained.
• Make similar plots for extract.

Prototype:

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 9 comments.

[forsyth2]

This is specifically for me to have a ready-to-go cfg. The blank slate cfg is perf.cfg.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

[forsyth2]

Latest plots can be found here.

#450 still implements performance for zstash check, but I did add the long-term record plots to resolve #449. It should be noted that the data points are from multiple branches/PRs over time; the records to include can be customized in the profiling cfg.

[forsyth2]

High-level visual inspection looks fine to me. As noted in the previous comment, the latest plots also look alright and #450 will remain open to add similar performance profiling for zstash check.

Note that nothing in this PR is user facing (i.e., it's for developers only).

This should be good to merge.