Re-build `access-om3/pr236-2` due to deletion from Retracting A Release

[chrisb13]

Been trying some semi-automagical-load-balancing. Manodeep suggested I focus on this PR. I was able to do a few runs successfully but the ones today have failed when ones from a few days ago worked.

Here's the (AI assisted) theory..

Test01/02/03 ran while the old ESMF spack hash still existed on disk. The binary (access-om3-MOM6-CICE6 from access-om3/pr236-2) was compiled against ESMF at:
/g/data/vk83/apps/spack/1.1/release/linux-x86_64_v4/esmf-8.9.1-u56mitgtfbi2qlj3x7lw7ofxj7lw4b2l/lib/libesmf.so

This got removed overnight as part of this #model-release > Retracting A Release @ 💬

@manodeep got me to check the shared libraries for the pre-release build:

[cyb561.gadi-login-05: dev-MC_25km_jra_iaf+wombatlite_test06_26nodes_216nonOCN]$ ldd which access-om3-MOM6-CICE6 linux-vdso.so.1 (0x00007fffff9db000) /g/data/vk83/apps/spack/1.1/release/linux-x86_64_v4/esmf-8.9.1-u56mitgtfbi2qlj3x7lw7ofxj7lw4b2l/lib/libesmf.so => not found libpiof.so => not found libpioc.so => not found

So I think that's consistent with Claude suggestion but I don't have a timemachine to go back run this command when it was working. So @CodeGat has suggested re-deploying the Pre-release build.

Described:

• https://access-nri.zulipchat.com/#narrow/dm/784078,784233,784257,784275,784286,784290-group/near/602014248 (private DM)

[CodeGat]

Thanks @chrisb13. I'm currently trying to resurrect those packages based on the lockfiles from the retracted builds in https://github.com/ACCESS-NRI/ACCESS-OM3/releases/tag/2026.05.001 and https://github.com/ACCESS-NRI/ACCESS-OM3/releases/tag/2026.05.000. When that is done, we will be able to test the previously-not-working builds. Preliminary thoughts are that it is rebuilding the exact hash of esmf that is missing, so you might be able to use the old builds after all!

[chrisb13]

@CodeGat sounds good... Seems my approach failed

==> [2026-06-11-16:30:42.111330] InstallError: Terminating after first install failure: ProcessError: Command exited with status 2:
    '/g/data/vk83/prerelease/apps/spack/1.1/release/linux-x86_64_v4/gmake-4.4.1-3sr5xy6gpvmfkr4ecptiqtbmcrqnczuk/bin/make' '-j4'
==> [2026-06-11-16:30:42.111415] Error: Terminating after first install failure: ProcessError: Command exited with status 2:
    '/g/data/vk83/prerelease/apps/spack/1.1/release/linux-x86_64_v4/gmake-4.4.1-3sr5xy6gpvmfkr4ecptiqtbmcrqnczuk/bin/make' '-j4'
Traceback (most recent call last):
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/installer.py", line 2362, in complete_task
    self._complete_task(task, install_status)
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/installer.py", line 1991, in _complete_task
    rc = task.complete()
         ^^^^^^^^^^^^^^^
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/installer.py", line 1376, in complete
    self.fail(e)
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/installer.py", line 1313, in fail
    raise inner_exception
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/installer.py", line 1365, in complete
    spack.package_base.PackageBase._verbose = self.process_handle.complete()
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/build_environment.py", line 1296, in complete
    return complete_build_process(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/build_environment.py", line 1465, in complete_build_process
    raise child_result
spack.build_environment.ChildError: ProcessError: Command exited with status 2:
    '/g/data/vk83/prerelease/apps/spack/1.1/release/linux-x86_64_v4/gmake-4.4.1-3sr5xy6gpvmfkr4ecptiqtbmcrqnczuk/bin/make' '-j4'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/main.py", line 1124, in main
    return _main(argv)
           ^^^^^^^^^^^
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/main.py", line 1076, in _main
    return finish_parse_and_run(parser, cmd_name, args, env_format_error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/main.py", line 1107, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/main.py", line 651, in _invoke_command
    return_val = command(parser, args)
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/cmd/install.py", line 332, in install
    install_with_active_env(env, args, install_kwargs, reporter)
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/cmd/install.py", line 405, in install_with_active_env
    env.install_specs(specs_to_install, **install_kwargs)
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/environment/environment.py", line 1990, in install_specs
    builder.install()
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/installer.py", line 2446, in install
    self._install()
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/installer.py", line 2493, in _install
    failure = self.complete_task(task, install_status)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/vk83/prerelease/apps/spack/1.1/spack/lib/spack/spack/installer.py", line 2419, in complete_task
    raise spack.error.InstallError(f"{_FAIL_FAST_ERR}: {str(exc)}", pkg=pkg) from exc
spack.error.InstallError: Terminating after first install failure: ProcessError: Command exited with status 2:
    '/g/data/vk83/prerelease/apps/spack/1.1/release/linux-x86_64_v4/gmake-4.4.1-3sr5xy6gpvmfkr4ecptiqtbmcrqnczuk/bin/make' '-j4'
==> [2026-06-11-16:07:57.192876] Installing esmf-8.9.1-u56mitgtfbi2qlj3x7lw7ofxj7lw4b2l [32/39]
==> [2026-06-11-16:30:40.044844] Updating view at /g/data/vk83/prerelease/apps/spack/1.1/environments/access-om3-pr240-1/.spack-env/view
Error: Process completed with exit code 1.

Here's the post

[CodeGat]

The packages that were deleted have been restored now via the lockfiles, so the downstream builds (AKA Prereleases) should work again without any intervention - give em a go!

[chrisb13]

Confirming that this resolved the problem (both the old prerelease builds and this new one)

Thanks @CodeGat !