Some builds are too hungry for login nodes

[dougiesquire]

We've recently been experiencing builds failing due to using too much resource (I think memory) for the Gadi login node.

For example, see this ACCESS-OM3 prerelease which has particularly high memory usage due to lto flags. This failed with

ifx: error #10106: Fatal error in /apps/intel-tools/.packages/2025.2.0.575/compiler/2025.2/bin/compiler/ld.lld, terminated by kill signal

while building access3. I can reproduce the failure using my own spack instance on a login node, but can build successfully on a Sapphire-Rapids compute node (104 cpus, 496G mem).

Some options have been floated on Zulip for preventing these failures:

• Have the option to run particularly meaty builds on compute nodes. One complication of this is that Gadi compute nodes do not have network access, so sources would need to be mirrored on the login node first. I've typically done this in the past with something like:

  [login-node] $ spack env activate <env>
  [login-node] $ spack concretize -f --fresh
  [login-node] $ spack mirror create -d sources -a
  
  <Start interactive job>
  
  [compute-node] $ spack env activate <env>
  [compute-node] $ spack install
  

• From @aidanheerdegen

  We could ask NCI if they could increase the allowed memory for the service user ... it would be a lot simpler.

• From @aidanheerdegen

  We might be able to have a simpler qsub version of a build by using a persistent session tunnel to access the source code, so making it more transparent to spack.

• From Angus Gibson

  As far as I can tell there's a 4GB limit of resident memory (I'm not sure if per session or per process), and after that it'll start spilling into swap. There's only 16GB swap and that's shared across all users of the node. On e.g. gadi-login-06 it's currently almost all used:

  $ free -h
                total        used        free      shared  buff/cache   available
  Mem:          250Gi       108Gi        36Gi        13Gi       104Gi       126Gi
  Swap:          15Gi        15Gi       245M
  

  In fact, my test got killed after allocating only 2.8GB there...But on gadi-login-03 there's about 11Gi free and I could get a lot more. So a lot of non-determinism around where the builder lands, particularly if LTO is a bit hungry (or just a complex compile...)
  ...
  Seems to dip into swap at around 4GB combined, it's probably actually per-user because the cgroup is /sys/fs/cgroup/memory/user.slice/user-$(id).slice/memory.limit_in_bytes (= 4294967296)

  We could potentially try to target nodes that aren't under heavy load?

[aidanheerdegen]

A user based limit is a problem if we're running lots of builds through a service user, but it is also a use-case that makes a good reason to ask for a change in the process limits.

One user is doing a lot of work for many other users, so it is legitimate it might need more resources.

There are two major issues with using compute nodes for this purpose: lack of internet access and non-deterministic queue wait times.

(They also consume compute resources, but that is not a massive problem, and certainly not a deal-breaker)

Another option might be to spin up our own Nirin cloud nodes just for this purpose. They would consume a fair amount of resources as we'd need to size them with contention in mind. This would also take some time to get provisioned, configured, tested and deployed.

[CodeGat]

You'd noted @dougiesquire that we'd "probably not always" want to use compute nodes, but if we moved to that way of building, would there be any reason not to?

@aidanheerdegen I don't know too much about the Nirin nodes, but is that architecture similar to what we'd be running the executables on?
As for compute nodes, the queue wait times could be annoying, but also, quicker builds.

[dougiesquire]

You'd noted @dougiesquire that we'd "probably not always" want to use compute nodes, but if we moved to that way of building, would there be any reason not to?

Oh only for this reason:

(They also consume compute resources, but that is not a massive problem, and certainly not a deal-breaker)

[aidanheerdegen]

As for compute nodes, the queue wait times could be annoying, but also, quicker builds.

How much quicker?

Also the assumption they wouldn't use a lot of resources is just that, an assumption. Given the issues with queue waits I think we'd pretty quickly end up wanting to use express queues. And they can get quite expensive quite quickly. Will we have problems with cross compiling depending on which queue we opt for?

I think outside internet access is problematic. Not from a technical point of view, but given they are isolated from outside connection for a reason, it may not be well received.

By far the easiest and quickest solution is to ask if we can get the memory limits raised for the service user.

[aidanheerdegen]

I don't know too much about the Nirin nodes, but is that architecture similar to what we'd be running the executables on?

I think we can run what we want, so they could be rocky-linux to match gadi. Also it is possible to mount gadi filesystems so we could potentially set it up to be very similar to a login node.

https://opus.nci.org.au/spaces/Help/pages/152207472/Nirin+Cloud+User+Guide

[aidanheerdegen]

The immediate issue has been side-stepped as the allocation for the service user on login nodes has been increased from 8GiB/16GiB (up from 2GiB/4GiB) for soft/hard limits.

However we should investigate some no-regrets upgrades:

1. Logging of which nodes we're hitting and what resources we're consuming. Could return a log file as an artefact and report to a DB to have a better overview of usage. Also allows for more informed decision making as would better understand the SU costs of PBS based builds. Consider using GNU time to report memory usage.
2. Investigate if we could have a simple load-sharing/round-robin allocation of resource intensive jobs so we're not hitting one node while others are free.
3. Test the speed difference between building on-node vs login node. Does fast local SSD make a difference? If performance is much improved might want to make this change anyway.

Regretful options not worth doing unless we decide we need to build on PBS nodes:

1. Run concretisation and source code caching as separate steps from build, which would allow for builds in PBS jobs at the expense of making workflow more complex.