Add utility for locking directories

[jgfouca]

Description

This allows clients to achieve a MUTEX for a specific directory. This
can be useful for buildnmls that want to make changes to shared
directories, allowing them to avoid race conditions.

Checklist

• My code follows the style guidelines of this project (black formatting)
• I have performed a self-review of my own code
• My changes generate no new warnings
• I have added tests that exercise my feature/fix and existing tests continue to pass
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding additions and changes to the documentation

[Copilot]

Pull request overview

This PR introduces a new utility context manager in CIME.utils to provide a filesystem-based mutex for a given directory, intended to help clients (e.g., buildnml workflows) avoid race conditions when modifying shared directories.

Changes:

• Replaces the prior file-based “wait for unlocked” helper with a directory-based distributed lock implementation (distributed_dir_lock).
• Adds unit tests covering acquisition/release, exception safety, timeout behavior, retry behavior, and external cleanup edge cases.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
CIME/utils.py	Adds a directory-based lock context manager and removes the old wait_for_unlocked helper.
CIME/tests/test_unit_utils.py	Adds unit tests validating the new directory-lock behavior.

Suppressed comments (1)

CIME/utils.py:2750

• The lock acquisition timeout uses time.time(), which can jump if the system clock is adjusted (NTP/VM suspend), and the timeout check uses > rather than >=. Also, negative poll_interval / timeout will lead to confusing runtime errors; validate these inputs up front and use time.monotonic() for elapsed time measurement.

            # Check if we have timed out while waiting
            if timeout is not None and (time.time() - start_time) > timeout:
                raise TimeoutError(  # pylint: disable=raise-missing-from
                    f"Failed to acquire lock at {lock_dir_path} within {timeout} seconds."
                )

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[billsacks]

I haven't done a careful review, but I like the idea of this addition... I'll defer to others for a detailed review.