[yanksyoon]

Summary

Fixes ISD-7263. An orphaned scale set never drained when GARM reported one of its runners as running a job: the charm skipped that runner on every pass regardless of the drain's age, so the scale set, its template and its provider stayed behind forever.

Faulty vs fixed

FAULTY  (_remove_runners, both branches get past_deadline)

  for each runner of the draining scaleset
        |
        +-- status GARM refuses (creating/deleting/deleted)
        |      -> log (info before the deadline, error after)  -> skip
        |
        +-- _is_running_job(instance)          active, or a fresh queued/in-progress job
        |      -> "still running a job, will retry"            -> skip
        |          past_deadline IGNORED: the busy protection never expires,
        |          so a lost job-completion strands the runner, and the
        |          scaleset can never be deleted while it is listed
        |
        +-- delete_instance(force=_is_delete_stuck())

FIXED

  for each runner of the draining scaleset
        |
        +-- status GARM refuses -> log (info before, error after)  -> skip
        |
        +-- past_deadline?  --yes--> warn once per pass, delete_instance
        |                            (force only if _is_delete_stuck)
        |                            past MAX_JOB_RUNTIME no job can still be running
        | no
        +-- _is_running_job(instance)? --yes--> "still running a job"  -> skip
        |                                        (a real job is never cut short)
        +-- delete_instance(force=_is_delete_stuck())

Notes

• The runner is deleted plain, not forced. _delete_runner forces only when GARM already accepted a delete it could not carry out; forcing a busy runner would drop the record while the VM may still exist.
• _advance_retirement already calls _remove_runners(past_deadline=True) only after _drain_deadline_passed() is true, so the retirement path inherits the same rule rather than deleting a busy runner early.
• _drain_deadline_passed() stays on updated_at, and its docstring now records why that clock is sound: GARM's updateScaleSet writes only the fields that differ and issues no statement when none do, so the idempotent re-disable on every sweep leaves updated_at at the disable moment.

Tests

• test_drain_past_the_deadline_removes_a_runner_still_marked_busy — past the deadline the busy runner gets delete_instance("runner-1", force=False, ...) and the scale set is deleted on a later pass.
• test_drain_within_the_deadline_leaves_a_runner_marked_busy_alone — inside the deadline nothing is deleted, which is the reason the deadline exists.
• Full suite: 467 passed; ruff, pyright, CCR001 and codespell clean on the touched files.

Not in this PR

• A terminal (deleted) runner past the deadline is still reported on every pass rather than once. Deduping it needs per-scale-set state the charm does not keep, so it is left for a follow-up, together with the recover-scaleset action from the ticket.

[yanksyoon]

I'm going to think a bit more about this. I think our fundamental design of the reconciliation can be improved. The data flow isn't so straightforward.