Promote graceful scale-in for actions-runner to sandbox

[akuzminsky]

Summary

• Promotes the graceful scale-in implementation from environments/development to environments/sandbox (follow-up to Graceful scale-in for actions-runner (development) #265).
• Copies the 9 files listed in Promote graceful scale-in (actions-runner) to sandbox #266: 4 new files (gha-lifecycle-heartbeater.sh, gha-lifecycle-heartbeater.timer, gha-on-runner-exit.sh, gha-lifecycle-heartbeater.service.erb) and updates 5 existing ones (gha_prerun.sh, register.pp, service.pp, actions-runner.service.erb, start-actions-runner.sh.erb).
• After this merges, environments/sandbox/modules/profile/github_runner/ is byte-identical to the development copy.

Closes #266.

Deployment strategy

Gradual rollout — Puppet first, Terraform second, deliberately exercising the old-Terraform / new-Puppet combination in sandbox before any Terraform bump.

1. Merge + deploy this PR to sandbox.
2. Observe sandbox runner behavior under the old IAM / old deregistration_hookname-less fact. Expected fallout is scoped and visible:
   • delete_registration_token may return AccessDeniedException (old IAM) — heartbeater / lifecycle logic should still degrade cleanly.
   • Absent fact should not crash the heartbeater; confirm it no-ops or logs and exits non-fatally.
3. Only once the new-Puppet behavior is understood in sandbox, apply the terraform-aws-actions-runner Part 1 release (new deregistration_hookname fact + IAM) to sandbox.
4. Then validate the full happy path.

Test plan

Phase 1 — old-Terraform / new-Puppet (this PR)

• puppet-lint --fail-on-warnings clean on sandbox profile module (done locally).
• Heartbeater service/timer come up on sandbox runners without crashing.
• Characterize behavior of delete_registration_token under old IAM — document whether it fails loudly (AccessDeniedException) and whether the runner still terminates cleanly.
• Characterize behavior with the deregistration_hookname fact absent — confirm no fatal errors in Puppet runs or in the lifecycle scripts.
• Confirm no stuck Terminating:Wait instances beyond heartbeat_timeout that would not also have been stuck pre-change.

Phase 2 — after Terraform Part 1 is applied to sandbox (separate change)

• No ValidationError on SetInstanceProtection in CloudTrail during a sandbox scale-in event.
• No stuck Terminating:Wait instances beyond heartbeat_timeout.
• delete_registration_token succeeds (no AccessDeniedException).

🤖 Generated with Claude Code

[akuzminsky]

Characterize behavior of delete_registration_token under old IAM — document whether it fails loudly (AccessDeniedException) and whether the runner still terminates cleanly.

clean run

[INFO]	2026-04-19T16:35:14.559Z	ea990de8-4c04-42d9-99f8-91bc59c6681b	Found runner ip-10-1-2-75
[INFO]	2026-04-19T16:35:14.739Z	ea990de8-4c04-42d9-99f8-91bc59c6681b	Instance i-0025918052d63f00b is terminated. Will deregister the runner ip-10-1-2-75.
[INFO]	2026-04-19T16:35:15.312Z	ea990de8-4c04-42d9-99f8-91bc59c6681b	Found runner ip-10-1-2-93
END RequestId: ea990de8-4c04-42d9-99f8-91bc59c6681b

another run

[INFO]	2026-04-19T16:40:14.860Z	1431b09c-be0b-4e27-821d-a4d3082b9794	Found credentials in environment variables.
[INFO]	2026-04-19T16:40:16.486Z	1431b09c-be0b-4e27-821d-a4d3082b9794	Found credentials in environment variables.
[INFO]	2026-04-19T16:40:16.875Z	1431b09c-be0b-4e27-821d-a4d3082b9794	Deleted secret GH-reg-token-FJVASl-i-005754ffd87d1ef1d
[INFO]	2026-04-19T16:40:17.841Z	1431b09c-be0b-4e27-821d-a4d3082b9794	Lifecycle hook deregistration for i-005754ffd87d1ef1d is complete with result CONTINUE.
END RequestId: 1431b09c-be0b-4e27-821d-a4d3082b9794
REPORT RequestId: 1431b09c-be0b-4e27-821d-a4d3082b9794	Duration: 3028.92 ms	Billed Duration: 4807 ms	Memory Size: 512 MB	Max Memory Used: 158 MB	Init Duration: 1777.10