Add --colocate-memory-peak-device: overlap the trainer/rollout handoff on the GPU

[yueming-yuan]

--colocate-memory-peak-device {cpu,gpu} (default cpu = current behavior). In colocate mode with --offload-train --offload-rollout, the step-boundary handoff briefly holds both the engine's host weight mirror and the trainer's backup in host RAM. gpu mode reorders the handoff so the two host copies never coexist — the overlap moves to GPU memory instead:

• init: keep the engine weights resident (offload_kv only) through trainer init
• after generate: offload_kv → onload trainer → offload_weights
• before update_weights: trainer clear_memory + grad-buffer release → engine weight resume → offload_train
• asserts --offload-train --offload-rollout, rejects the critic path

run_deepseek_v4.py passes gpu when the resolved --hardware is GB300: host RAM there (898 GiB cgroup) is smaller than the engine weight mirror plus the trainer backup.

Split out of #2706 (it is orthogonal to the dual-backend work). Validated on the 8×GB300 DSv4 50+ step runs in #2706's A/B (jobs 2695/2703); the cpu default path is unchanged.

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}