Fix GB300 torch_dist checkpoint save crashes from forked local writers

[zyzshishui]

What does this PR do ?

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share a design doc with the team. If you're unsure what's the best way to do so, contact the @mcore-oncall.

Contribution process

Summary

Fix GB300 torch_dist checkpoint save crashes from forked local writers.

Symptom & Reproduction

• Symptom: GB300 checkpoint save produced partial iter_0000000 directories and failed before .metadata was written.
• Reproduction: Run the Qwen3.5-35B-A3B direct-save repro on a 2-node GB300 job with torch_dist checkpointing enabled.

Root Cause

1. write_preloaded_data_multiproc forks one writer per local bucket.
2. write_preloaded_data serializes parent-staged CPU tensors.
3. PyTorchStreamWriter::writeRecord crashes in crc32_16bytes.

Fix

Run local checkpoint buckets sequentially in the current writer process instead of forking local writer subprocesses. This preserves the existing public async-writer entry point and global result queue, while removing the unsafe local process boundary around staged checkpoint tensors.

Note: This does not make --async-save supported on GB300. --async-save still moves the checkpoint write into an outer async worker process, and that path reproduced the same crc32_16bytes crash in validation.

Verification

• sync no-MSC save succeeded with 16 .distcp files, .metadata, and metadata.json.
• sync MSC save succeeded with 16 .distcp files, .metadata, and metadata.json.
• load/resume from the saved checkpoint succeeded and produced a complete follow-up checkpoint.

Pre-checks

• I want this PR in a versioned release and have added the appropriate Milestone (e.g., Core 0.8)
• I have added relevant unit tests
• I have added relevant functional tests
• I have added proper typing to my code Typing guidelines
• I have added relevant documentation
• I have run the autoformatter.sh on my PR

Code review

The following process is enforced via the CODEOWNERS file for changes into megatron/core. For changes outside of megatron/core, it is up to the PR author whether or not to tag the Final Reviewer team.

For MRs into `main` branch

Feel free to message or comment the @mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

(Step 1): Add PR label Expert Review

(Step 2): Collect the expert reviewers reviews

1. Attach the Expert Review label when your PR is ready for review.
2. GitHub auto-assigns expert reviewers based on your changes. They will get notified and pick up your PR soon.

⚠️ Only proceed to the next step once all reviewers have approved, merge-conflict are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

(Step 3): Final Review

1. Add Final Review label
2. GitHub auto-assigns final reviewers based on your changes. They will get notified and pick up your PR soon.

(Optional Step 4): Cherry-pick into release branch

If this PR also needs to be merged into core_r* release branches, after this PR has been merged, select Cherry-pick to open a new PR into the release branch.

For MRs into `dev` branch

The proposed review process for `dev` branch is under active discussion.

MRs are mergable after one approval by either eharper@nvidia.com or zijiey@nvidia.com.

Merging your PR

Any member of core-adlr and core-nemo will be able to merge your PR.