feat(rocm): add end-to-end Vime rollout/training attention ablation matrix - #385
Merged
Merged
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
Signed-off-by: vensen <vensenmu@gmail.com>
Signed-off-by: vensen <vensenmu@gmail.com>
Flink-ddd
force-pushed
the
bench/rocm-pr230-attention-microprobes
branch
from
September 3, 2026 10:46
4422132 to
d0472ea
Compare
Signed-off-by: Codex <codex@openai.com>
Flink-ddd
marked this pull request as draft
September 3, 2026 13:01
Flink-ddd
marked this pull request as ready for review
September 3, 2026 14:58
inaniloquentee
approved these changes
Sep 4, 2026
maxiaosong1124
approved these changes
Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ablation matrix
Pselects the framework-native implementation.Rselects the RL-Kernel ROCm implementation.Every cell starts in a fresh process and uses the same model, checkpoint, prompt data, seeds, token limits, rollout count, and pre-update state.
Implementation
Runtime acceptance boundary
A subprocess exit code of zero is not sufficient for a passing cell.
Each cell must emit executed Attention readbacks from both Megatron training and vLLM rollout. For every R side, the runner requires evidence for:
The runner also rejects missing, malformed, stale, duplicated, or case-inconsistent readbacks.
Replay invariants
The matrix freezes the command and replay-defining environment before the first cell. Only the Attention case selection and case-local output paths may change.
An executable run requires identifiable model, distributed checkpoint, and prompt inputs. It also enforces NUM_ROLLOUT=1, ensuring every cell begins from the same pre-update checkpoint state.
Running the matrix
Review the resolved launch contract without allocating a GPU run:
Execute all four cells:
Individual cells can be selected with repeatable --case arguments while debugging. The final hardware acceptance run should execute the complete four-cell matrix.
Results and artifacts
Runtime logs and unmodified framework readbacks are written to the requested external output directory. The aggregate report is a human-readable summary.md.
No generated benchmark or result JSON is checked into the repository.
Validation
Added end-to-end matrix, replay-freezing, readback-validation, and failure-boundary tests.
Extended Attention binding and framework-runtime adapter coverage for ROCm.
Added the ROCm end-to-end tests to the default CI workflow.
Source formatting, Python compilation/AST validation, shell syntax validation, and the isolated matrix/readback dry-run contract passed locally.
A full physical MI300X Vime matrix remains the hardware acceptance step before moving the PR out of Draft.
Reference: #230