Add e2e diff validation test (Scenario 0)#21
Merged
Phlogistique merged 12 commits intomainfrom Dec 27, 2025
Merged
Conversation
Add a comprehensive test that validates the action preserves PR diffs: 1. Part A - Without Action (proves diffs break): - Create a 3-PR stack without the action installed - Capture initial diffs (each shows 1 line change) - Merge bottom PR - Verify diffs are NOW BROKEN (accumulated changes visible) 2. Part B - With Action (proves diffs are preserved): - Install the action workflow - Create a new 3-PR stack - Capture initial diffs - Merge bottom PR, wait for action - Verify diffs are IDENTICAL to initial diffs This approach avoids race conditions by observing the broken state before installing the action, then verifying the fixed state after.
- Each feature in a stack now changes a DIFFERENT line (lines 3, 4, 5 for noact_ stack and lines 6, 7, 2 for act_ stack) - This ensures after retarget, the diff clearly shows accumulated changes (2+ lines instead of just 1) - Manually retarget PR to main before deleting branch (GitHub doesn't auto-retarget when base branch is deleted) - Use ^+ prefix in grep patterns to only match actual added lines, not context lines in the diff - Add pollution detection: verify PR2 doesn't show Feature 1's changes before action runs, and still doesn't show them after
- Scenario 0: Use auto-delete branch setting to trigger GitHub's auto-retarget naturally, instead of manual retarget - Scenario 0: Reduce to 2 PRs (3rd was unnecessary) - Scenario 0: Use retry loop instead of arbitrary sleep - Scenario 0: Remove redundant pollution checks (identity check suffices) - Remove scenario 0b: merge diff validation into scenario 1 - Scenario 1: Add diff preservation check (compare before/after) - Add wait_for_pr_base_change helper function
GitHub's auto-retarget when a branch is deleted with auto-delete enabled is not reliable. Add fallback logic to manually retarget the PR if auto-retarget doesn't occur within 5 attempts.
Instead of silently falling back to manual retarget when GitHub's auto-retarget doesn't trigger, fail with debug info to understand the root cause.
GitHub's branch deletion after merge is asynchronous. Add a retry loop to wait for the branch to be deleted before checking if auto-retarget happened.
The repo setting 'delete_branch_on_merge' only works for web UI merges, not for gh CLI merges. Instead, explicitly delete the branch via API after merge to trigger GitHub's auto-retarget.
When a branch is deleted separately from a merge, GitHub closes the dependent PR instead of retargeting it. The branch deletion must be part of the merge operation to trigger auto-retarget.
The gh pr merge --delete-branch flag doesn't reliably trigger GitHub's auto-retarget. The repo setting delete_branch_on_merge causes GitHub to atomically handle branch deletion and PR retargeting together when a PR is merged.
Each PR now modifies: - Line 2 (shared, accumulates through stack - tests merge handling) - A unique line (3, 4, 5 for PR2, PR3, PR4 - for diff pollution visibility) This ensures that if the action doesn't properly handle merges, conflicts would occur on line 2.
The compare_diffs identity check already verifies that diffs are preserved correctly. The manual line-by-line checks were redundant.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a comprehensive test that validates the action preserves PR diffs:
Part A - Without Action (proves diffs break):
Part B - With Action (proves diffs are preserved):
This approach avoids race conditions by observing the broken state
before installing the action, then verifying the fixed state after.