Skip to content

ci(pr-gate): wait for slow Backend CI instead of timing out at 40 min - #3595

Merged
PetrefiedThunder merged 2 commits into
mainfrom
fix/pr-gate-backend-skip
Oct 25, 2026
Merged

PetrefiedThunder merged 2 commits into
mainfrom
fix/pr-gate-backend-skip

Conversation

@PetrefiedThunder

@PetrefiedThunder PetrefiedThunder commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Root cause

"PR Gate: Backend" is not hanging on PRs without backend changes: path detection (three-dot diff mirroring backend-ci.yml paths:) already skips correctly. Every affected PR touched backend paths (e.g. #3586 changes scripts/build_audit_bundle.sh and services/ingestion/tests/…), so Backend CI did trigger.

The real failure: the wait step polled "Backend CI Status Gate" for a fixed 40 min (job timeout 45). Under runner contention, Backend Services CI/CD now takes 60–80+ min (jobs sit queued), so the gate times out and fails, and it never re-evaluates when the backend gate later goes green.

Fix (pr-gate.yml backend job only)

  • Job timeout-minutes 45 → 150; poll deadline 40 → 140 min (still bounded, can't hang forever).
  • While no completed gate check exists, the step also reads the backend-ci.yml workflow runs for the head sha:
    • no run after 10 min → fail (path-list drift), never pass;
    • all runs completed but no gate check (e.g. cancelled), after one recheck → fail with the conclusion;
    • run queued/in progress → keep waiting.
  • Pass still requires a completed "Backend CI Status Gate" with conclusion success. Frontend job, detection, and the no-backend-change skip are unchanged.

Affected PRs (same cause)

After merge, these PRs need main merged in (or a gate rerun) to go green.

Verified locally: YAML parses, scripts/check_workflow_inventory.py passes, node --check passes on the script. Live Actions behavior has not been tested yet.


Note

Medium Risk
Changes merge-blocking PR gate behavior and failure modes for backend PRs; incorrect new logic could block or delay merges, but pass still requires a successful Backend CI Status Gate.

Overview
Fixes PR Gate: Backend failing around 40 minutes while Backend CI is still queued or running (often 60–80+ minutes under runner contention).

The backend job timeout rises from 45 → 150 minutes and the Backend CI Status Gate poll window from 40 → 140 minutes. While waiting, the step now inspects backend-ci.yml workflow runs for the PR head SHA: if backend paths changed but no run appears within 10 minutes, it fails (likely path-list drift vs backend-ci.yml); if the workflow finishes without a gate check, it rechecks once then fails with the run conclusion; otherwise it keeps polling until a completed gate with success.

Frontend gate timing, path detection, and vacuous pass when backend paths are untouched are unchanged.

Reviewed by Cursor Bugbot for commit 040e914. Bugbot is set up for automated code reviews on this repo. Configure here.

Signed-off-by: Christopher Sellers <1@christophersellers.com>
@vercel

vercel Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
regengine Ready Ready Preview Oct 25, 2026 7:05am UTC

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-25T06:34:36.468671Z 040e914 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

Run configuration
Commits

Reviewing files that changed from the base of the PR and between 040e914 and f2e9e9a.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between bfe805c and 040e914.

Files selected for processing: 1
  • .github/workflows/pr-gate.yml

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 4 remain after this review.

Walkthrough

The backend PR gate timeout increases to 150 minutes, and the check polling deadline increases to 140 minutes. While polling, the gate queries backend-ci.yml runs for the PR head SHA. If backend paths were detected but no run appears within 10 minutes, the gate fails. If runs complete without the expected check, the gate rechecks once and then fails with the run conclusions if the check remains absent.

Priority: Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: Minimal · up to 040e9

The longer backend wait and new run checks have no identified merge-blocking issue. Live GitHub Actions behavior remains untested.

Architecture risk: Low · up to 040e9

The changed surface does not map to a changed system, dependency edge, entrypoint, or external dependency.

Changed systems: None identified.

Architecture concerns
No architecture-level concerns identified.

Review details

Before / after behavior

  • observed — Modified behavior in .github/workflows/pr-gate.yml: The backend gate timeout increases from 45 to 150 minutes.
  • observed — Modified behavior in .github/workflows/pr-gate.yml: The backend check wait description and deadline change from a 40-minute polling window to a 140-minute window, within the job’s new 150-minute timeout.
  • observed — Modified behavior in .github/workflows/pr-gate.yml: Adds state to track whether a completed backend workflow has already been observed without its expected check.
  • observed — Modified behavior in .github/workflows/pr-gate.yml: While waiting for the backend check, the gate now queries backend-ci.yml runs for the PR head SHA. It fails if no run appears within 10 minutes after backend paths were detected. If runs complete without the expected check, it rechecks once, then fails with the run conclusions if the check is still absent; otherwise it logs workflow status while continuing to poll.
Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Description check Warning The description provides detailed root cause, implementation changes, affected PRs, and validation results, but it does not follow the required template. It omits the Summary, Type of Change, Checklis… Rewrite the description using the repository template. Add the Summary, Type of Change, Test Plan, Checklist, and Related Issues sections. Mark the CI/CD or infrastructure change and document that live GitHub Actions behavior was not tested…
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly describes the primary change: extending the PR gate wait period for slow Backend CI.
Docstring Coverage Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

Full details: Description check

Explanation

The description provides detailed root cause, implementation changes, affected PRs, and validation results, but it does not follow the required template. It omits the Summary, Type of Change, Checklist, and Related Issues sections, and it does not use the required test-plan checklist.

Resolution

Rewrite the description using the repository template. Add the Summary, Type of Change, Test Plan, Checklist, and Related Issues sections. Mark the CI/CD or infrastructure change and document that live GitHub Actions behavior was not tested.

Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

PR Blocker Checklist

Head SHA: f2e9e9ae1946518bda328432aeabd383dca46d7a

Status: Ready

  • All review threads resolved
  • No reviews requesting changes
  • No blocked review engines

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Vercel preview access note removed (A-2177 secret-hygiene cleanup).
Access is via Vercel login (Deployment Protection).

@PetrefiedThunder PetrefiedThunder left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REVIEW: PASS deepseek 040e914

Reviewed the change to .github/workflows/pr-gate.yml (backend job only).

  • Run lookup. listWorkflowRuns filters on workflow_id: 'backend-ci.yml', head_sha: HEAD_SHA (the PR head, passed via env) and event: 'pull_request', so it finds only this commit's PR runs. Schedule and push runs are excluded.
  • No false pass. The only success path is still the existing checks.listForRef(check_name="Backend CI Status Gate", filter: latest) with conclusion === 'success'. Every new branch either calls setFailed or keeps polling.
    • Multiple runs or reruns: the check fails only when every run is completed and no gate check exists, and only on two consecutive polls (the completedWithoutGate reset is in the else branch).
    • A rerun goes back to queued, which resets that flag, so the job keeps waiting.
  • 10-minute check. It fires only when workflowRuns.length === 0. A queued or waiting run is already in the list, so runner contention can't trip it. It catches only real path-list drift, and the message says so.
  • Permissions. Unchanged: contents/checks/actions: read, and the top level is permissions: {}. actions: read is what listWorkflowRuns needs.
  • Injection. There are no new ${{ }} expressions inside script:. The interpolated values are CHECK_NAME (a constant) and the API-enum conclusion and status.
  • Timeouts. The 140-minute poll is below the 150-minute job timeout. callWithRetry sleeps are capped at the deadline.

Non-blocking:

  • The pre-existing runs.find(r => r.status === 'completed') takes the first completed gate run. With filter: latest that's normally the only one, but runs.find on success with a sort by completed_at would be more robust.
  • per_page: 20 is fine in practice.

CI: everything is green except Engine Review Pass, PR Blocker Checklist and Head SHA Check Guard, which wait on this review and so are circular. PR Gate: Backend passed.

@PetrefiedThunder PetrefiedThunder left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REVIEW: PASS deepseek f2e9e9a

Delta review, by hand. Merge of main 91737af into 040e914 (already PASSed). The remerge-diff is empty. The diff vs main is still only .github/workflows/pr-gate.yml (+37/-4), identical to 040e914.

CI: all real checks green. Remaining failures are engine-review gates only (circular), plus Vercel Preview Built (deploy skipped by Ignored Build Step on non-frontend PRs, not a build failure): Engine Review Pass,Engine Review Pass,visual-report (non-required),PR Blocker Checklist,Head SHA Check Guard,

@PetrefiedThunder
PetrefiedThunder merged commit 1b8f83f into main Oct 25, 2026
47 of 53 checks passed
@PetrefiedThunder
PetrefiedThunder deleted the fix/pr-gate-backend-skip branch October 25, 2026 08:02

@PetrefiedThunder PetrefiedThunder left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REVIEW: PASS deepseek f2e9e9a

This follows up my earlier hand review. A dsh headless run on the full PR diff (merge-base..f2e9e9a) confirmed the PASS. Excerpt:

- The state machine cannot false-pass: `completedWithoutGate` requires two consecutive completed-without-gate observations, and any newly appearing run resets it.
- I compared the gate's 18 backend paths with `backend-ci.yml`'s 18 `pull_request` paths: identical sets and order, so the new 10-minute "not triggered" failure should only fire on real drift/cancellation, not on ordinary backend PRs.
- The JS block is syntactically valid when wrapped as the async function body `actions/github-script` uses (`node --check` passes).
- No production code, copy, migrations, tenant-scoped queries, or fixtures are touched; the existing workflow-contract checks (`scripts/check_workflow_inventory.py`, `tests/scripts/test_*`) do not pin the old 40/45-minute values.

No concrete bug found in the diff. Non-blocking observations: the new branch logic has no automated test (the repo has no harness for the embedded script), and the 10-minute heuristic could turn a rare GitHub-side run-creation delay into a failure — but it fails closed and is recoverable by rerun, which matches the PR's stated intent.

VERDICT: PASS

This branch was successfully deployed

1 active deployment
Preview — f2e9e9ae Deployed Oct 25, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:infra Issues found by Bot-Infra agent ci

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant