Skip to content

Automate expired Cargo quarantine CI rechecks for Renovate PRs #58

Closed

Description

@codeandsolder-git-wrangler

Why

The centralized Cargo quarantine correctly rejects newly introduced crates.io versions younger than 72 hours. But a GitHub Actions job that fails on release age stays failed indefinitely after the 72 hours have elapsed unless its branch changes or someone reruns the job. This is the remaining manual bottleneck in an otherwise verified autonomous Renovate merge workflow.

Confirmed cases:

  • rust-web-fullstack #117: only the release-age job was red for six crates from Nov 4–5. After they aged out, rerunning the original failed quarantine job on Nov 12 made the PR green; Renovate itself merged it at 17:32:34 UTC.
  • sccache-deduplicated #160: release-age failure for arc-swap 1.9.5; eligible Nov 12 at 19:33 UTC, but the existing failed GitHub check will not turn green automatically.
  • all-db-bench-rs #54: three surrealkv-compat-* crates 0.2.4 dated Nov 11 21:57 UTC; no legitimate CI recheck before Nov 14 ~21:57 UTC.
  • warpproxy #65: age gate until at least Nov 13 ~21:09 UTC.
  • sentinelO2-cloud-core #157: age gate until at least Nov 14 11:03 UTC.

This is distinct from Renovate #45236, whose missing-timestamp internal stability checks were fixed in our shared preset PR #57.

Desired design

Implement one centrally owned, idempotent recheck mechanism; do not weaken or bypass any release-age or build checks and do not hand-merge PRs.

Suggested approach: a lightweight scheduled fleet sweep using the existing scoped GitHub App identity on the authorized maintenance host (or an equivalently scoped GitHub Actions deployment). Only inspect bot-authored open PRs in repos the app can access. Find latest-commit failed quarantine/strict-Rust age steps. Parse the authoritative crate version/publish timestamps and the 72-hour threshold already reported in their logs, accounting for every newly introduced version in each failure. When and only when the newest hold has expired (plus sensible clock/registry skew margin), rerun failed jobs only with the Actions API. A subsequent failure may reveal a genuine lint/build error and must stay red.

Requirements:

  • No forced merge, no artificial success check, no change to global three-day quarantine, and no sccache bypass.
  • Handle multiple repos, multiple PRs, multiple failed age steps, and repeated execution generally and idempotently, not a hand-curated PR list.
  • Before rerun, verify the PR/head, current job/run attempt and status still match; never retry outdated or superseded checks, already-running jobs, or non-age failures.
  • Handle paginated GitHub App installation repos, throttling/API failures, missing or malformed timestamps fail-closed, and GitHub rate limits without busy loops.
  • Structured audit logs include repo/PR, blocking crates, exact first eligible time, rerun action and result. Dry-run mode and tests with time-controlled fixtures.
  • Store source/configuration and deployment instructions in this repository; one scheduled execution, not one daemon or timer per consumer repo. Preserve existing authentication scope and minimal runtime artifacts.

Success: an age-only failure becomes automatically rechecked after the age threshold; if source CI is green, the next normal Mend Renovate run merges without human intervention.

Full incident chronology: Repository maintenance sweep — 2026-11-12.

Activity

  1. codeandsolder-git-wrangler commented on Nov 12, 2026

    @codeandsolder-git-wrangler
    ContributorAuthor

    Deployed via PR #59, service and timer enabled on cold-storage; first live scan: 77 repos, 5 candidates, 0 errors, 0 actions. Await first real post-expiry rerun to close.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions