Skip to content

ci(conformance): GHA-native retry for the prod lane (max 2 attempts) + flake annotation #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pavel-y-ivanov merged 5 commits into from
Jul 15, 2026
Merged

ci(conformance): GHA-native retry for the prod lane (max 2 attempts) + flake annotation #19

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3e6081c
ci(conformance): retry prod lane once (GHA-native), flag retried passes
devin-ai-integration[bot] Jul 14, 2026
7fcd1f5
ci(conformance): raise prod retry per-attempt cap to 20m and job cap …
devin-ai-integration[bot] Jul 14, 2026
16e00b9
ci(conformance): raise prod job cap 45->50m to fit worst-case two-att…
devin-ai-integration[bot] Jul 15, 2026
1c8f901
ci(conformance): replace nick-fields/retry with an inline bash retry …
devin-ai-integration[bot] Jul 15, 2026
22f3f51
ci(conformance): label prod-lane exit 124 as TIMEOUT in the result su…
devin-ai-integration[bot] Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 38 additions & 10 deletions .github/workflows/conformance.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,8 +240,10 @@ jobs:
prod-lane:
runs-on: ubuntu-latest
# Prod scenarios are dominated by the ~30s duplicate-settle window (~40s
# each), so allow generous headroom over the fault lane.
timeout-minutes: 15
# each), so allow generous headroom over the fault lane. Sized to fit TWO
# full 20-min prod attempts (the runner step's inline retry loop caps each
# attempt at `timeout 1200`) plus the retry wait and checkout/build setup.
timeout-minutes: 50
steps:
- name: Gate (secrets + fork-safety)
id: gate
Expand Down Expand Up @@ -318,23 +320,48 @@ jobs:
# with RAINDROP_WRITE_KEY; the runner polls RAINDROP_QUERY_URL with
# RAINDROP_QUERY_API_KEY. Endpoints + credentials are scoped to this
# step so no other step in the job can read them.
#
# Inline retry loop (repo-controlled shell, no third-party action): the
# prod lane crosses the real network (ingest + Query API readback), so a
# transient blip can fail an otherwise-conformant run. Run the prod lane
# up to twice — green on a single pass — while the fault lane (a separate
# job) is never retried; each attempt is capped at 20 min by
# `timeout 1200`. We write `exit_code`/`total_attempts` ourselves so the
# summary + the flake annotation below are unchanged; a retried pass is
# never silent.
shell: bash
env:
RAINDROP_SINK_URL: https://api.raindrop.ai
RAINDROP_WRITE_KEY: ${{ secrets.RAINDROP_WRITE_KEY }}
RAINDROP_QUERY_URL: https://query.raindrop.ai
RAINDROP_QUERY_API_KEY: ${{ secrets.RAINDROP_QUERY_API_KEY }}
run: |
set +e
node harness/runner/dist/src/index.js \
--driver "$DRIVER_BIN" \
--failures conformance/failures.txt \
--lane prod \
--report "$RUNNER_TEMP/report.json" \
2>&1 | tee "$RUNNER_TEMP/runner-output.txt"
code=${PIPESTATUS[0]}
attempts=0
code=1
for i in 1 2; do
attempts=$i
set +e
timeout 1200 node harness/runner/dist/src/index.js \
--driver "$DRIVER_BIN" \
--failures conformance/failures.txt \
--lane prod \
--report "$RUNNER_TEMP/report.json" \
2>&1 | tee "$RUNNER_TEMP/runner-output.txt"

@cursor cursor Bot Jul 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Soft per-attempt timeout cap

Medium Severity

timeout 1200 sends SIGTERM and then waits for the process to exit. Without --kill-after, a runner that catches SIGTERM for graceful shutdown—or stays blocked on a hung Query API poll during that shutdown—never yields, so attempt 2 never starts and the job only dies at the 50-minute job ceiling. The stated 20-minute per-attempt cap is soft for the hang cases this retry is meant to cover.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 1c8f901. Configure here.

@devin-ai-integration devin-ai-integration Bot Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point — timeout (no --kill-after) sends only SIGTERM, so a process that traps SIGTERM and then hangs would keep the attempt alive until the 50-min job ceiling, defeating the per-attempt cap. In practice the runner is node (and the driver a short-lived child), which terminates promptly on SIGTERM, so the realistic hang case still exits 124 and retries — and the job ceiling is the hard backstop.

That said, timeout --kill-after=60 1200 … (SIGTERM, then SIGKILL 60s later) would make the cap truly hard and is a one-line, low-risk hardening. But the exact timeout 1200 <cmd> form is Pavel's explicit spec across all six PRs, so I don't want to silently deviate on all of them. Flagging for Pavel to confirm — if he's good with it I'll add --kill-after=60 uniformly (and a 137) TIMEOUT (SIGKILL) summary arm alongside the 124) one).

Comment thread
cursor[bot] marked this conversation as resolved.
code=${PIPESTATUS[0]}
set -e
[ "$code" -eq 0 ] && break
[ "$i" -eq 1 ] && { echo "raindrop-rust prod lane attempt 1 exited $code; retrying in 15s"; sleep 15; }
done
echo "exit_code=$code" >> "$GITHUB_OUTPUT"
echo "total_attempts=$attempts" >> "$GITHUB_OUTPUT"
exit "$code"

- name: Flag prod-lane flake (passed on retry)
# A prod pass that needed attempt 2 is a flake — surface it loudly so a
# retried green is visible in the run summary, never silent.
if: steps.gate.outputs.run == 'true' && steps.runner.outputs.total_attempts == '2' && steps.runner.outputs.exit_code == '0'
run: echo "::warning::raindrop-rust prod lane passed on retry (flake)"

- name: Report result
if: always()
run: |
Expand All @@ -352,6 +379,7 @@ jobs:
0) echo "- Result: **PASS** (warnings allowed)" ;;
1) echo "- Result: **FAIL** — scenario/ratchet failures (non-blocking for now)" ;;
2) echo "- Result: **ERROR** — infra/config problem (non-blocking for now)" ;;
124) echo "- Result: **TIMEOUT** — an attempt exceeded the 20-min \`timeout 1200\` cap and was killed (recorded as a failure)" ;;
*) echo "- Result: **UNKNOWN** (runner did not report an exit code)" ;;
esac
echo
Expand Down
Loading