Add script for retrying all missed job webhooks - #1049
Draft
mvandenburgh wants to merge 2 commits into
Draft
Conversation
mvandenburgh
force-pushed
the
webhook-backfill-script
branch
3 times, most recently
from
February 6, 2025 20:40
d13a646 to
b234057
Compare
mvandenburgh
force-pushed
the
webhook-backfill-script
branch
from
May 29, 2025 01:29
b234057 to
fd35a07
Compare
mvandenburgh
marked this pull request as ready for review
May 29, 2025 01:30
Member
Author
|
@jjnesbitt with the analytics webhook being down the last couple days, it seems like now would be a good time to test this out and get it merged. IIRC I ran it a while back and it appeared to work, but I don't think we've needed to retry this many job events before. |
mvandenburgh
force-pushed
the
webhook-backfill-script
branch
from
May 29, 2025 13:57
fd35a07 to
600ef38
Compare
Member
Author
|
Just realized this has to be updated for the new job model. So not ready for review yet, sorry for the confusion |
mvandenburgh
force-pushed
the
webhook-backfill-script
branch
2 times, most recently
from
May 29, 2025 14:52
fdb5af8 to
3dde210
Compare
Member
Author
|
This should be ready now. |
mvandenburgh
force-pushed
the
webhook-backfill-script
branch
from
May 29, 2025 17:53
4b5691b to
a9f4488
Compare
mvandenburgh
commented
Jun 2, 2025
Comment on lines
+30
to
+40
| def retry_webhook(webhook_event: WebhookEvent, dry_run: bool) -> None: | ||
| if dry_run: | ||
| click.echo(f"Would retry webhook {webhook_event}") | ||
| return | ||
|
|
||
| click.echo(f"Retrying webhook {webhook_event}") | ||
| gl = get_gitlab_handle() | ||
|
|
||
| # https://docs.gitlab.com/ee/api/project_webhooks.html#resend-a-project-webhook-event | ||
| retry_url = f"/projects/{webhook_event.project_id}/hooks/{webhook_event.webhook_id}/events/{webhook_event.webhook_event_id}/resend" | ||
| gl.http_post(retry_url) |
Member
Author
There was a problem hiding this comment.
TODO: try fetching the webhook payload from the web_hook_logs table and invoke the ingest function directly instead of sending a webhook
mvandenburgh
marked this pull request as draft
June 2, 2025 16:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds a Django management script that
web_hook_logstable of the GitLab DB for all webhook events in the past n seconds (n is an argument to the script)I've ran variations of this script in the past to retry missed job webhooks after fixing bugs in the webhook handler, downtime of the webhook handler, etc. and it works.