This repository was archived by the owner on May 21, 2026. It is now read-only.
fix: Improve Open Graph fetch reliability#489
Open
rmdes wants to merge 7 commits into
Open
Conversation
reamaining 1 Including the just-posted item in the calculation ensures the spacing window accounts for all posts in the sequence, which is especially important for small queues where the difference between dividing by 2 versus 3 significantly affects the delay.
Some websites return og:url values like 'https//example.com' (missing colon after protocol). These get treated as relative URLs and concatenated with the base URL, creating broken links. This fix: - Adds fixMalformedUrl() to correct 'https//' -> 'https://' - Improves URL validation regex to require proper protocol - Validates the corrected URL before using it
- Add default Chrome user agent when ogUserAgent config is empty - Add 10 second timeout to prevent hanging on slow sites - Add accept and accept-language headers that some sites require This should reduce the 'Error fetching Open Graph data' failures, especially for the 26 bot configs that have empty ogUserAgent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Many Open Graph fetches fail with "Error fetching Open Graph data" because:
ogUserAgentconfig causes requests without a proper user agentacceptheaders that some sites requireSolution
ogUserAgentis emptyacceptandaccept-languageheadersTesting
Before: BBC News/Politico fetch with empty UA - fails
After: BBC News/Politico fetch with default UA + headers - succeeds