Handle <picture>/<source> elements in image extraction - #56
Merged
Conversation
Image extraction only looked at <img> elements and rewrote img.src to the downloaded local copy. But sites like Medium wrap every article image in <picture><source srcset=...><img></picture>, and the browser gives <source> precedence over the <img>. So even after downloading the image and rewriting img.src, the rendered article kept loading the original remote URLs (which fail due to hotlink/referrer/CSP protection) and the saved copies were ignored. extractImageUrls now: - falls back to the largest <source> srcset when the <img> has no srcset - strips <source> siblings so the browser uses the rewritten <img src> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vi4V3Z1xQ9GBCyT3uDSbwS
✅ Deploy Preview for savrdev ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
The previous fix wasn't enough for Medium. Inspecting the real proxied HTML showed Medium's body images carry NO src on the <img> — the URL lives only in the sibling <source srcSet>. Readability discards such "empty" <img> elements (and their <picture>/<source>) during parsing, so by the time extractImageUrls runs there are no article images left at all (only the author avatar, which has a real src, survived). Add hoistPictureSources(), called in readabilityToArticle BEFORE Readability parses: for each <picture> whose <img> has no src, hoist the largest <source> srcset onto the <img>. Readability then keeps the image, and the existing extractImageUrls logic downloads it and strips the <source> siblings. Verified against the live article HTML (fetched through the app's CORS proxy): Readability previously kept 0 body images; after hoisting it keeps them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vi4V3Z1xQ9GBCyT3uDSbwS
Capture the background behind hoistPictureSources: why Medium images come through empty (srcless <img>, URL only in sibling <source srcSet>, dropped by Readability before extraction), why upstream Readability doesn't handle it (_fixLazyImages only inspects an element's own attributes; long-open Bugzilla 1355164), and the criteria for removing the workaround later. Adds docs/readability-lazy-picture-workaround.md and an inline pointer to it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vi4V3Z1xQ9GBCyT3uDSbwS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds proper handling for HTML
<picture>elements with<source>children during image extraction. The browser gives<source>elements precedence over an<img>'s own src/srcset attributes, which was causing extracted articles to continue loading remote images instead of using the downloaded local copies.Key Changes
extractImageUrlsfunction to enable testing of the new functionality<picture>/<source>fallback logic: When an<img>element lacks its own srcset, the code now checks for srcset attributes on sibling<source>elements and uses the largest image from the first available source<source>elements after extraction: After processing, all<source>elements within a<picture>are removed to ensure the browser uses the rewritten<img src>pointing to downloaded images instead of the original remote URLs<img>,<source>override removed)<source>srcset when<img>has no srcset<img>srcset over<source>elements<img>elements without<picture>wrapper remain untouchedImplementation Details
The solution prioritizes image sources in this order:
<img>element's own srcset (if present)<source>element's srcset within the parent<picture>(if no img srcset)<img>element's src attribute (fallback)All
<source>elements are removed after extraction to prevent the browser from overriding the locally-rewritten image URLs.https://claude.ai/code/session_01Vi4V3Z1xQ9GBCyT3uDSbwS