fix(seo): submit only changed URLs to IndexNow, and fix the workflow - #2590
fix(seo): submit only changed URLs to IndexNow, and fix the workflow#2590innolope-dev wants to merge 2 commits into
Conversation
The workflow has never succeeded: it resolves Node from a `.node-version` file that does not exist in this repo, so every production deploy since February failed at setup-node before reaching the script. 0 of 2750 runs sent a ping. Behind that, the script resubmitted all ~900 URLs on every successful production deploy (~14/day). IndexNow is for URLs that were added, updated or deleted; resubmitting an unchanged site burns the daily quota and gets the host deprioritised. - Pin node-version: '20' so the job runs at all. - Take URLs from sitemap.ts instead of a second, drifted copy of the page list — the copy was missing blog posts, stories, use-cases, withdraw, deposit rails, pricing, supported-networks and the content hub, i.e. exactly the new-article pages worth submitting (916 vs 1121 URLs). - Default to a delta: diff the sitemap's URL set against the set submitted last time (cached between runs) and map content files changed since then back to the pages they render. A typical content deploy now submits ~15 URLs instead of ~1100, and a deploy that touches no content submits none. - Skip install entirely when nothing that feeds the sitemap has changed. - workflow_dispatch gains a `full` input for a deliberate full resubmission. Rewrite sitemap URLs onto the production origin, since BASE_URL is env-dependent and a stray NEXT_PUBLIC_BASE_URL would otherwise submit preview URLs under peanut.me.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe IndexNow workflow supports manual full and automatic delta submissions, detects sitemap-relevant content changes, restores and saves cached state, and serializes runs. The ping script derives URLs from the sitemap, submits batches with per-batch failure handling, and persists submission metadata. ChangesIndexNow delta submission
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant GitHubActions
participant StateCache
participant ChangeDetector
participant ping-indexnow.ts
participant IndexNowAPI
GitHubActions->>StateCache: restore submission state
GitHubActions->>ChangeDetector: detect sitemap-relevant changes
ChangeDetector-->>GitHubActions: return run decision and submission metadata
GitHubActions->>ping-indexnow.ts: pass full mode and changed files
ping-indexnow.ts->>IndexNowAPI: submit URL batches
GitHubActions->>StateCache: save updated submission state
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
ESLint install failed: dependency version conflict. Check your lock file or package.json. Comment |
Code-analysis diffPainscore total: 6350.19 → 6350.19 (0) |
🧪 UI test report — ✅ all greenSuites
📊 Coverage (unit)
⏱ 10 slowest test cases
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
scripts/ping-indexnow.ts (1)
119-153: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
submit()doesn't catchfetchfailures — a network error/timeout crashes the whole run instead of being counted as a batch failure.
fetch()is only guarded for HTTP status codes ≥400; a rejected promise (DNS failure, connection reset, or the 30sAbortControllertimeout firing) propagates as an unhandled rejection. That abortsmain()mid-loop, skips any remaining batches, and — critically — skips thewriteState(current)call that followssubmit(urls)in every caller inmain(), so a single flaky batch loses all progress for the run instead of failing gracefully and reporting which batches actually failed.🛠️ Proposed fix: treat network/timeout errors the same as HTTP failures
const controller = new AbortController() const timeout = setTimeout(() => controller.abort(), 30_000) - const res = await fetch(INDEXNOW_ENDPOINT, { - method: 'POST', - headers: { 'Content-Type': 'application/json; charset=utf-8' }, - body: JSON.stringify({ - host: 'peanut.me', - key: KEY, - keyLocation: `${PRODUCTION_ORIGIN}/${KEY}.txt`, - urlList: batch, - }), - signal: controller.signal, - }).finally(() => clearTimeout(timeout)) - - console.log( - `Batch ${Math.floor(i / MAX_URLS_PER_REQUEST) + 1}: ${res.status} ${res.statusText} (${batch.length} URLs)` - ) - - if (res.status >= 400) { - console.error(' Error:', await res.text()) - failures++ - } + try { + const res = await fetch(INDEXNOW_ENDPOINT, { + method: 'POST', + headers: { 'Content-Type': 'application/json; charset=utf-8' }, + body: JSON.stringify({ + host: 'peanut.me', + key: KEY, + keyLocation: `${PRODUCTION_ORIGIN}/${KEY}.txt`, + urlList: batch, + }), + signal: controller.signal, + }) + console.log( + `Batch ${Math.floor(i / MAX_URLS_PER_REQUEST) + 1}: ${res.status} ${res.statusText} (${batch.length} URLs)` + ) + if (res.status >= 400) { + console.error(' Error:', await res.text()) + failures++ + } + } catch (err) { + console.error(` Batch ${Math.floor(i / MAX_URLS_PER_REQUEST) + 1} failed:`, err) + failures++ + } finally { + clearTimeout(timeout) + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/ping-indexnow.ts` around lines 119 - 153, Update submit() to catch fetch and timeout/network errors for each batch, count them in failures, and continue processing remaining batches. Report the affected batch and error details, while preserving the existing HTTP failure handling and final nonzero exit when any batch fails.
🧹 Nitpick comments (1)
scripts/ping-indexnow.ts (1)
184-203: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winDelta submission only ever grows the tracked URL set — removed pages are never resubmitted for deindexing.
added/updatedare both derived by filteringcurrent, so a URL that existed inprevious.urlsbut no longer appears incurrent(a deleted page, retired corridor, etc.) is simply dropped from tracking and never gets IndexNow-notified. IndexNow's purpose per the file's own header comment is signalling "added, updated or deleted" URLs — deleted ones currently get no signal, leaving stale pages indexed longer than necessary.Consider also computing
removed = previous.urls.filter((url) => !current.includes(url))and including those in the submission batch (IndexNow will re-crawl and see the 404, prompting deindexing).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/ping-indexnow.ts` around lines 184 - 203, Update the delta computation around added and updated to also derive removed URLs from previous.urls entries absent from current. Include removed in the urls submission batch and report its count in the summary, while preserving the existing no-change handling and state update flow.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/indexnow.yml:
- Around line 59-71: Update the fallback branch around changed_files in the
workflow to emit an unknown/force_full signal when prev_content_sha is
unavailable, rather than only leaving changed empty. Propagate that signal into
the existing INDEXNOW_FULL calculation by OR-ing it with the current full-run
conditions, matching the sources_changed fallback behavior and ensuring
changedSlugSets() processes all URLs.
---
Outside diff comments:
In `@scripts/ping-indexnow.ts`:
- Around line 119-153: Update submit() to catch fetch and timeout/network errors
for each batch, count them in failures, and continue processing remaining
batches. Report the affected batch and error details, while preserving the
existing HTTP failure handling and final nonzero exit when any batch fails.
---
Nitpick comments:
In `@scripts/ping-indexnow.ts`:
- Around line 184-203: Update the delta computation around added and updated to
also derive removed URLs from previous.urls entries absent from current. Include
removed in the urls submission batch and report its count in the summary, while
preserving the existing no-change handling and state update flow.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 707cca41-f0dd-4057-9251-6c32fdba2a4e
📒 Files selected for processing (3)
.github/workflows/indexnow.yml.gitignorescripts/ping-indexnow.ts
…le state Both from CodeRabbit review on #2590. A rejected fetch — DNS failure, connection reset, or the 30s abort firing — propagated out of submit() and aborted the run mid-loop, skipping the remaining batches with a stack trace instead of a counted failure. Catch it, count it like an HTTP error, and keep going; the non-zero exit still leaves the state file untouched so the next run retries. When the content commit from the last submission could not be resolved, the diff came back empty and edits to existing pages were silently dropped (new pages were still caught by the sitemap diff). Unshallow the content submodule so the commit is almost always reachable, and fall back to a full submission when it still is not, rather than skipping.
|
Both CodeRabbit findings addressed in c7d84d6.
Unresolvable previous content commit — real, though narrower than described: the sitemap diff still catches new pages, so what was lost was re-pings for edited existing pages in that window. Fixed at the root as well as the symptom:
Verified against real git history: unresolvable The |
1 participant
The workflow has never run successfully
.github/workflows/indexnow.ymlresolves Node fromnode-version-file: '.node-version', but that file does not exist in this repo. Every production deploy since the workflow landed in February has failed atactions/setup-node, beforepnpm install:0 successes out of 2750 runs (348 failures, the rest correctly skipped preview deploys). IndexNow has never actually been pinged from CI. The key at
public/054e10e6239a45cb2d06e92d669f5b6f.txtis in place, so nothing else was wrong — the job just never got that far.Behind that, it was pinging far too much
deployment_statusfires on every successful Production deploy (~14/day), and the script rebuilt and submitted the entire URL list each time, unconditionally. IndexNow is for URLs that were added, updated or deleted — resubmitting an unchanged site burns the daily quota and is the pattern that gets a host's submissions deprioritised. Nothing changes on/en/argentinabecause a mobile jank fix merged.And the URL list had drifted from the sitemap
scripts/ping-indexnow.tskept its own copy of the page list. It had fallen behindsitemap.tsby 205 URLs — missing blog posts, stories, use-cases, withdraw, deposit rails, pricing, supported-networks and the content hub. Those are exactly the new-article pages worth submitting.Changes
node-version: '20'(matchingtests.yml) so the job runs at all.sitemap.tsinstead of a second, drifting copy — 916 → 1121 URLs, and no way to drift again.src/data/seo,sitemap.ts,src/i18n,src/lib/{content,blog}.ts).workflow_dispatchgains afullinput for a deliberate full resubmission.BASE_URLis env-dependent, and a strayNEXT_PUBLIC_BASE_URLwould otherwise submit preview URLs underpeanut.me.concurrency: indexnowserialises overlapping deploys so they can't race on the cache.Effect
fullVerification
Ran locally against a mock IndexNow endpoint, and the workflow's resolve step against real git history:
Nothing changed — skipping IndexNow submission.content/help/passkeys/*.md+content/send-to/argentina/from/spain/en.mdchanged → submits exactly/{locale}/help/passkeysand/{locale}/send-money-from/spain/to/argentina, 10 URLscontent/help/en.mdmatches/{locale}/helponly, not all 37 help articlesrun=falsewhen nothing changed, andrun=truefor commit032683ba9("register Faster Payments + SPEI as deposit rails") — a data-only SEO change that adds URLs without moving the content submodulepnpm typecheckclean;prettier --checkcleanThe first run after merge will submit the full sitemap once (no cached state), which is correct — it's also the first ping this site has ever sent.
Summary by CodeRabbit
New Features
Bug Fixes
Performance / Reliability
Chores