Fix #391: automate models.dev pricing refresh - #522
Conversation
|
Warning Review limit reached
Next review available in: 49 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe PR adds a scheduled and manual workflow that refreshes ChangesPricing refresh automation
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant GitHubActions
participant PricingUpdate as pnpm run pricing:update
participant ModelsDev as models.dev
participant Validation as Node validation step
participant RustSDK as cargo test -p relayburn-sdk
participant PullRequest as create-pull-request
GitHubActions->>PricingUpdate: start pricing refresh
PricingUpdate->>ModelsDev: retrieve current pricing
PricingUpdate->>GitHubActions: write refreshed snapshot
GitHubActions->>Validation: validate snapshot and detect changes
Validation->>GitHubActions: publish summaries and changed output
GitHubActions->>RustSDK: run tests when pricing changed
GitHubActions->>PullRequest: create or update pricing pull request
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 44c82c9753
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
.github/workflows/update-pricing.yml (2)
18-23: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winConsider bounding the job with
timeout-minutes.This job has no
timeout-minutes, so it defaults to GitHub's 360-minute job timeout. The job performs a network fetch, a Rust toolchain install, and acargo testrun; a hang in any of these could occupy a runner for hours before the default timeout kicks in. Add an explicit, tighter timeout (for example 30-60 minutes) to fail fast on stuck steps.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/update-pricing.yml around lines 18 - 23, Add an explicit timeout-minutes value to the update-pricing job, near runs-on and permissions, using a tighter bound such as 30–60 minutes so stalled network, toolchain, or cargo test steps terminate promptly.
89-96: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winHardcoded critical model IDs will need routine upkeep.
The critical-entry guard hardcodes exact model IDs (
claude-sonnet-4-6,gpt-5.5). When Anthropic or OpenAI rename or retire these specific IDs upstream, this step throws and blocks every subsequent weekly run until a maintainer updates the string, even though the refresh itself is healthy. Consider matching by a stable prefix (for example, "at least oneanthropicmodel ID starting withclaude-") or moving the critical-list to a small config file that is easier to keep current, to reduce how often this check needs a manual bump.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/update-pricing.yml around lines 89 - 96, Update the critical-entry guard in the pricing validation loop to avoid exact model IDs that require routine updates. Validate that each provider has at least one current model matching a stable provider-specific prefix, such as Anthropic IDs beginning with “claude-” and OpenAI IDs beginning with the appropriate stable prefix, while preserving the existing failure for missing critical coverage.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/update-pricing.yml:
- Around line 24-27: Update the actions/checkout step named “Checkout main” to
disable persisted credentials by setting persist-credentials to false, allowing
create-pull-request to configure the remote credentials without conflicts.
---
Nitpick comments:
In @.github/workflows/update-pricing.yml:
- Around line 18-23: Add an explicit timeout-minutes value to the update-pricing
job, near runs-on and permissions, using a tighter bound such as 30–60 minutes
so stalled network, toolchain, or cargo test steps terminate promptly.
- Around line 89-96: Update the critical-entry guard in the pricing validation
loop to avoid exact model IDs that require routine updates. Validate that each
provider has at least one current model matching a stable provider-specific
prefix, such as Anthropic IDs beginning with “claude-” and OpenAI IDs beginning
with the appropriate stable prefix, while preserving the existing failure for
missing critical coverage.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 42901b70-36b3-4896-9b73-d318e4609880
📒 Files selected for processing (2)
.github/workflows/update-pricing.ymlREADME.md
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 2 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
Automated review follow-up:
No changes were made for the three inapplicable findings. |
1 participant
Fixes #391
This adds a weekly Monday 09:17 UTC workflow, with manual dispatch, that runs the sanctioned
pnpm run pricing:updaterefresh and opens or updates a non-draft review PR on the fixedautomation/models-dev-pricingbranch. No-change runs create no empty PR, and the workflow never pushes tomain, publishes, tags, or releases.The updater validates the fetched JSON, rejects a provider-count regression or a model-count collapse greater than 10%, retains critical Anthropic/OpenAI entries, and runs the full
relayburn-sdktest suite against the newly embedded snapshot before creating a PR. Its generated PR body reports primary-provider model and price changes separately, links an extended workflow summary, and is clamped below GitHub's body limit. Actions use immutable SHAs, and write permissions are limited to the update job.Against current
main, today's upstream refresh correctly fails the full SDK gate because the existing last-provider-wins flattening lets reseller entries replace first-party tariffs. Open PR #510 fixes that root cause by giving Anthropic, OpenAI, Google, Google Vertex, and xAI entries precedence; the first pricing refresh after #510 merges is expected to pass.Local verification covered the refresh script, JSON/integrity and summary generation, repeat-refresh no-change behavior, the SDK safety gate,
actionlint, YAML parsing, and the PR-body clamp. Cron dispatch, create-pull-request's fixed-branch create/update/no-change behavior, andGITHUB_TOKEN's suppression of follow-on pull-request workflows remain unproven until the workflow runs on GitHub.🤖 Generated with Claude Code.