Skip to content

ci: show why a Claude review failed, and run it on an API key with Claude Code directly - #53

Merged
blurbery merged 16 commits into
mainfrom
ci/claude-review-errors
Nov 18, 2026
Merged

blurbery merged 16 commits into
mainfrom
ci/claude-review-errors

Conversation

@blurbery

@blurbery blurbery commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

What this changes

The Claude review stopped working on 17 November, and the log only said Claude result reported subtype success with is_error:true. This PR finds out why, and gets reviews running again on a Claude API key.

  • Failures say why. When a review fails, a new step prints Claude's own result as an error annotation (for example "Credit balance is too low" or "Invalid API key"). That's how each problem below was found.
  • API key instead of a subscription. The CLAUDE_CODE_OAUTH_TOKEN subscription token had been revoked. The review now uses only the ANTHROPIC_API_KEY secret, billed to API credit like Vivid's review, and skips with a notice when it isn't set.
  • Claude Code instead of claude-code-action. With an API key, every claude-code-action run (v1.0.248 and v1.0.280) was refused on its first request with 400 invalid_request_error: Credit balance is too low. The same key worked for direct API calls and for the Claude Code CLI in the same job, and the organisation has credit. So the review now installs Claude Code 2.1.327 and runs it directly, with the same prompt, model (claude-sonnet-5-5), 25-turn limit and read-only tools as before. Inline comments are posted with gh api (the CLI has no inline-comment tool), and Claude may only run that command for this PR. The summary comment and the "already reviewed" marker are unchanged.

How I tested it

  • This PR's own review job, run by the new step: Claude Code made 4 turns for about US$0.07, posted its summary and two inline comments, and the job passed.
  • Both of the review's findings are fixed here: this description (which still described a workspace header the final change doesn't use), and the failure step, which now reads the CLI's single result object. I checked that step's jq against a sample failed result.
  • Earlier runs on this PR checked the key directly: /v1/messages with the key returned 200 on Haiku and Sonnet, and claude -p with Claude Code 2.1.327 returned a normal answer, while claude-code-action failed with the error above.

Release notes

BEGIN_COMMIT_OVERRIDE
ci: show why a Claude review failed, and run it on an API key with Claude Code directly
END_COMMIT_OVERRIDE

AI disclosure (required)

  • No AI tools were used for any part of this change.
  • AI tools were used for this change, as described below.

AI tools used and what they did:

I designed and directed this change (surfacing Claude's own error when a review fails, moving the review to an API key, and running Claude Code directly once claude-code-action kept refusing a working key) and used Claude Opus to help write the workflow, track down the failures and write this description.

Checklist

  • I've read CONTRIBUTING.md.
  • The title follows Conventional Commits, like fix: keep posters on Jellyfin 12.
  • I understand every line of this change and can explain it.
  • New source files have the CineSets header, and the unit tests pass.
  • Nothing private is included (server addresses, API keys, user names, logs).

The review action hides Claude's output, so a failed review only logged "is_error: true" (it did on #52, with no tokens used). A new step after a failed review prints Claude's error and last result, which name the cause, such as an expired token.
The subscription token in CLAUDE_CODE_OAUTH_TOKEN was revoked. The review now uses ANTHROPIC_API_KEY whenever it's set, like Vivid's review, and falls back to the subscription token only when there's no key.
@blurbery blurbery changed the title ci: show why a Claude review failed ci: show why a Claude review failed, and prefer an API key Nov 17, 2026
Reviews should come out of API credit, not a Claude subscription, so the workflow no longer reads CLAUDE_CODE_OAUTH_TOKEN at all. It needs the ANTHROPIC_API_KEY secret and skips the review without it.
@blurbery blurbery changed the title ci: show why a Claude review failed, and prefer an API key ci: show why a Claude review failed, and run it on an API key Nov 17, 2026
An API key linked to an account rather than one workspace (sk-ant-usr-...) needs the workspace named on each request, or the API answers "Credit balance is too low" (it did on #53). The review now sends the ANTHROPIC_WORKSPACE_ID secret as the anthropic-workspace-id header, as Vivid's review does.
v1.0.248 bundles Claude Code 2.1.296, which answered every review with "Credit balance is too low" even though the same API key and workspace work for direct API calls and for Claude Code 2.1.327. v1.0.280 bundles 2.1.327. This also drops the temporary checks used to find that.
With an API key, every claude-code-action run (v1.0.248 and v1.0.280) was refused on its first request with "Credit balance is too low" (400), while the same key works for direct API calls and for the Claude Code CLI in the same job. The review now runs Claude Code 2.1.327 itself with the same prompt, model, turn limit and read-only tools, and posts inline comments with gh api. It no longer uses the workspace header, which the key doesn't need.
github_token: ${{ secrets.GITHUB_TOKEN }}
prompt: |
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description says the review sends ANTHROPIC_WORKSPACE_ID as an anthropic-workspace-id header via ANTHROPIC_CUSTOM_HEADERS, but neither appears in this diff. If the sk-ant-usr-... key still needs that header, the CLI will hit the same billing error. Add the env var here, or update the description to match what the PR does.

run: |
LOG="$RUNNER_TEMP/claude-execution-output.json"
[ -s "$LOG" ] || { echo "::error::Claude left no output to read."; exit 0; }
jq -r '(if type == "array" then . else [.] end)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--output-format json writes one final result object, so there are no assistant messages and the Claude error: ... line will never print (the description says it did, which was the action's output file). Use --output-format stream-json --verbose or drop that branch and rely on the result text.

@github-actions

Copy link
Copy Markdown
Contributor

Two things to fix: the workspace header described in the PR isn't in the diff, and the failure step's assistant error filter can't match the CLI's single JSON result.

Claude Code's --output-format json writes one result object, so the failure step now prints its result alone (found by the review on #53).
@blurbery blurbery changed the title ci: show why a Claude review failed, and run it on an API key ci: show why a Claude review failed, and run it on an API key with Claude Code directly Nov 18, 2026
@blurbery
blurbery merged commit 4bb53a3 into main Nov 18, 2026
18 checks passed
@blurbery
blurbery deleted the ci/claude-review-errors branch November 18, 2026 00:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant