Skip to content

fix(tags): report server 4xx rejections instead of masking them#132

Open
ryanhayame wants to merge 5 commits into
mainfrom
fix/one-group-tag-per-request
Open

fix(tags): report server 4xx rejections instead of masking them#132
ryanhayame wants to merge 5 commits into
mainfrom
fix/one-group-tag-per-request

Conversation

@ryanhayame

@ryanhayame ryanhayame commented Jul 8, 2026

Copy link
Copy Markdown

This PR does two independent, forward-compatible things: (1) makes the SDK report server errors honestly instead of masking/retrying them, and (2) sends a sampleIndex so lists of media keep their logged order in the UI.

Both are safe to merge before any server/DB change — see "Merge safety" at the bottom.


Part 1 — report server 4xx rejections instead of masking them

Problem. On a non-2xx write, iface._try retried everything (incl. 4xx) with backoff, then either silently no-op'd (/api/runs/tags/update — a failed run.add_tags(...) looked like it worked) or raised a misleading ConnectionError("...Check connection...") (/api/runs/create — blamed the network). The server's real reason lived only in a buried warning log. This first surfaced with the server's new "a run can have at most one group:* tag" 400.

Change. Surface the server's reason; don't rewrite the caller's request.

  • iface.py: new PlutoRequestError (+ _server_error_message, parses the JSON error field). _try treats permanent 4xx as terminal (no retry, no wasted backoff) and, when raise_on_error=True, raises PlutoRequestError with the server's message. Transient 4xx — 401/408/429 (RETRYABLE_STATUS_CODES) — and 5xx/network stay retryable (auth races, timeouts, rate limits), mirroring query.py.
  • op.py: create/resume pass raise_on_error=True → a rejection raises RuntimeError("Failed to create run: <server reason>"); a genuine unreachable server still raises ConnectionError. add_tags/remove_tags surface a rejection at WARNING instead of DEBUG.
  • sync/process.py: the sync-subprocess uploader (_post_with_retry, the primary run.add_tags path) uses raise_for_status() and applies the same terminal-4xx / retryable-401/408/429 policy, so the sync loop logs the real reason instead of a bare 400.

No client-side tag rewriting — if a caller sends two group:* tags, the request fails and the server's reason is reported (we do not silently drop one).

Part 2 — sampleIndex so media lists keep logged order

Problem. pluto.log({"eval/img": [img0, img1, img2]}, step=s) came back out of order in the UI because the server sorts a step's files by fileName and nothing recorded each image's position.

Change (SDK side). Each object in the POST /files files: [...] array now carries a non-negative, 0-based integer sampleIndex = the file's position within a single (logName, step) log() call.

  • A list of N media → 0..N-1; a single (non-list) media → 0.
  • Transparent — no new public API. The index is derived purely from list position via enumerate(), captured at log() time (op.py, where order is known) and threaded through the sync DB to the payload:
    • op.py: enumerate items in the log loop → _process_log_item_sync_enqueue_file_syncSyncProcessManager.enqueue_file
    • sync/store.py: new sample_index column (additive migration, defaults 0), persisted on enqueue and read back into FileRecord
    • sync/process.py: _get_presigned_urls adds sampleIndex to each entry
    • api.py: make_compat_file_v1 (legacy builder) enumerates each logName's list
  • Deriving from the stored position (not by numbering the upload batch) is deliberate — sync batches can split or mix steps, so batch-numbering would be wrong.

Tests

  • tests/test_iface_errors.py: _try doesn't retry a 400 and raises PlutoRequestError with the server message (not ConnectionError); persistent 5xx/401 retry then raise with the real status_code; transient 401 that clears on retry succeeds; network error returns None; the sync uploader doesn't retry a 400 and surfaces the reason.
  • tests/test_sync_process.py: store round-trip (sample_index list → 0..N-1, default 0); presign payload carries sampleIndex; make_compat_file_v1 numbers a list 0..N-1 and a single media 0.

Ruff + mypy clean on changed files.

Merge safety (verified against server-private)

Safe to merge before the backend/DB changes:

  • Part 1 is pure client-side behavior — no backend dependency. If the server doesn't yet enforce the group:* rule, there's simply no 400 to surface; behavior is unchanged.
  • Part 2 adds an extra JSON field. The ingest upload endpoint (ingest/src/models/files.rs, FileInput) has no #[serde(deny_unknown_fields)], so serde silently ignores the unknown sampleIndex — same pattern as the already-shipped caption field. Until the backend adds sample_index (column + sort), media just keeps its current fileName sort; after, ordering works. No coordinated deploy required.

🤖 Generated with Claude Code

The backend (server-private, staging-v2) now rejects with HTTP 400 any
create / tag-update payload whose tags array carries more than one
group:* tag ({"error":"A run can have at most one group:* tag."}).
The SDK handled the rejection badly: the write client _try retried the
400 with backoff, then either silently no-op'd (tags/update) or raised a
misleading ConnectionError("...Check connection...") (create). The real
reason was only ever in a buried warning log.

This makes the SDK surface server validation errors loudly and honestly,
without second-guessing or rewriting the caller's tags:

- iface.py: add PlutoRequestError + _server_error_message(). _try now
  treats 4xx as terminal (no retry, no wasted backoff) and, when
  raise_on_error=True, raises PlutoRequestError carrying the server's
  reason (parsed from the JSON "error" field). Only 5xx and
  network/timeout errors are retried — mirroring query.py.
- op.py: create/resume pass raise_on_error=True. A server rejection now
  raises RuntimeError("Failed to create run: <server reason>"); a genuine
  unreachable server still raises the ConnectionError. add_tags/
  remove_tags surface a rejection at WARNING instead of burying it at
  DEBUG.
- sync/process.py: the sync-subprocess uploader (_post_with_retry) — the
  primary path for run.add_tags(), which sends the full tags array — also
  treats 4xx as terminal and raises PlutoRequestError with the server's
  reason, so the sync loop logs the actual message rather than a bare
  "400 Bad Request".

No client-side tag rewriting: if a caller sends two group:* tags, the
request fails and the server's reason is reported, rather than the SDK
quietly dropping one.

Tests (tests/test_iface_errors.py): _try does not retry a 400 and raises
PlutoRequestError with the server message (not ConnectionError), retries
5xx, and returns None on network errors; the sync uploader does not retry
a 400 and surfaces the reason, and still retries 5xx.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces PlutoRequestError and a helper function _server_error_message to capture and surface human-readable server validation errors (such as 4xx responses) instead of raising generic connection errors. It updates the ServerInterface and the sync uploader to immediately halt retries on 4xx client errors, while adding comprehensive unit tests to verify this behavior. The reviewer suggests an improvement to extract and propagate the actual HTTP status code from the error message when raising PlutoRequestError on persistent server errors (e.g., 5xx errors) after exhausting retries, along with a corresponding test assertion to verify this behavior.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread pluto/iface.py Outdated
Comment thread tests/test_iface_errors.py
@ryanhayame

Copy link
Copy Markdown
Author

@cursor review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 63bc5a2. Configure here.

…n path

Addresses PR review: on the persistent-5xx exhaustion path _try raised
PlutoRequestError with status_code=None, inconsistent with the 4xx path
which sets the real code. Thread the last HTTP status through the retry
recursion (last_status) so the exception reports the actual code (e.g.
500) instead of None. Prefer threading over parsing it back out of the
formatted error_info string, which would couple the value to the log
format. Assert the code is populated in the 5xx test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…atus_code

The previous rewrite of _post_with_retry read response.status_code
directly, which broke ~11 existing sync-uploader tests: they mock the
response with a no-op raise_for_status() and never set status_code, so
`MagicMock() < 400` raised TypeError.

Restore the original contract — call raise_for_status(); success if it
doesn't raise — and branch on the httpx.HTTPStatusError it raises: 4xx is
terminal and raises PlutoRequestError with the server's reason, 5xx and
network/timeout errors stay retryable. Same external behavior as intended,
compatible with the existing mocks.

Update the two new sync tests to attach a request to their canned
httpx.Response objects (raise_for_status needs one to build its error).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Making *all* 4xx terminal regressed resilience against transient auth
failures. In CI, the auth/token-validation step intermittently times out
and the create endpoint returns a transient 401; previously _try retried
all non-2xx up to 5x so it recovered, but the "4xx is terminal" change
hard-failed on the first 401 (RuntimeError: Failed to create run:
Unauthorized).

Introduce RETRYABLE_STATUS_CODES = {401, 408, 429} — auth races, server
timeouts, rate limits — and treat only the *other* 4xx (400, 403, 404,
409, 422, ...) as terminal. Genuine client/validation errors like 400
"A run can have at most one group:* tag." still fail fast with the
server's reason; transient auth/timeout/rate-limit 4xx retry like 5xx and,
if they never clear, still raise PlutoRequestError with the real code
after exhausting retries. Applied to both the write client (_try) and the
sync uploader (_post_with_retry).

Tests: transient 401 that clears on retry succeeds; persistent 401 is
retried then raises with status_code 401; 400 stays terminal/no-retry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When a user logs a list of media at one step —
pluto.log({"eval/img": [img0, img1, img2]}, step=s) — the server sorted a
step's files by fileName, losing the logged order. Send an explicit
0-based sampleIndex per file so the server can restore it.

Contract (matches the server): each object in the POST /files `files: [...]`
array gets a non-negative integer `sampleIndex` = the file's position within
a single (logName, step) log() call. A list of N media → 0..N-1; a scalar
(single, non-list) media → 0. Missing sampleIndex is treated as 0 server-side
(older SDKs keep working), so no version gate is needed.

Transparent to users — no new public API. The index is derived purely from
list position via enumerate(), captured at log() time (op.py, where order is
known) and threaded through the sync DB to the upload payload:
- op.py: enumerate items in the log loop → _process_log_item_sync →
  _enqueue_file_sync → SyncProcessManager.enqueue_file.
- sync/store.py: new `sample_index` column (additive migration, defaults 0),
  persisted on enqueue and read back into FileRecord.
- sync/process.py: `_get_presigned_urls` adds `sampleIndex` to each entry.
- api.py: make_compat_file_v1 (legacy builder) enumerates each logName list.

Deriving from stored position (not from the batch) is deliberate — sync
batches can split or mix steps, so numbering the batch would be wrong.

Tests: store round-trip (list → 0..N-1, default 0); sync presign payload
carries sampleIndex; make_compat_file_v1 numbers a list 0..N-1 and a single
media 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant