diff --git a/README.md b/README.md index d09e31a..610d57f 100644 --- a/README.md +++ b/README.md @@ -82,6 +82,7 @@ The source baseline is `3c5b227649371507ddc4eb46ac5133afe6db1cab` (ZCode 3.16.2, - Text, thoughts, tools, subagent progress, usage and history use V4 snapshots and deltas. Tool output is marked when ZCode truncates it. Usage is cumulative; repeated snapshots do not add it again. - Text submitted during generation uses native guidance. Attachments use the native queue. Paseo reports one run across these native turns and completes it only after all accepted input has been consumed and foreground work has ended. - Paseo sends every message typed in the app during a run as a steer that asks to clear pending permissions. The plugin queues it first, then declines pending permissions, questions and Plan approvals with their deny action until ZCode reads the message, including requests opened in the meantime. Requests that appear after it is read are left to the user. Declining a Plan approval in Plan mode ends ZCode's turn; the message then starts the next native turn in the same Paseo run (ADR 22). +- A failed turn reports a code classified from the attribution of ZCode's error: `MODEL_RATE_LIMITED`, `MODEL_AUTH_FAILED`, `MODEL_CONTEXT_EXCEEDED`, `MODEL_NETWORK_ERROR`, `MODEL_PROVIDER_ERROR` or `MODEL_REQUEST_FAILED` when the model provider or network failed, and `NATIVE_TURN_FAILED` otherwise. The message is the plugin's own text with the HTTP status and the provider's error code when known. The provider's own message, which can include a quota reset time, is not shown; Coding Plan reset times appear on Paseo 0.11's Usage screen (ADR 23). - Acceptance is reported once, after the native acknowledgement. Unknown delivery results are never automatically resent. Stop identifies the current native execution, disables automatic queue execution and removes pending input. A new explicit run enables the queue again. - As in ZCode Desktop, successful hooks are not shown. A finished hook invocation appears once, as a notification, only when a hook was blocked, failed, timed out or was cancelled; context compaction also appears once, when it ends. Paseo appends every notification it receives, so progress is not reported. - Native permission IDs, option values, questions and Plan approval retain their meanings. Question auto-resolution is disabled for each owned Server. Workspace hook trust is reviewed through the official CLI; it is not represented as “allow once.” diff --git "a/docs/adr/0007-\346\234\200\344\275\216\343\203\220\343\203\274\343\202\270\343\203\247\343\203\263\343\201\250\345\256\237\350\241\214\346\231\202\346\244\234\350\250\274\343\201\247zcode\343\201\256\346\233\264\346\226\260\343\202\222\350\250\261\345\217\257\343\201\231\343\202\213.md" "b/docs/adr/0007-\346\234\200\344\275\216\343\203\220\343\203\274\343\202\270\343\203\247\343\203\263\343\201\250\345\256\237\350\241\214\346\231\202\346\244\234\350\250\274\343\201\247zcode\343\201\256\346\233\264\346\226\260\343\202\222\350\250\261\345\217\257\343\201\231\343\202\213.md" index ec9a435..de5059f 100644 --- "a/docs/adr/0007-\346\234\200\344\275\216\343\203\220\343\203\274\343\202\270\343\203\247\343\203\263\343\201\250\345\256\237\350\241\214\346\231\202\346\244\234\350\250\274\343\201\247zcode\343\201\256\346\233\264\346\226\260\343\202\222\350\250\261\345\217\257\343\201\231\343\202\213.md" +++ "b/docs/adr/0007-\346\234\200\344\275\216\343\203\220\343\203\274\343\202\270\343\203\247\343\203\263\343\201\250\345\256\237\350\241\214\346\231\202\346\244\234\350\250\274\343\201\247zcode\343\201\256\346\233\264\346\226\260\343\202\222\350\250\261\345\217\257\343\201\231\343\202\213.md" @@ -14,6 +14,8 @@ links: kind: amendedby - target: 13 kind: amendedby + - target: 23 + kind: amendedby --- # 最低バージョンと実行時検証でZCodeの更新を許可する diff --git "a/docs/adr/0023-\343\202\277\343\203\274\343\203\263\343\201\256\345\244\261\346\225\227\343\201\257v4\343\201\256lasterror\343\201\256attribution\343\201\247\345\210\206\351\241\236\343\201\227\343\201\237\343\202\263\343\203\274\343\203\211\343\201\250\345\233\272\345\256\232\343\201\256\346\226\207\350\250\200\343\201\247\344\274\235\343\201\210\343\202\213.md" "b/docs/adr/0023-\343\202\277\343\203\274\343\203\263\343\201\256\345\244\261\346\225\227\343\201\257v4\343\201\256lasterror\343\201\256attribution\343\201\247\345\210\206\351\241\236\343\201\227\343\201\237\343\202\263\343\203\274\343\203\211\343\201\250\345\233\272\345\256\232\343\201\256\346\226\207\350\250\200\343\201\247\344\274\235\343\201\210\343\202\213.md" new file mode 100644 index 0000000..756ee2c --- /dev/null +++ "b/docs/adr/0023-\343\202\277\343\203\274\343\203\263\343\201\256\345\244\261\346\225\227\343\201\257v4\343\201\256lasterror\343\201\256attribution\343\201\247\345\210\206\351\241\236\343\201\227\343\201\237\343\202\263\343\203\274\343\203\211\343\201\250\345\233\272\345\256\232\343\201\256\346\226\207\350\250\200\343\201\247\344\274\235\343\201\210\343\202\213.md" @@ -0,0 +1,75 @@ +--- +number: 23 +title: ターンの失敗はV4のlastErrorのattributionで分類したコードと固定の文言で伝える +status: accepted +date: 2026-11-09 +links: + - target: 7 + kind: amends +--- + +# ターンの失敗はV4のlastErrorのattributionで分類したコードと固定の文言で伝える + +## Context and Problem Statement + +ZCode のターンが失敗すると、Paseo には `[System Error] ZCode turn failed` と `code: NATIVE_PROTOCOL_ERROR` だけが表示されていた([Issue #86](https://github.com/supermomonga/paseo-plugin-zcode-provider/issues/86))。プラグインは失敗した turnHeader だけを見て、固定の文言とコードを返していた。利用者もオーケストレーターのエージェントも、利用上限・認証・ネットワークのどれが原因かを判断できなかった。`NATIVE_PROTOCOL_ERROR` は通信の形式の誤りを表すコードなので、プラグインの不具合のようにも見えた。 + +3c5b227 の CLI は、TurnError を受けると、turnHeader を `failed` にするのと同じフレームで `control.lastError`(`sessionErrorInfoSchema`)を設定し、次のターンの開始時に `null` に戻す(`product-projection.ts`)。Desktop の `ChatErrorBanner` はこの値を表示する。`lastError` には ZCode の `message`・`detail`・`underlyingErrorMessage` と、列挙値と数値からなる `attribution`(`source`、`reason`、`errorPhase`、`exceptionKind`、`statusCode`、`providerErrorCode`、`retryable` など)がある。 + +管理下ランタイム 3.16.2 で、ローカルのモデルから GLM の 429(エラーコード 1310)を返して確認した。`message` はプロバイダーの原文(週・月の利用上限とリセット時刻を伝える中国語の文)、`code` は `"1310"`、`attribution.reason` は `rate_limited`、`statusCode` は 429 だった。`recoverable` は `true` で、`attribution.retryable: false` と食い違っていた。 + +ADR 7 は、ネイティブのエラーメッセージを ProviderError.diagnostic とログに出さないと決めている。プロバイダーのエラー文には、プロバイダーによって API キーの一部やリクエストの内容が入ることがある。Paseo はターンの失敗の文言・コード・診断をタイムラインに保存し、オーケストレーターのエージェントにも渡す(`agent-manager.ts` の `formatTurnFailedMessage`)。 + +## Decision Drivers + +- 利用者とエージェントが、失敗の種類と次に取る行動を判断できるようにする。 +- ADR 7 のとおり、ネイティブの文字列を出力に含めない。 +- ZCode が公式に投影する V4 の状態だけを使う。旧プロトコルのイベントやエラー文の照合には頼らない。 + +## Considered Options + +- `lastError.attribution` で分類したコードと、プラグインの固定の文言を返す。 +- Desktop と同じく `lastError.message`(プロバイダーの原文)を表示する。 +- 旧プロトコルの `turn.failed` イベントのエラー詳細を使う(Issue #86 の提案)。 + +## Decision Outcome + +Chosen option: 「`lastError.attribution` で分類したコードと、プラグインの固定の文言を返す」。ADR 7 を変えずに、失敗の種類と対処を伝えられるため。 + +- ターンが失敗したら、そのフレームの `control.lastError` を読む。分類には `attribution.source`(無ければ `lastError.source`)と `attribution.reason` だけを使う。`lastError.code` にはプロバイダー固有のコードがそのまま入ることがあるので使わない。 +- `source` が `provider` か `network` のとき、`reason`(ZCode の `ModelFailureReason`)を次のコードに対応付ける。`MODEL_RATE_LIMITED`(`rate_limited`、`offpeak_queued`)、`MODEL_AUTH_FAILED`(`auth_failed`、`auth_refresh`、`provider_not_configured`)、`MODEL_CONTEXT_EXCEEDED`(`context_exceeded`)、`MODEL_NETWORK_ERROR`(`network_error`、`timeout`、`stream_idle_timeout`、`stale_connection`、`proxy_error`、`tls_error`)、`MODEL_PROVIDER_ERROR`(`server_error`、`provider_overloaded`)。それ以外の `reason` は、`network` なら `MODEL_NETWORK_ERROR`、`provider` なら `MODEL_REQUEST_FAILED` とする。 +- `runtime`・`tool` の失敗と、`lastError` が無い失敗は `NATIVE_TURN_FAILED`(`ZCode turn failed.`)とする。ターンの失敗に `NATIVE_PROTOCOL_ERROR` は使わない。 +- 文言はプラグインの固定の英文とし、HTTP ステータスと、数字だけか英字の語だけからなるプロバイダーのエラーコードを添える。利用上限では、Coding Plan のリセット時刻を Paseo 0.11 以降の Usage 画面で確認できると案内する(ADR 18)。 +- ProviderError.diagnostic に `stage: "turn"` と `failure` を加える。`failure` には、許可リストに一致した `source`・`reason`・`errorPhase`・`exceptionKind`、100〜599 の `statusCode`、上記の条件を満たす `providerErrorCode`、`retryable` だけを入れる。`providerId`・`modelId` は利用者が付けた名前を含みうるので入れない。 + +### Consequences + +- Good, because 利用上限・認証・ネットワークなどの失敗を、コードと文言で見分けられる。 +- Good, because ネイティブの文字列は、文言にも診断にも入らない。 +- Bad, because プロバイダーの原文にしか無い情報(Coding Plan 以外の利用上限のリセット時刻など)は伝わらない。 +- Bad, because ZCode が `reason` の語彙を変えると、分類が `MODEL_REQUEST_FAILED` などの汎用のコードに落ちる。 + +### Confirmation + +`server/turn-failure.test.ts` で、各 `reason` と `source` の分類、ネイティブの文字列が文言と診断に入らないこと、報告できないプロバイダーのエラーコードを落とすことを確認する。`test/provider.test.ts` で、3.16.2 の 429 と同じ形の `lastError` が分類されて Paseo の `session.turn` に届くことと、`lastError` が無いときの汎用の失敗を確認する。`npm run test:stdio-runtime` は、ローカルのモデルが 429(1310)を返したときに `MODEL_RATE_LIMITED` が届き、原文が含まれず、次のプロンプトが完了することを実ランタイムで確認する。 + +## Pros and Cons of the Options + +### 分類したコードと固定の文言を返す + +- Good, because ADR 7 の範囲で実現でき、表示する内容をプラグインが決められる。 +- Bad, because 固定の文言では、プロバイダーごとの事情までは表せない。 + +### `lastError.message` を表示する + +- Good, because Desktop と同じ内容になり、リセット時刻などがそのまま分かる。 +- Bad, because ADR 7 の見直しが要る。プロバイダーの原文がタイムラインに保存され、エージェントにも渡る。原文はプロバイダーの言語(GLM なら中国語)で返る。 + +### 旧プロトコルの `turn.failed` イベントを使う + +- Good, because エラー詳細の構造は `lastError` と同じである。 +- Bad, because ADR 13 によりプラグインは V4 の会話状態だけを使い、旧プロトコルのイベントを購読していない。同じ内容は V4 の `lastError` に公式に投影されている。 + +## More Information + +ADR 7 の診断の方針を補う。ZCode 側の根拠は 3c5b227 の `apps/zcode-cli/packages/bootstrap/src/zcode-protocol-v4/product-projection.ts`(TurnError の投影と、ターン開始時の消去)、`packages/shared/src/zcode-protocol-v4/snapshot.ts`(`sessionErrorInfoSchema`、`errorAttributionSchema`)、`apps/zcode-cli/packages/contracts/src/model/index.ts`(`ModelFailureReason`)、`apps/zcode-cli/packages/adapters/src/model/failure-provider-business-codes.ts`、`packages/ui/src/ChatErrorBanner.tsx`。Paseo 側は `packages/server/src/server/agent/agent-manager.ts` の `formatTurnFailedMessage`。ZCode が `lastError` にリセット時刻などを構造化して加えたとき、または Paseo がエラーコードに応じた表示を持ったときに見直す。 diff --git a/docs/adr/README.md b/docs/adr/README.md index 24b1b7f..b05a882 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -20,3 +20,4 @@ * [20. タグの無い上流の同期コミットをpackage versionで特定してZCodeのソース基準にする](0020-タグの無い上流の同期コミットをpackage-versionで特定してzcodeのソース基準にする.md) * [21. Desktop専用のvisualizeプラグインもADR 19の抑止対象に加える](0021-desktop専用のvisualizeプラグインもadr-19の抑止対象に加える.md) * [22. Paseoが求めるときは受け付けたSteerが読まれるまで保留中の権限確認を拒否する](0022-paseoが求めるときは受け付けたsteerが読まれるまで保留中の権限確認を拒否する.md) +* [23. ターンの失敗はV4のlastErrorのattributionで分類したコードと固定の文言で伝える](0023-ターンの失敗はv4のlasterrorのattributionで分類したコードと固定の文言で伝える.md) diff --git a/docs/development.md b/docs/development.md index af19093..5d28679 100644 --- a/docs/development.md +++ b/docs/development.md @@ -12,7 +12,7 @@ npm run check:zcode-source -- ~/ghq/github.com/zai-org/ZCode npm run test:upstream -- /absolute/path/to/paseo ``` -`test:upstream` uses Paseo's actual plugin compiler, Git preparation and Provider adapter. Use commit `cafe4b01c8522ff39560241c5deb01427dc8dc5b` (0.12.0), matching the installed development SDK. It checks clean npm preparation of a Git checkout both with `NODE_ENV` unset and `production`, installation of the `npm pack` tarball with Paseo's npm acquisition options, server/client registration and the provider icon of both, V4 timeline, guidance, attachment queue aggregation, stop, provider replacement, history and Plan configuration on resume. The adapter fixture uses the official V4 schemas and delta application; it does not prove native behavior. +`test:upstream` uses Paseo's actual plugin compiler, Git preparation and Provider adapter. Use commit `cafe4b01c8522ff39560241c5deb01427dc8dc5b` (0.12.0), matching the installed development SDK. It checks clean npm preparation of a Git checkout both with `NODE_ENV` unset and `production`, installation of the `npm pack` tarball with Paseo's npm acquisition options, server/client registration and the provider icon of both, V4 timeline, guidance, attachment queue aggregation, a classified turn failure, stop, provider replacement, history and Plan configuration on resume. The adapter fixture uses the official V4 schemas and delta application; it does not prove native behavior. ## Source of truth @@ -73,7 +73,7 @@ The baseline's remote terminal service cannot locate the integrated distribution `npm run test:usage-runtime` runs the provider's `status()` and the Coding Plan usage source (ADR 18) against the same runtime and existing ZCode sign-in, and validates the results with Paseo's `ProviderStatusSchema` and `UsageReportSchema`. ZCode requests the account's quota from Z.ai or BigModel; the check writes no settings and prints no account names or keys. -`npm run test:stdio-runtime` exercises the real extracted Server and Agent against a deterministic **local HTTP model fixture**, without billable API calls. It isolates HOME, ZCode settings, the SQLite database, temp sockets, workspace and plugin mappings. It checks catalog, draft, Bash, acknowledgements, completion, paged replay, mode/Plan restore **with both saved Paseo settings**, questions, tool approval, targeted stop and restart, session listing and owned Agent exit after Server SIGKILL. The test captures the settings before shutdown, closes the Server, opens a separate connection and requires the restored settings to be reported before `session.ready`. Any failed assertion exits nonzero. It deletes test data after cleanup. Loopback and Unix sockets require execution outside a restrictive network sandbox. +`npm run test:stdio-runtime` exercises the real extracted Server and Agent against a deterministic **local HTTP model fixture**, without billable API calls. It isolates HOME, ZCode settings, the SQLite database, temp sockets, workspace and plugin mappings. It checks catalog, draft, Bash, acknowledgements, completion, paged replay, mode/Plan restore **with both saved Paseo settings**, questions, tool approval, targeted stop and restart, a provider quota failure (HTTP 429 with GLM's code 1310) that must arrive as `MODEL_RATE_LIMITED` without the provider's text and a following completed turn, session listing and owned Agent exit after Server SIGKILL. The test captures the settings before shutdown, closes the Server, opens a separate connection and requires the restored settings to be reported before `session.ready`. Any failed assertion exits nonzero. It deletes test data after cleanup. Loopback and Unix sockets require execution outside a restrictive network sandbox. `npm run test:native-restore` adds a strict check of cold resume **without explicit settings** to the same isolated runner. It fails on the pinned source because ZCode replaces saved execution state with an old message mode and Plan OFF. This path is outside the supported restoration contract, so this command is separate from required CI and release conditions; there is no `continue-on-error` or blanket error suppression. Run it when evaluating official runtime updates and record the outcome. A successful upstream check is evidence for reconsidering the guarantee, not an automatic expansion of it. See [ADR 14](adr/0014-paseoの保存設定を再適用する復元を保証範囲とする.md). diff --git a/docs/verification.md b/docs/verification.md index b1eba1b..f2e2871 100644 --- a/docs/verification.md +++ b/docs/verification.md @@ -1,3 +1,33 @@ +# ターンの失敗が原因を問わず `NATIVE_PROTOCOL_ERROR` になる(2026-11-09) + +[Issue #86](https://github.com/supermomonga/paseo-plugin-zcode-provider/issues/86) の対応。ZCode のターンが失敗すると、原因が利用上限・認証・ネットワークのどれであっても、Paseo には `[System Error] ZCode turn failed` と `code: NATIVE_PROTOCOL_ERROR` だけが表示されていた。 + +## 原因 + +- `server/session.ts` の `maybeComplete()` は失敗した turnHeader だけを見て、固定の文言と `NATIVE_PROTOCOL_ERROR` を返していた。`server/provider.ts` も文言を `ZCode turn failed` に固定していた。 +- ZCode は失敗の内容を V4 の `control.lastError` に入れていたが、プラグインはこれを読んでいなかった。 + +## 修正 + +- ターンが失敗したら `control.lastError` の `attribution` を読み、`MODEL_RATE_LIMITED` などのコードとプラグインの固定の英文で伝える。HTTP ステータスとプロバイダーのエラーコードを添え、診断には許可リストに一致した項目だけを加える。プロバイダーの原文は出さない。判断は [ADR 23](adr/0023-ターンの失敗はv4のlasterrorのattributionで分類したコードと固定の文言で伝える.md) に記録した。 +- `lastError` が無い失敗と、`runtime`・`tool` の失敗は `NATIVE_TURN_FAILED` とする。 +- `test:stdio-runtime` に、ローカルのモデルが 429 を返すシナリオを加えた。`test:upstream` に、Paseo の実際のアダプターでの失敗したターンの確認を加えた。 + +## ZCode の挙動(3c5b227、管理下ランタイム 3.16.2-paseo.1) + +- TurnError では、turnHeader を `failed` にするのと同じフレームで `control.lastError` を設定し、次のターンの開始時に `null` に戻す(`product-projection.ts`)。 +- ローカルのモデルから GLM の 429(エラーコード 1310)を返すと、モデルへの要求は 1 回だけで、再試行されなかった。`lastError` は `code: "1310"`、`message` はプロバイダーの原文、`source: "provider"`、`recoverable: true` だった。`attribution` は `reason: "rate_limited"`、`statusCode: 429`、`providerErrorCode: "1310"`、`retryable: false`、`errorPhase: "stream"`、`exceptionKind: "provider_business"` だった。`detail` にはプロバイダー ID・モデル・リクエスト ID が入っていた。 +- Desktop の `ChatErrorBanner` は、ローカライズした文言がある code ならその文言を、無ければ `lastError.message` を表示する。 + +## 検証(Linux x64、Node.js 26.8.1、2026-11-09) + +- 追加したテストが修正前に失敗することを確認した。429 の `lastError` があっても、Paseo には `NATIVE_PROTOCOL_ERROR` が届いた。 +- `npm run typecheck`、`npm test`(23 ファイル、227 tests)、`npm run build`、`npm run format:check`、`npm run check:zcode-source` が成功した。 +- 管理下ランタイム 3.16.2-paseo.1 に対する `npm run test:stdio-runtime` が成功した。追加した 429 のシナリオでは、`MODEL_RATE_LIMITED` が原文を含まずに届き、続くプロンプトが完了した。 +- Paseo 0.12.0(`cafe4b01c8522ff39560241c5deb01427dc8dc5b`)に対する `npm run test:upstream` が成功した。ローカルの Paseo のチェックアウトは新しいコミットにあるため、`git clone --shared` で作った別の clone を `cafe4b01` にして、`NODE_PATH` を外して実行した。 + +未検証:実モデルでの利用上限・認証・ネットワークの失敗。実 daemon と実 UI での表示。Linux x64 以外のプラットフォーム。 + # アプリからの Steer が `Duplicate ZCode clientMessageId` で失敗する(2026-10-24) ZCode の実行中にアプリからメッセージを送ると、`[System Error] Duplicate ZCode clientMessageId` になり、メッセージが届かなかった。 diff --git a/scripts/check-stdio-runtime.mjs b/scripts/check-stdio-runtime.mjs index b3fd241..5bbdc4e 100644 --- a/scripts/check-stdio-runtime.mjs +++ b/scripts/check-stdio-runtime.mjs @@ -52,6 +52,9 @@ let connection, modelCalls = 0; const responses = new Set(); const modelRequests = []; +// GLM's weekly or monthly quota response (business code 1310). +const QUOTA_MESSAGE = + "您已达到每周/每月使用上限,您的限额将在 2026-11-07 16:34:13 重置。"; const modelServer = createServer(async (req, res) => { try { let body = ""; @@ -61,6 +64,13 @@ const modelServer = createServer(async (req, res) => { tools: (input.tools ?? []).map((t) => t.function.name), messages: JSON.stringify(input.messages ?? []), }); + if (scenario === "quota") { + res.writeHead(429, { "content-type": "application/json" }); + res.end( + JSON.stringify({ error: { code: "1310", message: QUOTA_MESSAGE } }), + ); + return; + } res.writeHead(200, { "content-type": "text/event-stream" }); responses.add(res); res.on("close", () => responses.delete(res)); @@ -557,6 +567,19 @@ try { await configure({ mode: "yolo" }); await prompt("bash"); assert.equal((await terminal()).state, "completed"); + // A provider failure reaches Paseo classified, without the provider's text (ADR 23). + await prompt("quota"); + const quota = await terminal(); + assert.equal(quota.state, "failed"); + assert.equal(quota.error.code, "MODEL_RATE_LIMITED"); + assert.match(quota.error.message, /\(HTTP 429, provider code 1310\)/u); + assert.equal( + JSON.parse(quota.error.diagnostic).failure.reason, + "rate_limited", + ); + assert.ok(!JSON.stringify(quota).includes(QUOTA_MESSAGE)); + await prompt("bash"); + assert.equal((await terminal()).state, "completed"); await prompt("command", "/plan"); assert.equal( events.find((e) => e.type === "session.prompt_result").result.type, @@ -625,6 +648,8 @@ try { ...(guarded ? ["guarded risky-command approval"] : []), "targeted stop", "restart after stop", + "provider quota failure", + "restart after failure", "slash command completion", "session listing v3", "EOF cleanup", diff --git a/scripts/check-upstream.mjs b/scripts/check-upstream.mjs index 5c63ab3..0e95f2e 100644 --- a/scripts/check-upstream.mjs +++ b/scripts/check-upstream.mjs @@ -249,6 +249,33 @@ try { await completeTurn(current); await queuedDone; current.deferConsumption = false; + // A classified model failure reaches Paseo's turn_failed without ZCode's text (ADR 23). + const failRun = await resumed.startTurn("Fail", { + clientMessageId: "fail", + }); + const failedEvent = waitForCompletion( + resumed, + failRun.turnId, + "turn_failed", + ); + await current.finish("failed", { + code: "1310", + message: "native-provider-text", + recoverable: true, + at: 3, + source: "provider", + attribution: { + source: "provider", + reason: "rate_limited", + statusCode: 429, + providerErrorCode: "1310", + }, + }); + const failed = await failedEvent; + assert.equal(failed.code, "MODEL_RATE_LIMITED"); + assert.match(failed.error, /\(HTTP 429, provider code 1310\)/u); + assert.equal(JSON.parse(failed.diagnostic).failure.reason, "rate_limited"); + assert.ok(!JSON.stringify(failed).includes("native-provider-text")); const stopRun = await resumed.startTurn("Stop", { clientMessageId: "stop", }); @@ -275,6 +302,7 @@ try { textSteering: "passed", attachmentQueue: "passed", singleCompletion: "passed", + turnFailure: "passed", providerReplacement: "passed", persistenceResume: "passed", planSettingsResume: "passed", @@ -559,13 +587,13 @@ async function checkContributions({ }; } -function waitForCompletion(session, turnId) { +function waitForCompletion(session, turnId, type = "turn_completed") { return new Promise((resolve, reject) => { const unsubscribe = session.subscribe((event) => { - if (event.type === "turn_completed" && event.turnId === turnId) { + if (event.type === type && event.turnId === turnId) { clearTimeout(timer); unsubscribe(); - resolve(); + resolve(event); } }); const timer = setTimeout(() => { diff --git a/server/diagnostics.ts b/server/diagnostics.ts index 66ec7e4..5d46aa9 100644 --- a/server/diagnostics.ts +++ b/server/diagnostics.ts @@ -15,6 +15,50 @@ const stages = [ "notification", "transport", "session", + "turn", +] as const; +// ZCode's ModelFailureReason (apps/zcode-cli/packages/contracts/src/model/index.ts at 3c5b227). +export const modelFailureReasons = [ + "rate_limited", + "provider_overloaded", + "server_error", + "network_error", + "timeout", + "stream_idle_timeout", + "stale_connection", + "auth_refresh", + "reasoning_signature_repair", + "offpeak_queued", + "auth_failed", + "cancelled", + "context_exceeded", + "invalid_request", + "provider_not_configured", + "proxy_error", + "tls_error", + "unknown", +] as const; +export type ModelFailureReason = (typeof modelFailureReasons)[number]; +// V4 errorAttributionSchema enums. +const failureSources = ["provider", "runtime", "tool", "network"] as const; +const errorPhases = [ + "prepare", + "configuration", + "connect", + "response", + "stream", + "parse", + "validation", + "unhandled", +] as const; +const exceptionKinds = [ + "api_call", + "generic", + "protocol", + "provider_business", + "transport", + "type_error", + "validation", ] as const; const checks = [ "bundle-metadata", @@ -146,6 +190,44 @@ export interface RuntimeDiagnostic { readonly nativeCode?: number; readonly exitCode?: number; readonly validation?: readonly { path: string; code: string }[]; + readonly failure?: TurnFailureDiagnostic; +} + +/** Attribution of a failed turn from V4 `control.lastError`; never its message. */ +export interface TurnFailureDiagnostic { + readonly source?: string; + readonly reason?: string; + readonly errorPhase?: string; + readonly exceptionKind?: string; + readonly statusCode?: number; + readonly providerErrorCode?: string; + readonly retryable?: boolean; +} + +export function reportableStatusCode( + value: number | undefined, +): number | undefined { + return Number.isInteger(value) && value! >= 100 && value! <= 599 + ? value + : undefined; +} + +/** Provider error codes are free text. Only numeric or word codes such as `1310` or `insufficient_quota` are reported. */ +export function reportableProviderErrorCode( + value: string | undefined, +): string | undefined { + return value && + value.length <= 64 && + /^(?:\d+|[A-Za-z]+(?:[_.][A-Za-z]+)*)$/u.test(value) + ? value + : undefined; +} + +function allowed( + values: readonly string[], + value: string | undefined, +): string | undefined { + return value !== undefined && values.includes(value) ? value : undefined; } export function runtimeDiagnostic( @@ -222,6 +304,21 @@ export function formatDiagnostic(error: unknown): string { for (const key of ["nativeCode", "exitCode"] as const) { if (Number.isSafeInteger(context[key])) result[key] = context[key]; } + if (context.failure) { + const f = context.failure; + const failure = Object.fromEntries( + Object.entries({ + source: allowed(failureSources, f.source), + reason: allowed(modelFailureReasons, f.reason), + errorPhase: allowed(errorPhases, f.errorPhase), + exceptionKind: allowed(exceptionKinds, f.exceptionKind), + statusCode: reportableStatusCode(f.statusCode), + providerErrorCode: reportableProviderErrorCode(f.providerErrorCode), + retryable: typeof f.retryable === "boolean" ? f.retryable : undefined, + }).filter(([, value]) => value !== undefined), + ); + if (Object.keys(failure).length) result.failure = failure; + } if (context.validation) result.validation = context.validation.slice(0, 10).map((issue) => ({ path: issue.path diff --git a/server/errors.ts b/server/errors.ts index 9171d86..fd94b5b 100644 --- a/server/errors.ts +++ b/server/errors.ts @@ -14,6 +14,13 @@ export type ZCodeErrorCode = | "NATIVE_INPUT_REJECTED" | "NATIVE_TIMEOUT" | "NATIVE_EXITED" + | "NATIVE_TURN_FAILED" + | "MODEL_RATE_LIMITED" + | "MODEL_AUTH_FAILED" + | "MODEL_CONTEXT_EXCEEDED" + | "MODEL_NETWORK_ERROR" + | "MODEL_PROVIDER_ERROR" + | "MODEL_REQUEST_FAILED" | "AUTH_REQUIRED" | "UNSUPPORTED_CONTENT" | "SESSION_NOT_FOUND" diff --git a/server/provider.ts b/server/provider.ts index 27ba7c8..ee4c56e 100644 --- a/server/provider.ts +++ b/server/provider.ts @@ -727,7 +727,7 @@ export class ZCodeConnection implements ProviderConnection { state: "failed", error: { code: event.code, - message: "ZCode turn failed", + message: event.error, diagnostic: event.diagnostic, }, }); diff --git a/server/session.ts b/server/session.ts index 3b5ee4b..a2477db 100644 --- a/server/session.ts +++ b/server/session.ts @@ -50,6 +50,7 @@ import type { } from "./session-types.js"; import { AdapterError } from "./errors.js"; import { diagnosticError, formatDiagnostic } from "./diagnostics.js"; +import { turnFailure } from "./turn-failure.js"; interface Admission { clientMessageId?: string; accepted: boolean; @@ -378,23 +379,29 @@ export class ZCodeSession { headers.some((h) => h.state === "failed") || this.state.control.phase === "error"; this.active = undefined; - this.emit( - failed - ? { - type: "turn_failed", - turnId: run.id, - error: "ZCode execution failed", - code: "NATIVE_PROTOCOL_ERROR", - } - : { - type: - run.cancelled || - headers.some((h) => h.state === "completedInterrupted") - ? "turn_canceled" - : "turn_completed", - turnId: run.id, - }, - ); + if (failed) { + // ZCode sets lastError in the frame that fails the turn and clears it + // when the next turn starts. + const failure = turnFailure( + this.state.control.lastError, + this.bridge.diagnostic, + ); + this.emit({ + type: "turn_failed", + turnId: run.id, + error: failure.message, + code: failure.code, + diagnostic: formatDiagnostic(failure), + }); + } else + this.emit({ + type: + run.cancelled || + headers.some((h) => h.state === "completedInterrupted") + ? "turn_canceled" + : "turn_completed", + turnId: run.id, + }); this.changed(); } private changed(): void { diff --git a/server/turn-failure.test.ts b/server/turn-failure.test.ts new file mode 100644 index 0000000..273ef5a --- /dev/null +++ b/server/turn-failure.test.ts @@ -0,0 +1,128 @@ +import { expect, test } from "vitest"; +import type { SessionErrorInfo } from "./vendor/zcode/packages/shared/src/zcode-protocol-v4/snapshot.js"; +import { formatDiagnostic } from "./diagnostics.js"; +import { turnFailure } from "./turn-failure.js"; + +function lastError( + attribution: SessionErrorInfo["attribution"], + source: SessionErrorInfo["source"] = "provider", +): SessionErrorInfo { + return { + code: "secret-code", + message: "secret-message", + recoverable: true, + at: 1, + source, + detail: "secret-detail", + underlyingErrorMessage: "secret-underlying", + attribution, + }; +} + +test.each([ + ["rate_limited", "MODEL_RATE_LIMITED"], + ["offpeak_queued", "MODEL_RATE_LIMITED"], + ["auth_failed", "MODEL_AUTH_FAILED"], + ["provider_not_configured", "MODEL_AUTH_FAILED"], + ["context_exceeded", "MODEL_CONTEXT_EXCEEDED"], + ["timeout", "MODEL_NETWORK_ERROR"], + ["tls_error", "MODEL_NETWORK_ERROR"], + ["server_error", "MODEL_PROVIDER_ERROR"], + ["provider_overloaded", "MODEL_PROVIDER_ERROR"], + ["invalid_request", "MODEL_REQUEST_FAILED"], + ["unknown", "MODEL_REQUEST_FAILED"], + ["secret-reason", "MODEL_REQUEST_FAILED"], +])("classifies the provider reason %s as %s", (reason, code) => { + const failure = turnFailure(lastError({ source: "provider", reason }), {}); + expect(failure.code).toBe(code); + expect(failure.message).not.toContain("secret"); + expect(formatDiagnostic(failure)).not.toContain("secret"); +}); + +test("falls back to the source when the reason is missing", () => { + expect(turnFailure(lastError(undefined, "network"), {}).code).toBe( + "MODEL_NETWORK_ERROR", + ); + expect(turnFailure(lastError(undefined, "provider"), {}).code).toBe( + "MODEL_REQUEST_FAILED", + ); + // The attribution's source wins over the coarse lastError source. + expect( + turnFailure(lastError({ source: "network" }, "runtime"), {}).code, + ).toBe("MODEL_NETWORK_ERROR"); +}); + +test("does not attribute runtime or tool failures to the model provider", () => { + for (const failure of [ + turnFailure(lastError({ reason: "hook_blocked" }, "runtime"), {}), + turnFailure(lastError({ reason: "timeout" }, "tool"), {}), + turnFailure(null, {}), + ]) { + expect(failure.code).toBe("NATIVE_TURN_FAILED"); + expect(failure.message).toBe("ZCode turn failed."); + } +}); + +test("adds only reportable status and provider codes to the message", () => { + expect( + turnFailure( + lastError({ + source: "provider", + reason: "rate_limited", + statusCode: 429, + providerErrorCode: "insufficient_quota", + }), + {}, + ).message, + ).toMatch(/\(HTTP 429, provider code insufficient_quota\)\. Wait/u); + const key = "0123456789abcdef0123456789abcdef.SecretKey123"; + const failure = turnFailure( + lastError({ + source: "provider", + reason: "auth_failed", + statusCode: 401, + providerErrorCode: key, + }), + {}, + ); + expect(failure.message).toMatch( + /^The model provider is not signed in or rejected the credentials \(HTTP 401\)\./u, + ); + expect(formatDiagnostic(failure)).not.toContain(key); +}); + +test("reports the turn stage and allowlisted attribution with the runtime context", () => { + const failure = turnFailure( + lastError({ + source: "provider", + reason: "rate_limited", + errorPhase: "stream", + exceptionKind: "provider_business", + providerId: "secret-provider", + modelId: "secret-model", + providerKind: "secret-kind", + transport: "sse", + statusCode: 429, + providerErrorCode: "1310", + retryable: false, + }), + { appVersion: "3.16.2", platform: "linux-x64", stage: "session" }, + ); + const diagnostic = formatDiagnostic(failure); + expect(JSON.parse(diagnostic)).toMatchObject({ + code: "MODEL_RATE_LIMITED", + appVersion: "3.16.2", + platform: "linux-x64", + stage: "turn", + failure: { + source: "provider", + reason: "rate_limited", + errorPhase: "stream", + exceptionKind: "provider_business", + statusCode: 429, + providerErrorCode: "1310", + retryable: false, + }, + }); + expect(diagnostic).not.toContain("secret"); +}); diff --git a/server/turn-failure.ts b/server/turn-failure.ts new file mode 100644 index 0000000..b701074 --- /dev/null +++ b/server/turn-failure.ts @@ -0,0 +1,111 @@ +import type { SessionErrorInfo } from "./vendor/zcode/packages/shared/src/zcode-protocol-v4/snapshot.js"; +import { AdapterError, type ZCodeErrorCode } from "./errors.js"; +import { + modelFailureReasons, + reportableProviderErrorCode, + reportableStatusCode, + type ModelFailureReason, + type RuntimeDiagnostic, +} from "./diagnostics.js"; + +type TurnFailureCode = Extract< + ZCodeErrorCode, + "NATIVE_TURN_FAILED" | `MODEL_${string}` +>; + +// Reasons without an entry fall back to the attribution source. +const reasonCodes: Partial> = { + rate_limited: "MODEL_RATE_LIMITED", + offpeak_queued: "MODEL_RATE_LIMITED", + auth_failed: "MODEL_AUTH_FAILED", + auth_refresh: "MODEL_AUTH_FAILED", + provider_not_configured: "MODEL_AUTH_FAILED", + context_exceeded: "MODEL_CONTEXT_EXCEEDED", + network_error: "MODEL_NETWORK_ERROR", + timeout: "MODEL_NETWORK_ERROR", + stream_idle_timeout: "MODEL_NETWORK_ERROR", + stale_connection: "MODEL_NETWORK_ERROR", + proxy_error: "MODEL_NETWORK_ERROR", + tls_error: "MODEL_NETWORK_ERROR", + server_error: "MODEL_PROVIDER_ERROR", + provider_overloaded: "MODEL_PROVIDER_ERROR", +}; + +const messages: Record< + TurnFailureCode, + readonly [summary: string, advice?: string] +> = { + NATIVE_TURN_FAILED: ["ZCode turn failed"], + MODEL_RATE_LIMITED: [ + "The model provider's usage or rate limit was reached", + "Wait for it to reset or switch models. For a ZCode Coding Plan, Paseo 0.11 and later show the reset times on the Usage screen.", + ], + MODEL_AUTH_FAILED: [ + "The model provider is not signed in or rejected the credentials", + "Check the account and model providers in the Account tab of the zcode-provider Settings.", + ], + MODEL_CONTEXT_EXCEEDED: [ + "The conversation exceeds the model's context window", + "Start a new session or switch to a model with a larger context window.", + ], + MODEL_NETWORK_ERROR: [ + "ZCode could not reach the model provider", + "Check the network connection and try again.", + ], + MODEL_PROVIDER_ERROR: [ + "The model provider returned a server error or is overloaded", + "Try again later.", + ], + MODEL_REQUEST_FAILED: ["The model provider rejected the request"], +}; + +/** + * Classifies a failed turn by the attribution of V4 `control.lastError` + * (ADR 23). ZCode's message and detail are never used: they can carry + * provider text, request content or credentials (ADR 7). + */ +export function turnFailure( + lastError: SessionErrorInfo | null, + context: RuntimeDiagnostic, +): AdapterError { + const attribution = lastError?.attribution; + const source = attribution?.source ?? lastError?.source; + const reason = modelFailureReasons.find((r) => r === attribution?.reason); + const code: TurnFailureCode = + source === "provider" || source === "network" + ? ((reason && reasonCodes[reason]) ?? + (source === "network" ? "MODEL_NETWORK_ERROR" : "MODEL_REQUEST_FAILED")) + : "NATIVE_TURN_FAILED"; + const statusCode = reportableStatusCode(attribution?.statusCode); + const providerErrorCode = reportableProviderErrorCode( + attribution?.providerErrorCode, + ); + const facts = [ + ...(statusCode ? [`HTTP ${statusCode}`] : []), + ...(providerErrorCode ? [`provider code ${providerErrorCode}`] : []), + ]; + const [summary, advice] = messages[code]; + return new AdapterError( + code, + `${summary}${facts.length ? ` (${facts.join(", ")})` : ""}.${advice ? ` ${advice}` : ""}`, + {}, + undefined, + { + ...context, + stage: "turn", + ...(lastError + ? { + failure: { + source, + reason: attribution?.reason, + errorPhase: attribution?.errorPhase, + exceptionKind: attribution?.exceptionKind, + statusCode: attribution?.statusCode, + providerErrorCode: attribution?.providerErrorCode, + retryable: attribution?.retryable, + }, + } + : {}), + }, + ); +} diff --git a/test/fake-host.ts b/test/fake-host.ts index cb43c09..7ce648c 100644 --- a/test/fake-host.ts +++ b/test/fake-host.ts @@ -2,7 +2,10 @@ import { z } from "zod"; import type { HostBridge, HostSubscription } from "../server/host/bridge.js"; import { AdapterError } from "../server/errors.js"; import { applyConversationDeltas } from "../server/vendor/zcode/packages/shared/src/zcode-protocol-v4/apply.js"; -import { conversationSnapshotSchema } from "../server/vendor/zcode/packages/shared/src/zcode-protocol-v4/snapshot.js"; +import { + conversationSnapshotSchema, + type SessionErrorInfo, +} from "../server/vendor/zcode/packages/shared/src/zcode-protocol-v4/snapshot.js"; import type { ConversationDelta } from "../server/vendor/zcode/packages/shared/src/zcode-protocol-v4/delta.js"; import type { ConversationRow, @@ -200,7 +203,10 @@ export class FakeBridge implements HostBridge { }, ]); } - async finish(state: TurnHeaderRow["state"] = "completedSuccess") { + async finish( + state: TurnHeaderRow["state"] = "completedSuccess", + lastError?: SessionErrorInfo, + ) { const header = this.state.rows.window.find( (r): r is TurnHeaderRow => r.kind === "turnHeader" && r.productTurnId === this.activeProductTurn, @@ -222,6 +228,7 @@ export class FakeBridge implements HostBridge { : state, canStop: false, activeWorks: [], + ...(lastError ? { lastError } : {}), }, usage: { contextWindow: { diff --git a/test/provider.test.ts b/test/provider.test.ts index 25d8267..7e2e94f 100644 --- a/test/provider.test.ts +++ b/test/provider.test.ts @@ -312,6 +312,72 @@ it("cancels unconsumed queue input after a native failure and finishes once", as f.events.filter((e) => e.type === "session.turn" && e.state === "failed"), ).toHaveLength(1); }); +it("reports a provider quota failure by its attribution without ZCode's text", async () => { + const f = await fixture(); + await f.open(); + await f.prompt(); + // The shape the 3.16.2 runtime reported for a GLM weekly quota (429/1310). + const providerText = + "您已达到每周/每月使用上限,您的限额将在 2026-11-07 16:34:13 重置。"; + await f.host.finish("failed", { + code: "1310", + message: providerText, + recoverable: true, + at: 2, + source: "provider", + traceId: "trace-1", + detail: `Turn execution failed\nprovider=account:bigmodel model=GLM-5.3-Flash request=request-1 reason=rate_limited status=429`, + underlyingErrorMessage: providerText, + attribution: { + source: "provider", + reason: "rate_limited", + errorPhase: "stream", + exceptionKind: "provider_business", + providerId: "account:bigmodel", + modelId: "GLM-5.3-Flash", + statusCode: 429, + providerErrorCode: "1310", + retryable: false, + }, + }); + const failed = await f.wait( + (e) => e.type === "session.turn" && e.state === "failed", + ); + if (failed.type !== "session.turn" || !failed.error) + throw new Error("Expected a failed turn"); + expect(failed.error.code).toBe("MODEL_RATE_LIMITED"); + expect(failed.error.message).toMatch( + /^The model provider's usage or rate limit was reached \(HTTP 429, provider code 1310\)\./u, + ); + expect(JSON.parse(failed.error.diagnostic!)).toMatchObject({ + code: "MODEL_RATE_LIMITED", + stage: "turn", + failure: { + source: "provider", + reason: "rate_limited", + errorPhase: "stream", + exceptionKind: "provider_business", + statusCode: 429, + providerErrorCode: "1310", + retryable: false, + }, + }); + const output = JSON.stringify(failed); + for (const native of [providerText, "bigmodel", "request-1", "trace-1"]) + expect(output).not.toContain(native); +}); +it("keeps the generic turn failure when ZCode reports no error", async () => { + const f = await fixture(); + await f.open(); + await f.prompt(); + await f.host.finish("failed"); + const failed = await f.wait( + (e) => e.type === "session.turn" && e.state === "failed", + ); + expect(failed).toMatchObject({ + error: { code: "NATIVE_TURN_FAILED", message: "ZCode turn failed." }, + }); +}); it("renders configuration markers without inventing a product turn", async () => { const f = await fixture(); await f.open();