Skip to content

fix(wrangler): keep wrangler dev alive when forwarding one request to the Worker fails - #15448

Open
LiyanChen-X wants to merge 6 commits into
cloudflare:mainfrom
LiyanChen-X:fix/proxyworker-abandoned-queued-request
Open

fix(wrangler): keep wrangler dev alive when forwarding one request to the Worker fails#15448
LiyanChen-X wants to merge 6 commits into
cloudflare:mainfrom
LiyanChen-X:fix/proxyworker-abandoned-queued-request

Conversation

@LiyanChen-X

@LiyanChen-X LiyanChen-X commented Sep 1, 2026

Copy link
Copy Markdown

Fixes #15447. Also removes the crash reported in #15203 (verified with the same shape of repro, see below); the signature is the same one as #15317 / #4562.

What this does

ProxyWorker forwards every incoming request to the UserWorker with fetch(). When that fetch() rejects while the request still targets the current UserWorker, the rejection was sent to the ProxyController as { type: "error" }, which is fatal: wrangler dev exits.

The most common way to get such a rejection has nothing to do with the proxy or the Worker: the client disconnected while its request body was still being uploaded, so workerd fails the read of the body (Network connection lost. / Can't read from request stream because client disconnected). A single curl -m 0.4 -X POST --data-binary @2mb.bin http://127.0.0.1:8787/upload against an otherwise idle wrangler dev is enough to take the whole session down. Requests the proxy queued during startup/reload and that were abandoned before they could be replayed (the scenario in #15447) are the same failure, just later.

A rejected forward is the outcome of that one request, not a defect in the proxy. This PR:

  • wraps the rejection of the UserWorker fetch() in a UserWorkerFetchError marker so it can be told apart from errors thrown while post-processing the response (checkForPreviewTokenError, live-reload injection, …);
  • answers such a request with a 502 (Could not proxy this request to your Worker: <message>) and logs it through the existing debug-log channel, instead of failing the session;
  • leaves post-processing errors on the existing fatal path, and leaves the stale-UserWorker branch (GET/HEAD requeue, 503 for the rest) untouched.

The discrimination is structural (which promise rejected), not based on error message text.

Why not request.signal

request.signal.aborted would be the principled signal for "the client is gone", so I tried it first: it stays false for these requests, both before the replay and inside the rejection handler. The ProxyWorker runs with compatibilityFlags: ["nodejs_compat"], without enable_request_signal; adding enable_request_signal and request_signal_passthrough to the ProxyWorker alone does not change that either, because the request reaches the ProxyWorker through Miniflare's core entry worker (connect_pass_through only), so the abort never crosses that hop. Making the signal usable would mean turning request_signal_passthrough on for Miniflare's entry worker, which changes cancellation semantics for every user Worker in local dev — a much larger change than this bug warrants. And a rejected forward should not be fatal regardless of why it rejected.

Verification

All against a locally built wrangler from this branch vs main, on macOS with wrangler 4.127.1 / workerd 1.20260828.1.

Scenario main this branch
idle wrangler dev, curl -m 0.4 -X POST --data-binary @2mb.bin /upload process exits on the 1st request (Error in ProxyController: Error inside ProxyWorker, Network connection lost.) alive after 3 rounds, each answered 502, session keeps serving
reload in progress (touch the entry file), same aborted POST while queued (#15447) process exits alive after 3 rounds
assets.directory configured, 100 KB POSTs to an existing asset path (#15203 shape) process exits after 7 requests (405 500 405 500 405 000 000) alive after 60 requests (405/502 alternating, no exit)
new e2e client disconnects > does not exit wrangler dev when a client aborts a request mid-body connect ECONNREFUSED — the dev process exited

Note on the assets row: the alternating 405/502 on main's 405/500 is the pre-existing keep-alive behaviour after an unconsumed body (cf. #14641); this PR only stops it from being fatal.

The e2e destroys the TCP connection mid-body with a raw node:net socket (Content-Length: 2000000, one 64 KB chunk, destroy()), because an AbortController abort on an undici streaming body ends the request gracefully and does not reproduce the failure. pnpm check:type, tsc -p e2e/tsconfig.json, oxlint --type-aware and oxfmt pass on the touched files.

Notes for reviewers

  • This changes wrangler dev behaviour only for the case that previously terminated the process; a 502 with the underlying message is what the client (if it is still there) now sees.
  • Chosen over "make the error message type non-fatal in DevEnv.handleErrorEvent" because that would also swallow genuine proxy bugs; the marker keeps post-processing errors loud.
  • Prepared with an AI coding agent (Claude Code) working on my machine; the reproductions and numbers above come from those runs and I reviewed the change before opening this.

  • Tests
    • Tests included/updated
    • Automated tests not possible - manual testing has been completed as follows:
    • Additional testing not necessary because:
  • Public documentation
    • Cloudflare docs PR(s):
    • Documentation not necessary because: this only removes a crash in wrangler dev; no user-facing configuration or documented behaviour changes.

@changeset-bot

changeset-bot Bot commented Sep 1, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 339dd54

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
wrangler Patch
@cloudflare/vite-plugin Patch
@cloudflare/vitest-plugin Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-project-automation github-project-automation Bot moved this to Untriaged in workers-sdk Sep 1, 2026
@LiyanChen-X
LiyanChen-X force-pushed the fix/proxyworker-abandoned-queued-request branch from 2177181 to 15e40ed Compare September 1, 2026 03:32
@pkg-pr-new

pkg-pr-new Bot commented Sep 1, 2026

Copy link
Copy Markdown
@cloudflare/autoconfig

npm i https://pkg.pr.new/@cloudflare/autoconfig@15448

@cloudflare/build-output-utils

npm i https://pkg.pr.new/@cloudflare/build-output-utils@15448

@cloudflare/codemods

npm i https://pkg.pr.new/@cloudflare/codemods@15448

@cloudflare/config

npm i https://pkg.pr.new/@cloudflare/config@15448

create-cloudflare

npm i https://pkg.pr.new/create-cloudflare@15448

@cloudflare/deploy-helpers

npm i https://pkg.pr.new/@cloudflare/deploy-helpers@15448

@cloudflare/kv-asset-handler

npm i https://pkg.pr.new/@cloudflare/kv-asset-handler@15448

miniflare

npm i https://pkg.pr.new/miniflare@15448

@cloudflare/pages-functions

npm i https://pkg.pr.new/@cloudflare/pages-functions@15448

@cloudflare/pages-shared

npm i https://pkg.pr.new/@cloudflare/pages-shared@15448

@cloudflare/unenv-preset

npm i https://pkg.pr.new/@cloudflare/unenv-preset@15448

@cloudflare/vite-plugin

npm i https://pkg.pr.new/@cloudflare/vite-plugin@15448

@cloudflare/vitest-plugin

npm i https://pkg.pr.new/@cloudflare/vitest-plugin@15448

@cloudflare/workers-auth

npm i https://pkg.pr.new/@cloudflare/workers-auth@15448

@cloudflare/workers-editor-shared

npm i https://pkg.pr.new/@cloudflare/workers-editor-shared@15448

@cloudflare/workers-utils

npm i https://pkg.pr.new/@cloudflare/workers-utils@15448

wrangler

npm i https://pkg.pr.new/wrangler@15448

commit: 339dd54

…to the Worker fails

The ProxyWorker treated a rejected `fetch()` to the UserWorker as a fatal
ProxyWorker error, which made the whole dev session exit. The common way to
hit it is a client that disconnects while its request body is still being
uploaded ("Network connection lost."), including a request that was queued
during startup/reload and abandoned before it could be replayed.

A rejected forward is the outcome of that single request, not a proxy
defect: answer it with a 502, log at debug level and keep the session
running. Errors thrown while post-processing a response keep the existing
fatal handling.

Adds an e2e that destroys the connection mid-upload against a real
`wrangler dev` process and asserts the process survives (fails on main
with ECONNREFUSED because the process exited).
@LiyanChen-X
LiyanChen-X force-pushed the fix/proxyworker-abandoned-queued-request branch from 15e40ed to bae544f Compare September 1, 2026 04:13
@LiyanChen-X LiyanChen-X changed the title fix(wrangler): do not exit dev when a queued request was abandoned by its client fix(wrangler): keep wrangler dev alive when forwarding one request to the Worker fails Sep 1, 2026
@LiyanChen-X
LiyanChen-X marked this pull request as ready for review September 1, 2026 07:27
@workers-devprod
workers-devprod requested review from a team and dario-piotrowicz and removed request for a team September 1, 2026 07:28
@workers-devprod

Copy link
Copy Markdown
Contributor

Codeowners approval required for this PR:

  • @cloudflare/wrangler
Show detailed file reviewers
  • .changeset/proxyworker-per-request-fetch-failures.md: [@cloudflare/wrangler]
  • packages/wrangler/e2e/dev.test.ts: [@cloudflare/wrangler]
  • packages/wrangler/templates/startDevWorker/ProxyWorker.ts: [@cloudflare/wrangler]

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 1 flag

Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

@sarbogergely

Copy link
Copy Markdown

Tested the pkg.pr.new build of this PR against the setup from #15317 (single wrangler dev, Windows 11 workstation, no contention). Your repro kills 4.126.0 every time and this branch survives it every time — but the more interesting result came from running our ordinary browser suite on the branch, where the new non-fatal path fired on a GET with no request body.

1. Your aborted-upload repro, old vs. this branch

Your client disconnects e2e shape as a standalone script (raw node:net socket, Content-Length: 2000000, one 64 KB chunk, destroy(); Worker does await request.text() on POST). Three aborted POSTs per run, GET / health probe after each, three runs per build:

wrangler runs outcome
4.126.0 (workerd 1.20260825.1) 3 exits on the 1st aborted POST, 3/3. GET / refused afterwards
this PR (wrangler@f59fe68, workerd 1.20260831.1) 3 survives all 3 rounds, 3/3 — 9 aborted uploads, zero exits

A variant that keeps the socket open instead of destroying it gives the same split: process exits on 4.126.0, alive on this branch.

2. The part I think is new: it also fires on a bodyless GET

Five runs of our Playwright suite (~96 tests, ~2.5 min each, one dev server) on the branch. Four runs fully green; in the fifth, the new path fired once:

[ProxyWorker] Could not proxy request to the UserWorker: GET http://localhost:8788/api/version Network connection lost.
[wrangler:info] GET /dashboard          200 OK (70ms)
[wrangler:info] GET /api/<redacted>   200 OK (28ms)
[wrangler:info] GET /api/<redacted>   200 OK (82ms)
[wrangler:info] GET /api/version        502 Bad Gateway (92ms)     ← here
[wrangler:info] GET /api/<redacted>   200 OK (20ms)

The dev session stayed up and the suite ran to completion (95 passed). That request is a plain GET issued by the page — no request body at all, so "the client disconnected while its body was still uploading" cannot be the explanation here, and neither can the queued-during-reload case (the server had been serving for ~40 s, no reload in flight). The cause is nevertheless the same Network connection lost.

This seems to support the design argument in your PR description — "a rejected forward should not be fatal regardless of why it rejected" — with a case the description doesn't currently cover. On 4.126.0 the same rejection would have taken the type: "error" branch, which is what has been killing our sessions. If it helps, this is also a frequency data point: 1 such event in 5 suite runs, which matches the 1-in-5 death rate I measured on this machine on 2026-08-26.

3. One consequence worth a line in the changeset

For a browser-driven suite the 502 is not invisible: it surfaces as a page-level Failed to load resource: the server responded with a status of 502 (Bad Gateway). Our console guard treats any console.error as a failure, so that one run failed a single test — instead of losing the dev server and, with it, everything after. That is a strictly better failure mode and I am not asking for a change. But someone upgrading with a strict console/network assertion will see new 502s where they previously saw a dead server, and it is worth them knowing that is the same event, now survivable.

4. Log shapes — three different ones for the same cause

Worth flagging for anyone building log-based triage (we did, for our own tracking issue):

path stdout debug log
4.126.0, aborted upload [wrangler:error] Error: Network connection lost. + empty ✘ [ERROR] nothing about it — last entry is an ordinary [InspectorProxyWorker] message from before the request
4.126.0, the crash we caught 2026-08-25/26 empty ✘ [ERROR] only Error in ProxyController: Error inside ProxyWorker … cause: { message: 'Network connection lost.' }
this PR, bodyless GET 502 Bad Gateway in the request log [ProxyWorker] Could not proxy request to the UserWorker: … Network connection lost.

Our extractor keys on the cause block, so it silently finds nothing on the first shape.

5. Suite-level rates cannot show this PR's effect, and I would not use them

wrangler full runs runs with ≥1 dev-server exit
4.126.0 5 0
this PR 5 0 (one run had the converted 502 above)

⚠️ The baseline is 0, so this table demonstrates nothing about the fix — I include it only so a "0 vs 0" is not read as evidence either way, and as a check that the branch does not break an ordinary suite. The same suite on the same machine gave 1 death in 5 on 2026-08-26 and 0 in 5 today, which is consistent with @DamienGR's observation that a machine tends to be either healthy or persistently fatal. The deterministic repro in §1 and the converted 502 in §2 are the parts that actually discriminate.

Happy to run this on Linux CI as well, or with a larger N, if that would help before merging.

@workers-devprod

Copy link
Copy Markdown
Contributor

Codeowners approval required for this PR:

  • @cloudflare/wrangler
Show detailed file reviewers
  • .changeset/proxyworker-per-request-fetch-failures.md: [@cloudflare/wrangler]
  • packages/wrangler/e2e/dev.test.ts: [@cloudflare/wrangler]
  • packages/wrangler/templates/startDevWorker/ProxyWorker.ts: [@cloudflare/wrangler]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Untriaged

Development

Successfully merging this pull request may close these issues.

wrangler dev exits when a queued request is abandoned by its client during startup/reload (ProxyWorker treats replay failure as fatal)

3 participants