runtime_next: connector terminal errors are masked by ResourceExhausted / h2 protocol errors, which also occur independently

[jacobmarble]

Summary

When a materialization connector exits from a terminal error, the shard failure surfaced
to the user is frequently a generic transport error rather than the connector's own
message. Separately, the same transport error appears on shard failures where no
connector-side error exists at all, so it is not purely a masking artifact — something is
genuinely exhausting an HTTP/2 resource.

Both were observed on AltopEnergy/MarketData/materialize-clickhouse on 2026-08-11, and
the masking materially slowed down diagnosing a data-loss bug
(estuary/connectors#5044): anyone starting from flowctl catalog status saw a transport
error and had to know to go read the connector's own logs to find the real cause.

1. The connector's terminal error is masked

Two messages, in this order:

warn  runtime_next::shard::materialize::actor
      Materialize error (expected leader message) from leader
      Caused by: status: 'Some resource has been exhausted', self: "h2 protocol error: http2 error"

error runTransactions: readMessage: rpc error: code = ResourceExhausted desc = h2 protocol error: http2 error

The controller alert surfaced only the second:

"Observed 6 recent task shard failures, the latest at 2026-08-11 19:56:22 UTC. Last failure:
 runTransactions: readMessage: rpc error: code = ResourceExhausted desc = h2 protocol error: http2 error"

The actual cause, present in the task's logs at 18:54:16, was:

commit failed: moving stage to target marketdata_dartpricesieso: refusing to commit store
table: it contains 9 staged rows but 300699 rows were stored; not moving partitions so
staged rows are preserved for retry

That message is the one an operator needs. It names the table, the discrepancy, and the
fact that data was withheld rather than committed — and it never reached the shard's
failure status.

Requested: when a connector reports a terminal error and the runtime subsequently hits
a transport error on the same stream, propagate the connector's error as the shard failure
cause. The transport error is a consequence of the connector's exit, so it should not win
the race to become the reported failure.

2. The same error also occurs independently

On the same task, shard failures at 19:18:35 and 19:56:22 carry this error with
no connector-side error at those times — the connector's guard last fired at 19:17:25.
So this is not only a masking artifact; some HTTP/2 resource is genuinely being exhausted.

The task has 220 bindings, which makes an oversized message or a flow-control limit a
plausible suspect (a Loaded/Flushed response, or a StartedCommit state update,
scaling with binding count). I did not confirm that and am not guessing further.

Requested: determine what exhausts the h2 resource on a high-binding-count
materialization, and whether the limit should be raised, the messages chunked, or the
condition surfaced as something more actionable than ResourceExhausted.

Reproduction context

• Task: AltopEnergy/MarketData/materialize-clickhouse, runtime v2, 220 bindings.
• Window: 2026-08-11 18:54–19:56 UTC. The task's logs for that window contain both
  patterns, and flowctl catalog status shows the alert text quoted above.
• The connector-side data-loss bug that was being masked is
  materialize-clickhouse: sorting-key mismatch silently destroys rows, and the recovery path commits it anyway (+ log cleanup) connectors#5044; it is fixed in materialize-clickhouse: do not commit transactions whose staged rows do not add up connectors#5053, but the masking is
  independent of that fix and will affect the next connector failure the same way.