Summary
connector_init::rpc logs i/o error writing to connector stdin at warn when the
runtime writes to a connector that has already exited from a terminal error. It is pure
downstream noise — the write fails because the connector is already gone — but it is
logged at the same level as real problems and competes with the causal error for
attention.
Observed in a materialize-clickhouse task on 2026-08-11: three
occurrences, one per connector exit, each immediately following the connector's own
terminal error. While diagnosing a data-loss bug (estuary/connectors#5044) these were
three of the handful of warn/error lines in the window, and none of them said anything
about what actually went wrong.
Requested
Suppress the message, or demote it to debug, when the connector has already reported a
fatal error or its process has already exited. A failed write to a dead process's stdin is
expected in that situation and carries no diagnostic value.
If the write can also fail for reasons unrelated to the connector having exited, keep
warn for that case and distinguish the two — the point is that the expected case should
not look like a problem.
Context
Filed out of estuary/connectors#5044 (Part 3b), which tracked the log noise encountered
while diagnosing the incident. Related: the more consequential problem in that window was
the connector's terminal error being replaced by a transport error in the shard's failure
status, filed separately.
Summary
connector_init::rpclogsi/o error writing to connector stdinatwarnwhen theruntime writes to a connector that has already exited from a terminal error. It is pure
downstream noise — the write fails because the connector is already gone — but it is
logged at the same level as real problems and competes with the causal error for
attention.
Observed in a
materialize-clickhousetask on 2026-08-11: threeoccurrences, one per connector exit, each immediately following the connector's own
terminal error. While diagnosing a data-loss bug (estuary/connectors#5044) these were
three of the handful of
warn/errorlines in the window, and none of them said anythingabout what actually went wrong.
Requested
Suppress the message, or demote it to
debug, when the connector has already reported afatal error or its process has already exited. A failed write to a dead process's stdin is
expected in that situation and carries no diagnostic value.
If the write can also fail for reasons unrelated to the connector having exited, keep
warnfor that case and distinguish the two — the point is that the expected case shouldnot look like a problem.
Context
Filed out of estuary/connectors#5044 (Part 3b), which tracked the log noise encountered
while diagnosing the incident. Related: the more consequential problem in that window was
the connector's terminal error being replaced by a transport error in the shard's failure
status, filed separately.