[Feature Request] Retry `PERMISSION_DENIED` on established connections

[chris-olszewski]

Describe the solution you'd like

Retry PERMISSION_DENIED gRPC calls if they happen to an established connection in case there's a temporary change in permissions.

We keep avoiding retrying these status on initial connection since there's no proof that the credentials were ever valid.

Probably needs to be configurable in some manner.

Additional context

[sachinsharma3191]

Background

Today PERMISSION_DENIED is never retried. In crates/client/src/retry.rs:

• RETRYABLE_ERROR_CODES (retry.rs:18) is [DataLoss, Internal, Unknown, ResourceExhausted, Aborted, OutOfRange, Unavailable] — no PermissionDenied.
• TonicErrorHandler::handle (retry.rs:274) therefore falls through to RetryPolicy::ForwardError (retry.rs:355) for PermissionDenied.

This is intentional on the initial connection: we can't prove the credentials were ever valid, so retrying a permanent auth failure would just spin. The gap is a transient permission change on a connection that was previously working.

Proposal

Retry PERMISSION_DENIED only when both hold:

1. the client is "established" — it has completed at least one successful RPC since construction, and
2. a new opt-in config flag enables it.

Default behavior is unchanged (no retry).

Tracking "established"

Add an Arc<AtomicBool> established on the retry client, set to true on the first Ok from the FutureRetry wrapper (call_with_retry, retry.rs:149). Clone it into TonicErrorHandler (struct at retry.rs:200).

Decision change

In handle(), before the final fall-through:

if e.code() == Code::PermissionDenied
    && self.retry_permission_denied_when_established
    && self.established.load(Ordering::Relaxed)
{
    return match self.backoff.next_backoff() {
        Some(b) => RetryPolicy::WaitRetry(b),
        None => RetryPolicy::ForwardError(e),
    };
}

Config surface

New flag retry_permission_denied_when_established: bool (default false).
Open question: should this live on RetryOptions (retry.rs:32) or on the higher-level client/connect options? Happy to go either way.

Open questions

• Should we also cover Unauthenticated? My inclination is no — that is credential-validity and is better handled by credential refresh, not retry. Keeping scope to PermissionDenied.
• Does "established" reset on reconnect (GOAWAY / AZ failover), or stay sticky for the client's lifetime? Sticky is simplest and matches the "was it ever authorized" intent — but happy to make it per-connection.
• Should retries here be bounded by a shorter grace window (like the long-poll fatal grace at retry.rs:348) so a genuine permanent revocation gives up fast?

Testing

Unit tests in retry.rs mirroring long_poll_non_retryable_errors (retry.rs:398):

• flag on + established → PermissionDenied yields WaitRetry
• flag on + not established → ForwardError
• flag off → ForwardError (proves default unchanged)

Semver: minor (new public config field).

Happy to adjust any of this based on your preferences before I start — in particular the config location and the sticky-vs-per-connection question.