KAFKA-19981: Handle retriable remote storage exception in RemoteLogManager

[DL1231]

This PR distinguishes between RemoteStorageException and
RetriableRemoteStorageException in RemoteLogManager to handle
temporary storage degradations gracefully:

1. Copy path: Avoids deleting partially uploaded segments when
   RetriableRemoteStorageException is thrown;
2. Delete path: Skips incrementing failedRemoteDeleteRequestRate metric
   for retriable exceptions;
3. Documentation: Updates RemoteStorageManager Javadoc to clarify
   exception usage;
4. Testing: Adds UT for retriable scenarios.

[kamalcph]

Thanks for the patch! It looks good to me.

We can further update the RemoteLogManager javadoc to convey the contract to the plugin implementor.

@showuon Could you also review this PR? More details are on the ticket: KAFKA-19981. We want to take different actions in Kafka based on the error thrown by the plugin. Let me know your thoughts. Thanks!

[showuon]

I agree we should not treat the retriable error as a failure. Left some comments.

[showuon]

In L1058 below, when deleteRemoteLogSegment throws RetriableRemoteStorageException, we don't catch it. Then, the exception thrown will be the one from deleteRemoteLogSegment, not the one from copyLogSegmentData. With that, the RetriableRemoteStorageException from deleteRemoteLogSegment could be caught in L1009 above and not update the metrics as expected. Maybe we should catch all exceptions thrown from L1058 because we want to retry later?

[kamalcph]

We also have to handle the RetriableRemoteStorageException in RLMTask L844, so that it won't be logged as WARN log.

} catch (RetriableException | RetriableRemoteStorageException ex) {
                logger.debug("Encountered a retryable error while executing current task for partition {}", topicIdPartition, ex);
            }

[kamalcph]

@DL1231 Could you also address the above comment? Thanks!

[kamalcph]

Maybe we should catch all exceptions thrown from L1058 because we want to retry later?

since RRSE extends RSE, the current patch looks clean to me. (Open to suggestions)

[DL1231]

when deleteRemoteLogSegment throws RetriableRemoteStorageException, we don't catch it.

as @kamalcph pointed out, RRSE extends RSE, so when deleteRemoteLogSegment throws RRSE, it will be caught.

[kamalcph]

We also have to handle the RetriableRemoteStorageException in RLMTask L844, so that it won't be logged as WARN log.

} catch (RetriableException | RetriableRemoteStorageException ex) {
                logger.debug("Encountered a retryable error while executing current task for partition {}", topicIdPartition, ex);
            }

[kamalcph]

LGTM