[WIP]feat: honor fs.s3a.auth.profile.name and fs.s3a.auth.profile.file for ProfileCredentialsProvider#4246
Closed
karuppayya wants to merge 1 commit intoapache:mainfrom
Closed
[WIP]feat: honor fs.s3a.auth.profile.name and fs.s3a.auth.profile.file for ProfileCredentialsProvider#4246karuppayya wants to merge 1 commit intoapache:mainfrom
karuppayya wants to merge 1 commit intoapache:mainfrom
Conversation
… ProfileCredentialsProvider Follow-up to apache#4213. Resolve the AWS profile name and credentials file on the native side and forward them to `ProfileFileCredentialsProvider`, matching Hadoop S3A's `ProfileAWSCredentialsProvider` semantics without depending on a particular Hadoop version (the Hadoop class itself only exists in Hadoop 3.4.2+). - Read `fs.s3a.auth.profile.name` / `fs.s3a.auth.profile.file` (with bucket-scoped overrides) into the `Profile` credential provider metadata. - Wire them into `ProfileFileCredentialsProvider::builder()` via `profile_name(...)` and `profile_files(...)`. - Add unit tests covering the no-config, name+file, and bucket-override cases. - Document the supported options in the S3 datasources guide. Closes apache#4245 Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
|
Was incorrectly created, will reopen post end to end testing |
karuppayya
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #4245
Rationale for this change
Follow-up to #4213. The Profile credential provider added in #4213 ignored the
two Hadoop S3A configs that select which profile to use:
fs.s3a.auth.profile.namefs.s3a.auth.profile.fileAdding the above with this PR.
What changes are included in this PR?
Support
fs.s3a.auth.profile.namefs.s3a.auth.profile.fileHow are these changes tested?
TBD