-
Notifications
You must be signed in to change notification settings - Fork 416
[docs][lakehouse] add Apache Ozone storage docs and catalog examples #3421
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
xylaaaaa
wants to merge
1
commit into
apache:master
Choose a base branch
from
xylaaaaa:docs/ozone-storage-4.0.4
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| --- | ||
| { | ||
| "title": "Apache Ozone | Storages", | ||
| "language": "en", | ||
| "description": "Starting from version 4.0.4, Doris supports accessing Apache Ozone through the S3 Gateway.", | ||
| "sidebar_label": "Apache Ozone" | ||
| } | ||
| --- | ||
|
|
||
| # Apache Ozone | ||
|
|
||
| Starting from version 4.0.4, Doris supports accessing Apache Ozone through the S3 Gateway. | ||
|
|
||
| This document describes the parameters required to access Apache Ozone. These parameters apply to: | ||
|
|
||
| - Catalog properties | ||
| - Table Valued Function properties | ||
| - Broker Load properties | ||
| - Export properties | ||
| - Outfile properties | ||
|
|
||
| **To use Ozone as a dedicated storage type, configure `"fs.ozone.support" = "true"` explicitly.** | ||
|
|
||
| ## Parameter Overview | ||
|
|
||
| | Property Name | Legacy Name | Description | Default Value | Required | | ||
| | --- | --- | --- | --- | --- | | ||
| | ozone.endpoint | s3.endpoint | Ozone S3 Gateway endpoint, for example `http://ozone-s3g:9878` | None | Yes | | ||
| | ozone.region | s3.region | Region of Ozone S3 Gateway | `us-east-1` | No | | ||
| | ozone.access_key | s3.access_key, s3.access-key-id | Access key used for authentication | None | No* | | ||
| | ozone.secret_key | s3.secret_key, s3.secret-access-key | Secret key used for authentication | None | No* | | ||
| | ozone.session_token | s3.session_token, s3.session-token | Session token | None | No | | ||
| | ozone.connection.maximum | s3.connection.maximum | Maximum number of connections | `100` | No | | ||
| | ozone.connection.request.timeout | s3.connection.request.timeout | Request timeout in milliseconds | `10000` | No | | ||
| | ozone.connection.timeout | s3.connection.timeout | Connection timeout in milliseconds | `10000` | No | | ||
| | ozone.use_path_style | use_path_style, s3.path-style-access | Whether to use path-style access | `true` | No | | ||
| | ozone.force_parsing_by_standard_uri | force_parsing_by_standard_uri | Whether to force standard URI parsing | `false` | No | | ||
| | fs.ozone.support | | Whether to enable Ozone as storage type | `false` | Yes | | ||
|
|
||
| Notes: | ||
|
|
||
| - `ozone.access_key` and `ozone.secret_key` must be configured together. | ||
| - `fs.s3a.*` keys are not parsed directly by Ozone properties. Use `ozone.*` or compatible `s3.*` keys. | ||
| - Ozone supports `s3://`, `s3a://`, and `s3n://` URI schemas. | ||
|
|
||
| ## Example Configuration | ||
|
|
||
| Using `ozone.*` keys: | ||
|
|
||
| ```properties | ||
| "fs.ozone.support" = "true", | ||
| "ozone.endpoint" = "http://ozone-s3g:9878", | ||
| "ozone.access_key" = "hadoop", | ||
| "ozone.secret_key" = "hadoop", | ||
| "ozone.region" = "us-east-1", | ||
| "ozone.use_path_style" = "true" | ||
| ``` | ||
|
|
||
| Using compatible `s3.*` aliases: | ||
|
|
||
| ```properties | ||
| "fs.ozone.support" = "true", | ||
| "s3.endpoint" = "http://ozone-s3g:9878", | ||
| "s3.access_key" = "hadoop", | ||
| "s3.secret_key" = "hadoop", | ||
| "s3.region" = "us-east-1", | ||
| "s3.path-style-access" = "true" | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
68 changes: 68 additions & 0 deletions
68
i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/storages/ozone.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| --- | ||
| { | ||
| "title": "Apache Ozone | Storages", | ||
| "language": "zh-CN", | ||
| "description": "自 4.0.4 版本起，Doris 支持通过 S3 Gateway 访问 Apache Ozone。", | ||
| "sidebar_label": "Apache Ozone" | ||
| } | ||
|
Comment on lines
+1
to
+7
|
||
| --- | ||
|
|
||
| # Apache Ozone | ||
|
|
||
| 自 4.0.4 版本起，Doris 支持通过 S3 Gateway 访问 Apache Ozone。 | ||
|
|
||
| 本文档介绍访问 Apache Ozone 所需的参数，这些参数适用于以下场景： | ||
|
|
||
| - Catalog 属性 | ||
| - Table Valued Function 属性 | ||
| - Broker Load 属性 | ||
| - Export 属性 | ||
| - Outfile 属性 | ||
|
|
||
| **如果要将 Ozone 作为独立存储类型使用，需要显式配置 `"fs.ozone.support" = "true"`。** | ||
|
|
||
| ## 参数总览 | ||
|
|
||
| | 属性名称 | 曾用名 | 描述 | 默认值 | 是否必须 | | ||
| | --- | --- | --- | --- | --- | | ||
| | ozone.endpoint | s3.endpoint | Ozone S3 Gateway 访问端点，例如 `http://ozone-s3g:9878` | 无 | 是 | | ||
| | ozone.region | s3.region | Ozone S3 Gateway 区域 | `us-east-1` | 否 | | ||
| | ozone.access_key | s3.access_key, s3.access-key-id | 用于认证的 Access Key | 无 | 否* | | ||
| | ozone.secret_key | s3.secret_key, s3.secret-access-key | 用于认证的 Secret Key | 无 | 否* | | ||
| | ozone.session_token | s3.session_token, s3.session-token | Session Token | 无 | 否 | | ||
| | ozone.connection.maximum | s3.connection.maximum | 最大连接数 | `100` | 否 | | ||
| | ozone.connection.request.timeout | s3.connection.request.timeout | 请求超时时间（毫秒） | `10000` | 否 | | ||
| | ozone.connection.timeout | s3.connection.timeout | 连接超时时间（毫秒） | `10000` | 否 | | ||
| | ozone.use_path_style | use_path_style, s3.path-style-access | 是否使用 path-style 访问 | `true` | 否 | | ||
| | ozone.force_parsing_by_standard_uri | force_parsing_by_standard_uri | 是否强制使用标准 URI 解析 | `false` | 否 | | ||
| | fs.ozone.support | | 是否启用 Ozone 存储类型 | `false` | 是 | | ||
|
|
||
| 说明： | ||
|
|
||
| - `ozone.access_key` 和 `ozone.secret_key` 需要成对配置。 | ||
| - `fs.s3a.*` 参数不会被 Ozone 属性直接解析，请使用 `ozone.*` 或兼容的 `s3.*` 参数。 | ||
| - Ozone 支持 `s3://`、`s3a://`、`s3n://` 三种 URI 协议。 | ||
|
|
||
| ## 配置示例 | ||
|
|
||
| 使用 `ozone.*` 参数： | ||
|
|
||
| ```properties | ||
| "fs.ozone.support" = "true", | ||
| "ozone.endpoint" = "http://ozone-s3g:9878", | ||
| "ozone.access_key" = "hadoop", | ||
| "ozone.secret_key" = "hadoop", | ||
| "ozone.region" = "us-east-1", | ||
| "ozone.use_path_style" = "true" | ||
| ``` | ||
|
|
||
| 使用兼容的 `s3.*` 参数： | ||
|
|
||
| ```properties | ||
| "fs.ozone.support" = "true", | ||
| "s3.endpoint" = "http://ozone-s3g:9878", | ||
| "s3.access_key" = "hadoop", | ||
| "s3.secret_key" = "hadoop", | ||
| "s3.region" = "us-east-1", | ||
| "s3.path-style-access" = "true" | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Front matter
titleis set to"Apache Ozone | Storages", but other storage docs use just the storage name as the title (e.g.docs/lakehouse/storages/s3.md,azure-blob.md). Consider aligning this to"Apache Ozone"(and rely on sidebar/category for context) to keep titles consistent across storage pages.