load_table endpoint

[Zevrap-81]

When using rest api for loading a table, more specifically using the endpoint /v1/{prefix}/namespaces/{namespace}/tables/{table}, the config properties are returned along with the table. The rest api specification for iceberg doesn't specify what properties are set here

  "config": {
    "additionalProp1": "string",
    "additionalProp2": "string",
    "additionalProp3": "string"
  },

This returns some crutial information for setting table properties in pyiceberg, viz., s3.endpoint url. However, this is imporperly set to localhost instead of the correct uri. So my question is how are these properties set and how can i influence these so polaris returns these correctly?

[flyrain]

This is what credential vending does. The loadTable will get storage credential and put them in both config and storage-credentials, more details are here

polaris/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java

Line 893 in c6d966d

buildLoadTableResponseWithDelegationCredentials(

polaris/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java

Line 940 in c6d966d

.getStorageAccessConfig(

To control that, users can create a storage configuration with information like s3.endpoint, region while creating a new catalog.

[dimas-b]

@MonkeyCanCode : WDYT?

[MonkeyCanCode]

thanks for tagging @dimas-b .

yes, @Zevrap-81 , that is possible. the detail reference is in https://github.com/apache/polaris/blob/main/site/content/guides/assets/polaris/reference-setup-config.yaml.

In case if u want to set region and endpoint, u can set them via https://github.com/apache/polaris/blob/main/site/content/guides/assets/polaris/reference-setup-config.yaml#L178-L179.

[Zevrap-81]

Yes i have seen this but what i am actually asking is the storage credentials to be set during the catalog creation. or how to pass in the credentials for the bucket?

[flyrain]

If you want to know how to config the static storage credentials at the service level, check this page out, https://polaris.apache.org/in-dev/unreleased/configuration/configuration-reference/#storage--credentials.

[MonkeyCanCode]

@Zevrap-81 maybe I missed something here, the storage credential is for catalog server instead of logical catalog (which is what the CLI offers to set). What @flyrain shared above is something you can set on the catalog server level if that is what you are looking for.

[Zevrap-81]

Understood. Since the polaris server is already running on docker, is there a way to update its configuration without stopping it?
Also do you have some examples using named storages? polaris.storage.aws..access-key . Mainly once i define the named storage in the server configuration, how do i use it in the logical catalog?

[tokoko]

I don't think there is an example. but you need to have:

• polaris.features."RESOLVE_CREDENTIALS_BY_STORAGE_NAME" set to true
• polaris.storage.aws.myStorage.access-key and polaris.storage.aws.myStorage.secret-key set
• set storageName in storageConfigInfo when creating a catalog. payload will look something like this, simply set storageName right next to storageType:

curl -X POST http://localhost:8181/api/management/v1/catalogs \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "catalog": {
      "name": "my_catalog",
      "type": "INTERNAL",
      "properties": { "default-base-location": "s3://example-bucket/my_data" },
      "storageConfigInfo": {
        "storageType": "S3",
        "storageName": "myStorage",
        "allowedLocations": ["s3://example-bucket/"],
        "s3.endpoint": "http://127.0.0.1:9000",
        "s3.pathStyleAccess": true,
        "s3.roleArn": "arn:aws:iam::000000000000:role/example",
        "region": "us-east-1"
      }
    }
  }'

[Zevrap-81]

Thanks everyone for the prompt replies

[Zevrap-81]

The thing is i want to have multiple storage buckets for one rest catalog, and one bucket for one of raw, silver, gold layer. So the s3.endpoint is specific to the table itself and not the catalog. How can i go about doing that?

[dimas-b]

This is being discussed in https://lists.apache.org/thread/nwcc8krhvm35olgc22676mm12cjkpgn5

[flyrain]

We could allow storage configuration on the table/namespace level. It is only with catalog now. Thanks @dimas-b for sharing the link. Other than the discussion, there are previous work trying to support it, #3563.

[Zevrap-81]

Thank you for the related info @dimas-b and @flyrain. Could you also share is the ongoing work planned for a particular release? if so which?

[flyrain]

There is no timeline yet AFAIK. The user case is valid, we will need more discussion and contributors to work on it.

[tokoko]

#4023 this is most likely the PR where the feature will land. it's blocked by #3699 currently though... slight caveat is that this will probably allow overriding credentials for tables, but not endpoints, but that should follow shortly afterwards