Iceberg encryption support

[flyrain]

There are interests to bring Iceberg table encryption support in Polaris. Filing this issue to facilitate discussion.

[obelix74]

Broadly speaking, there is a need to store encrypted data in tables for Iceberg. Clients get a table specific key and decrypt the data in the client. This allows storage of sensitive data in public cloud tables.

The iceberg specification specifies encryption-keys in the table-metadata-fields section. https://iceberg.apache.org/spec/#table-metadata-fields

My understanding of how this works:

• Client-side encryption: Iceberg encrypts data on the client before writing it to storage and decrypts it on the client after reading it. The storage backend, like Amazon S3, never sees the plain text data.
• Key Management Service (KMS): Encryption and decryption keys are managed by a KMS, which handles key access, authentication, and authorization.
• To access an encrypted table, a user must have permissions to both the data and the KMS key used for encryption.
• The user uses the KMS key to decrypt the data.

I think we need to be able to use encrypted data only on certain tables, so that only sensitive data goes through this approach.

[hkwi]

For clarity, the following compares the current status across the Hive Catalog implementation, the REST Catalog specification, and the Polaris REST Catalog implementation.

1. Client-driven encryption support

In this model, the client encrypts and decrypts table data. The catalog server does not need to access plaintext data or connect to the KMS.

Hive Catalog Implementation

Hive Catalog currently supports:

• catalog-level KMS client configuration;
• EncryptionManager and EncryptingFileIO;
• encrypted data files, manifests, and manifest lists;
• encrypted table metadata and wrapped encryption keys.

REST Catalog Specification

The REST Catalog API already defines the metadata needed to represent encrypted tables, including:

• encryption-keys in table metadata;
• add-encryption-key and remove-encryption-key updates;
• key-metadata in content files.

Iceberg PR #13225 adds client-side encryption support for REST Catalogs. The PR is currently open.

This is primarily an Iceberg client implementation. It does not require the REST server to own or access the KMS, as long as the server preserves and commits the encryption metadata.

Polaris REST Catalog Implementation

For client-driven encryption, Polaris should not need to connect to the KMS. The server can preserve and commit the encryption metadata through the normal table update paths.

Polaris does not currently have an explicit table-encryption integration or a dedicated encryption end-to-end test. The client-driven path should therefore be considered separately from server-side encryption operations.

2. Full REST Catalog semantics for encrypted tables covered by the existing specification

This area concerns operations where the REST server reads, plans, or deletes table files.

Hive Catalog Implementation

Hive supports encrypted purge through the normal Catalog API:

HiveCatalog.dropTable(identifier, true)
  -> CatalogUtil.dropTableData(ops.io(), metadata)
  -> HiveTableOperations.io()
  -> EncryptingFileIO

This allows encrypted manifest lists and manifests to be read before the referenced files are deleted.

Hive also supports client-side scan planning for encrypted tables through the encrypted table FileIO.

REST Catalog Specification

The REST API defines:

• purgeRequested for table deletion;
• server-side scan planning endpoints;
• storage credentials associated with scan results;
• encrypted content-file metadata.

The specification defines the REST contract, while the server implementation is responsible for applying the corresponding table operations.

Polaris REST Catalog Implementation

The main identified gap is encrypted server-side purge.

Polaris schedules cleanup tasks that create a normal storage FileIO. Those tasks then read manifest lists and manifests through that FileIO. The current path does not construct an EncryptionManager or combine the storage FileIO with EncryptingFileIO.

Therefore, encrypted server-side purge is not currently handled through the same path as Hive.

Polaris also does not currently implement the general server-side scan planning endpoint. This is a broader REST Catalog implementation gap. If server-side scan planning is added, encrypted tables will need to preserve and return the relevant file encryption metadata and storage credentials required by the client.

The metadata transport itself is a separate matter. Since Polaris applies table updates and persists table metadata through the normal paths, encryption-keys and key-metadata do not necessarily require a Polaris-specific encryption implementation.

3. Functionality not yet specified by the REST Catalog API

The following topics appear to require additional specification:

• how a REST server obtains and configures its KMS client;
• whether KMS ownership belongs to the client, the server, or both;
• how client and server KMS permissions and trust relationships are established;
• whether credential vending also applies to KMS access;
• an equivalent of Hive Metastore metadata hashing and tamper protection;
• automated encryption-key retirement and removal of obsolete keys;
• server-side maintenance operations beyond the existing REST Catalog API.

These topics should be distinguished from the behavior already represented by the current REST API.

Proposed scope for this issue

The discussion could track the following separately:

• Iceberg client support for REST Catalog encryption, including PR #13225;
• Polaris implementation of existing REST semantics, especially encrypted server-side purge;
• additional specification work for KMS ownership, trust boundaries, and server-side KMS configuration.

[hkwi]

Follow-up based on the implementation and review discussion in #5060:

A clarification to my previous comment is needed.

• Iceberg native encryption protects data files, delete files, manifest files, and manifest-list files. The table metadata JSON itself remains plaintext; it carries the wrapped encryption keys needed to open the encrypted files.
• The statement that the REST server does not need KMS access applies to the normal client-written commit path. The engine/client writes the encrypted data, manifests, and manifest lists, while Polaris preserves and commits the resulting metadata updates. Polaris should not initialize writer-side encryption or write manifests.
• Server-side purge is a separate read path. To enumerate the referenced files safely, Polaris must read and decrypt the manifest lists and manifests written by the engine. Therefore, a Polaris cleanup worker does require KMS access and the table's wrapped keys when purging an encrypted table. It does not rewrite or re-encrypt those manifests.

#5060 has been updated to reflect this responsibility split:

• writer-side encryption support was removed from LocalIcebergCatalog;
• encryption context is preserved only for asynchronous cleanup tasks, which reconstruct an encryption-aware FileIO to read encrypted manifest lists and manifests during purge; and
• the test now uses a REST integration path where test-side engine code performs encrypted writes and Polaris handles the purge.

Standard engine-side encryption through RESTCatalog still depends on the open Iceberg PR apache/iceberg#13225. Until that support is available, the Polaris integration test uses a test-only adapter to model the client/engine behavior.

This narrows the remaining areas as follows: client-side REST encryption support belongs in Iceberg; encrypted server-side purge is the Polaris implementation work in #5060; and production KMS configuration, credentials, multiple-store routing, and trust boundaries still need separate documentation and community discussion.