Update dependency org.apache.hive:hive-metastore to v4

[renovate-bot]

This PR contains the following updates:

Package	Change	Age	Confidence
org.apache.hive:hive-metastore (source)	3.1.3 → 4.2.0

---

Warning

Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

---

Configuration

📅 Schedule: (UTC)

• Branch creation
  • At any time (no schedule defined)
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.

[flyingImer]

This is a major version bump-up, the checks seem to be just fine.

[flyrain]

Looks like there is no tests for the Hive federation. I'd suggest add integration tests before we merge this.

Minor finding:
Interesting that the comment says it is using hive 4.1.0,

polaris/extensions/federation/hive/build.gradle.kts

Line 35 in 8cbfb26

// Add our own Hive 4.1.0 dependencies

.

[flyrain]

Need integration tests to verify if there are regression for the affect component (HMS federation).

[jbonofre]

@flyrain I'm also not convinced we should update HMS version, because, afair, HMS 4.x plans to add IRC support.

[nandorKollar]

@flyrain I'm also not convinced we should update HMS version, because, afair, HMS 4.x plans to add IRC support.

That's what I heard too. With IRC support I don't think one should use HMS federation with using the Thrift interface. Integration tests would still make sense though.

[flyrain]

@flyrain I'm also not convinced we should update HMS version, because, afair, HMS 4.x plans to add IRC support.

That's what I heard too. With IRC support I don't think one should use HMS federation with using the Thrift interface. Integration tests would still make sense though.

I think we only care about the Hive client side lib here, which isn't necessarily related to the IRC support. In case of HMS federation support in Polaris, it should always use the thrift clients, no matter what the client lib version is.

For the use case of federation via HMS IRC interface, that would belong to the REST federation, which doesn't depend on the hive version at all.

[nandorKollar]

@flyrain I'm also not convinced we should update HMS version, because, afair, HMS 4.x plans to add IRC support.

That's what I heard too. With IRC support I don't think one should use HMS federation with using the Thrift interface. Integration tests would still make sense though.

I think we only care about the Hive client side lib here, which isn't necessarily related to the IRC support. In case of HMS federation support in Polaris, it should always use the thrift clients, no matter what the client lib version is.

For the use case of federation via HMS IRC interface, that would belong to the REST federation, which doesn't depend on the hive version at all.

Ah okay, makes sense. I recall that I tried to upgrade the HMS client lib version, and there were indeed problems with connecting to HMS 3.x server. I'll try to reproduce that.

[jbonofre]

OK for the client side. I would just suggest to include it in the CI to test it. Else, we are kind of blind for this kind of update.

[jbonofre]

@flyrain I think it depends to IRC support in HMS: if HMS supports IRC, we don't need to use Hive client anymore, just the regular IRC client. Just my $0.01

[flyrain]

@flyrain I think it depends to IRC support in HMS: if HMS supports IRC, we don't need to use Hive client anymore, just the regular IRC client. Just my $0.01

@jbonofre, yes, as I said above. That use case falls under REST federation, which operates in parallel to the current HMS federation logic. For this PR, we are specifically focused on the Thrift-based client. IRC federation doesn't need Hive client at all.

@nandorKollar , it'd be great if you can identify any issues or adding tests.

[nandorKollar]

OK for the client side. I would just suggest to include it in the CI to test it. Else, we are kind of blind for this kind of update.

We should, for sure. I have an open PR to at least run the build with the extension, but an integration test is still
needed i think.

I think with HMS 4.x client, and 3.x server this was the exception, looks like a breaking change in Hive (HIVE-26537: Deprecate older APIs in the HMS thrift interface):

polaris Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'create_database_req'                                                                       
polaris   at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)                                                                                 
polaris   at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_database_req(ThriftHiveMetastore.java:1439)

If this indeed doesn't work, probably we shouldn't proceed with bumping the client version.

[flyrain]

#4315 has added HMS integration tests, we can reconsider this upgrade.

[flyrain]

It did fail on new added hive integration tests. cc @adutra

HiveCatalogFederationIT > testFederatedWriteVisibleInHms() FAILED
    java.lang.NoSuchFieldError at HiveCatalog.java:113

HiveCatalogFederationIT > testFederatedReadSeesHmsTables() FAILED
    org.opentest4j.AssertionFailedError at ManagementApi.java:71

HiveCatalogFederationIT > testFederatedCatalogEnforcesNamespaceRbac() FAILED
    org.opentest4j.AssertionFailedError at ManagementApi.java:71