Add member ID 32-bit int type validation for Iceberg table schemas#475
Add member ID 32-bit int type validation for Iceberg table schemas#475li-ukumar wants to merge 1 commit intolinkedin:mainfrom
Conversation
Adds validateMemberIdColumnTypes() to OpenHouseTablesApiValidator that rejects Iceberg table schemas using INTEGER type for member identity columns (memberId, actorId, profileId, customerId, sourceId, destId, etc.). Member IDs will overflow 32-bit int — all must use LONG. Integrated into both validateCreateTable() and validateUpdateTable(). See go/project-2b for details.
| validationFailures.addAll( | ||
| validateUpdateTimestampForReplicatedTable(createUpdateTableRequestBody)); | ||
| if (createUpdateTableRequestBody.getSchema() != null) { | ||
| validateMemberIdColumnTypes(createUpdateTableRequestBody.getSchema(), validationFailures); |
There was a problem hiding this comment.
Seems like this is specific usecase for LInkedIn memberId. Can we have this validation in the internal li repo as the validation is specific to LinkedIn usecase considering this is OSS codebase? We would need to extend OpenHouseTablesApiValidator in the internal repo and that can be enabled by adding something like @primary annotation.
|
this doesn't belong in api layer. it should live next to the existing schema validation logic on the server. check if that exists at all in the linkedin implementation. also, what is the rollout plan? just fail users? thats not acceptable. we need an answer to:
and minimum shippable code would have debug / log info only. also need to attach e2e tests with spark or create table curl requests |
Summary
validateMemberIdColumnTypes()toOpenHouseTablesApiValidatorthat rejects Iceberg table schemas usingINTEGERtype for member identity columns (memberId, actorId, profileId, customerId, sourceId, destId, etc.)LONGvalidateCreateTable()andvalidateUpdateTable()Testing Done
longmemberId column passesintmemberId column rejected (create path)intcolumn (retryCount) passesmidrejectedmember_id) rejectedintmemberId rejected via update path./gradlew :services:tables:buildpasses