Skip to content

Conversation

@XuQianJin-Stars
Copy link
Contributor

Purpose

Linked issue: close #2251

This PR adds support for NestedRow types in Paimon lake tiering.

Brief change log

  • Modified FlussRowAsPaimonRow.getRow() to support nested row fields by creating new FlussRowAsPaimonRow instances recursively
  • Modified FlussArrayAsPaimonArray.getRow() to support arrays containing nested row elements
  • Added comprehensive test cases in FlussRecordAsPaimonRowTest covering various nested row scenarios:
    • Simple nested rows with primitive types
    • Deeply nested rows (row within row)
    • Arrays of nested rows
    • Nested rows containing array fields
    • Nested rows with all primitive types (boolean, byte, short, int, long, float, double, string)
    • Null nested rows and nullable fields within nested rows
    • Nested rows with complex types (Decimal, Timestamp)

Tests

Unit Tests:

  • FlussRecordAsPaimonRowTest#testSimpleNestedRow - Tests basic nested row conversion
  • FlussRecordAsPaimonRowTest#testDeeplyNestedRow - Tests multi-level nested rows
  • FlussRecordAsPaimonRowTest#testArrayOfNestedRows - Tests arrays containing nested rows
  • FlussRecordAsPaimonRowTest#testNestedRowWithArrayField - Tests nested rows with array fields
  • FlussRecordAsPaimonRowTest#testNestedRowWithAllPrimitiveTypes - Tests all primitive types in nested rows
  • FlussRecordAsPaimonRowTest#testNullNestedRow - Tests null nested row handling
  • FlussRecordAsPaimonRowTest#testNestedRowWithNullableFields - Tests nullable fields in nested rows
  • FlussRecordAsPaimonRowTest#testNestedRowWithDecimalAndTimestamp - Tests complex types in nested rows

All tests verify correct data conversion between Fluss nested rows and Paimon nested rows.

API and Format

API: No public API changes

Format: No storage format changes. This change only affects the internal data conversion layer between Fluss and Paimon.

Documentation

New Feature: No

This change extends existing Paimon lake tiering functionality to support NestedRow types, which is an internal enhancement. The data type conversion mapping table in the documentation (website/docs/streaming-lakehouse/integrate-data-lakes/paimon.md) already covers ROW types, so no documentation update is required.

@XuQianJin-Stars XuQianJin-Stars changed the title [FLUSS-2251] Support NestedRow types for paimon [lake/paimon] Support NestedRow types for tiering paimon Dec 25, 2025
@luoyuxia luoyuxia requested a review from Copilot December 26, 2025 01:35
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds support for NestedRow types in Paimon lake tiering by implementing the previously unsupported getRow() methods in the data conversion layer between Fluss and Paimon.

Key Changes:

  • Implemented recursive nested row conversion by creating new FlussRowAsPaimonRow instances for nested row fields
  • Extended array support to handle arrays containing nested row elements
  • Added 8 comprehensive test cases covering simple nested rows, deeply nested structures, arrays of rows, null handling, and various data types

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
FlussRowAsPaimonRow.java Implemented getRow() method to convert nested Fluss rows to Paimon rows recursively with proper null handling
FlussArrayAsPaimonArray.java Implemented getRow() method to support arrays containing nested row elements
FlussRecordAsPaimonRowTest.java Added 8 test methods covering nested row scenarios including primitive types, complex types, null handling, deep nesting, and arrays of rows

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Contributor

@luoyuxia luoyuxia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@XuQianJin-Stars Thanks for the pr. Let's add two test:

  • add test in PaimonRowAsFlussRow for nestrow type
  • add IT for nested-row type, you can refer to what we do for array type

}

@Test
void testSimpleNestedRow() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please combine all test methods for NestedRow to a single method just like main branch for array type?
Two many methods make it hard to track...

@XuQianJin-Stars XuQianJin-Stars force-pushed the feature/issue-2251-NestedRow-paimon branch from f5ef8a4 to 8280f47 Compare December 26, 2025 06:20
- Created FlussNestedRowAsPaimonRow adapter class to convert Fluss nested rows to Paimon nested rows
- Implemented getRow method in FlussRowAsPaimonRow to support nested row fields in tables
- Implemented getRow method in FlussArrayAsPaimonArray to support arrays of nested rows
- Added comprehensive test cases covering:
  * Simple nested rows with primitive types
  * Deeply nested rows (row within row)
  * Arrays of nested rows
  * Nested rows with array fields
  * Nested rows with all primitive types
  * Null nested rows
  * Nested rows with nullable fields
  * Nested rows with decimal and timestamp fields
@XuQianJin-Stars XuQianJin-Stars force-pushed the feature/issue-2251-NestedRow-paimon branch from 8280f47 to 48fa51d Compare December 26, 2025 06:40
@XuQianJin-Stars
Copy link
Contributor Author

@luoyuxia Hi, i already updated the pr. Please help review when you got some time

}

@ParameterizedTest
@MethodSource("tieringNestedRowWriteArgs")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace by @ValueSource(booleans = {true, false}) and remove tieringNestedRowWriteArgs method.

@XuQianJin-Stars XuQianJin-Stars force-pushed the feature/issue-2251-NestedRow-paimon branch from d678a38 to fe6404d Compare December 27, 2025 06:39
@XuQianJin-Stars
Copy link
Contributor Author

@wuchong Hi, i already updated the pr. Please help review when you got some time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support NestedRow types for paimon

3 participants