Skip to content

Add list column support to the record batch API - #101

Draft
yogendra-17 wants to merge 2 commits into
arxlang:mainfrom
yogendra-17:feat/arrow-list-types
Draft

Add list column support to the record batch API#101
yogendra-17 wants to merge 2 commits into
arxlang:mainfrom
yogendra-17:feat/arrow-list-types

Conversation

@yogendra-17

Copy link
Copy Markdown
Contributor

Pull Request description

This adds list column support to the record batch API. It's the first bit of nested type support (lists now, structs later).
Right now the list element can be any fixed-width type - the int/uint/float types and the date/time types from #100. I left out bool, utf8 and lists-of-lists for now, those can come in a follow up. To make room for the nested stuff I added a small recursive type descriptor (IrxRbType) so we can describe the element type through the C ABI and the ctypes layer.

New things we can do:

  • schema.add_list_field(name, elem_type) to declare a list column
  • builder.append_list(col, values) to add one list
  • batch.get_list(col, row) to read it back (returns None for a null slot,empty list for an empty one)

Reads are zero-copy - it uses the offsets buffer + the flattened child buffer instead of copying element by element. Also checked that it round-trips with pyarrow both ways.

Note: This is stacked on top of #100 (date/time), so it'll look cleaner to review once that one lands.

How to test these changes

  • cd packages/irx
  • python -m pytest tests/test_record_batch.py
    The list tests cover int/float/date lists, null and empty lists, the raw buffer round-trip, rejecting unsupported element types, and pyarrow interop.
  • ...

Pull Request checklists

This PR is a:

  • bug-fix
  • new feature
  • maintenance

About this PR:

  • it includes tests.
  • the tests are executed on CI.
  • the tests generate log file(s) (path).
  • pre-commit hooks were executed locally.
  • this PR requires a project documentation update.

Author's checklist:

  • I have reviewed the changes and it contains no misspelling.
  • The code is well commented, especially in the parts that contain more complexity.
  • New and old tests passed locally.

Additional information

Reviewer's checklist

Copy and paste this template for your review's note:

## Reviewer's Checklist

- [ ] I managed to reproduce the problem locally from the `main` branch
- [ ] I managed to test the new changes locally
- [ ] I confirm that the issues mentioned were fixed/resolved .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant