Decode improvements

[marcelomarkus]

Pull Request

📋 Description

This Pull Request introduces the Rust-powered binary decoder for the B-FAST serialization protocol, natively integrated into the bfast-py core package via PyO3. It also adds support for parallel LZ4 chunk decompression fallback in the bfast-client TypeScript client package and fixes critical segmentation faults related to multithreaded Rayon allocations.

🔗 Related Issues

Fixes # (No open issues, requested directly by user)

🧪 Type of Change

• 🐛 Bug fix (non-breaking change which fixes an issue)
• ✨ New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• 📚 Documentation update
• 🔧 Refactoring (no functional changes)
• ⚡ Performance improvement
• 🧹 Code cleanup

🚀 Changes Made

1. Rust Core & Python Bindings (bfast-py)

• Natively implemented the B-FAST binary decoder in Rust: Ported the full decoder logic to Rust, exposing the method decode_packed(bytes, *, decompress=True) -> PyObject on the BFast class.
• Decoded native binary tags: Decoded Null, Booleans, Int64, Float64, Small Integers (bit-packed), Strings, Lists, Objects/Dicts, Raw Bytes, NumPy Arrays (f64), Datetime/Date/Time (ISO 8601), UUIDs, and Decimals.
• Robust Parser Protections:
  • Implemented bounds checking to prevent buffer overflow vulnerabilities.
  • Implemented recursion depth tracking (MAX_RECURSION_DEPTH = 128) to prevent stack overflow crashes.
  • Capped capacity allocations for dynamic vectors based on remaining buffer sizes to protect against Out-of-Memory (OOM) attacks.

2. Thread-Safety & Allocator Bugfix

• Fixed Rayon worker thread segmentation faults: Disabled the custom PyMemAllocator global allocator inside src/allocator.rs. The custom allocator was calling Python's GIL-bound allocation functions (PyMem_Malloc) on background threads without the GIL, causing memory corruption. Reverted to Rust's native thread-safe system allocator.

3. TypeScript Client (bfast-client)

• Added parallel decompression support: Implemented a fallback decoder in client-ts/index.ts to decompress chunks in parallel using.
• Memory bug mitigation: Copied sub-chunk buffers to a fresh ArrayBuffer slice to avoid offset-resolution.
• Improved developer environment: Included test files in tsconfig.json so they are compiled cleanly.

4. Releases & Versioning

• Bumped bfast-py package version to 1.3.0 (in pyproject.toml, Cargo.toml, and python/b_fast/__init__.py).
• Bumped bfast-client (TypeScript client) version to 1.2.0 (in client-ts/package.json).
• Updated CHANGELOG.md to document version 1.3.0 releases.

🧪 Testing

Test Results

🐍 Python Test Suite (All 33 tests passed in 0.59s)

$ maturin develop && .venv/bin/pytest tests/
============================= test session starts ==============================
platform linux -- Python 3.11.13, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/markus/dev/b-fast
configfile: pyproject.toml
plugins: anyio-4.13.0
collecting ... collected 33 items                                                             

tests/test_basic.py .....                                                [ 15%]
tests/test_decode.py ........                                            [ 39%]
tests/test_extended_types.py ...................                         [ 96%]
tests/test_integration_types.py .                                        [100%]

============================== 33 passed in 0.59s ==============================

⚡ TypeScript Integration Suite (100% type preservation and parallel LZ4 decoding validation)

$ npm run build && node dist/tests/type-preservation.test.js
🧪 Testing B-FAST Type Preservation

============================================================

✅ Decoded successfully

Type validation:
  name: "John Doe" (string) ✅
  age: 30 (number) ✅
  created_at: 2024-01-15T13:30:45.000Z (Date) ✅
  birth_date: 1994-05-20 (Date) ✅
  wake_time: "07:30:00" (string) ✅
  user_id: "550e8400-e29b-41d4-a716-446655440000" (string) ✅
  balance: 1234.56 (number) ✅
  active: true (boolean) ✅

🧪 Testing B-FAST Parallel Decompression (Large Payload)

  Decoded large array length: 12000 (expected: 12000) ✅

============================================================

✅ All type preservation tests passed!

📊 Performance Impact

• No performance impact
• Performance improved
• Performance regression (explain why acceptable)

Benchmarks (if applicable)

The Rust decoder achieves ultra-fast deserialization compared to Python decoding by parsing binary structures in native Rust memory, yielding a major performance boost for large payloads.

📊 Documentation

• Code is self-documenting
• Docstrings/comments added/updated
• README updated (if needed)
• Documentation in docs/ updated (if needed)

✅ Checklist

• My code follows the project's coding standards
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

🔍 Review Notes

Please check the decompress_packed implementation in src/lib.rs and the allocator replacement in src/allocator.rs. Rayon multi-threaded operations now execute completely in Rust's system memory space, which eliminates Python GIL dependency issues and segfaults.

📸 Screenshots (if applicable)

N/A

🤝 Additional Context

N/A