Automatic chunk size based on available RAM

[atrabattoni]

This pull request discuss ways to implement automatic chunk size selection base on available hardware.

[codecov]

Codecov Report

❌ Patch coverage is 29.16667% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.96%. Comparing base (e657bd9) to head (b241aa9).

Files with missing lines	Patch %	Lines
xdas/core/routines.py	29.16%	17 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##              dev      #57      +/-   ##
==========================================
- Coverage   83.24%   82.96%   -0.29%     
==========================================
  Files          43       43              
  Lines        4578     4602      +24     
==========================================
+ Hits         3811     3818       +7     
- Misses        767      784      +17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aurelienfalco]

Idea for setting a limit to memory allocation:

import resource
import sys

Set maximum heap size (e.g., 500 MB)

max_heap_size = 500 * 1024 * 1024  # 500 MB in bytes

Set memory limit

resource.setrlimit(resource.RLIMIT_AS, (max_heap_size, max_heap_size))

try:
# Simulate a memory-heavy operation
print("Allocating memory...")
large_list = [0] * (10**8)  # This will consume significant memory
except MemoryError:
print("Memory limit reached! Exiting gracefully.")
sys.exit(1)

print("Memory allocation successful within limit.")