Input connector is not working in XL _in the read section.
This does not work beause the data is not sent to the backend!
read:
- union:
sources:
- input: {}
- file:
name: test_file.xlsx
PY code minimally handles it: connector just defines the schema, and recipe.py maps it to input dataframe. Need to identify the exact place in XL where logic assumes that if there is a read: then it does NOT send any excel selected data.
We know it has to do with read: because this actually works:
wrangles:
- recipe:
read:
- union:
sources:
- input: {}
- file:
name: test_file.xlsx
Fixing this will require expanding how XL handles recipes to allow it to do some processing before sending recipe & data to the lambda (for example dealing with batching etc). While doing this, we should also figure out how we could handle a local XL read that joins multiple sheets/tables. Could look like this:
read:
- excel.sql:
SELECT *
FROM table1 JOIN table2 ...
Note: excel SQL capability already exists.
Let's discuss to flesh out the requirements.
Input connector is not working in XL _in the read section.
This does not work beause the data is not sent to the backend!
PY code minimally handles it: connector just defines the schema, and recipe.py maps it to input dataframe. Need to identify the exact place in XL where logic assumes that if there is a read: then it does NOT send any excel selected data.
We know it has to do with read: because this actually works:
wrangles:
read:
- union:
sources:
- input: {}
- file:
name: test_file.xlsx
Fixing this will require expanding how XL handles recipes to allow it to do some processing before sending recipe & data to the lambda (for example dealing with batching etc). While doing this, we should also figure out how we could handle a local XL read that joins multiple sheets/tables. Could look like this:
read:
SELECT *
FROM table1 JOIN table2 ...
Note: excel SQL capability already exists.
Let's discuss to flesh out the requirements.