Current version has a compounding issue in files with more than two tables in ExcelParser.get_cell_value_map. Once you get to a third table the combined row count of the previous tables is added to the row value after each subsequent table causing a file with even small, 20 rows, tables to hit tens of thousands of rows or more, most blank, after just a handful of tables written and rapidly reach excels row limit causing a crash.
My fix tested on a markdown file with 178, 22 row, tables:
def get_cell_value_map(self, all_data_html: Tag) -> Dict[int, List[Tuple[int, Tag]]]:
"""
iterates over the html body and creates
a cell value mapping
Parameters
----------
all_data_html: Tag
Html body
Returns
-------
cell_map_dict : Dict[int, List[Tuple[int, Tag]]]
dictionary that maps a row to its corresponding columns and values
"""
# cell_map_dict = {"1":[(1, 'a'), (2, 'b'), (3,'c'), (4,'d'), (5,'e'), (6,'f')], "2":[(3,'g'),(5,'h'), (6,'i')], "3":[(2,'j')]}
cell_map_dict = defaultdict(list)
valid_cols_for_rows: Dict[int, Dict[int, Optional[bool]]] = defaultdict(lambda : defaultdict(lambda: None))
offset = 0
table_row = 0 """!!!!!!!!!!NEW LINE!!!!!!!!!!"""
for each in all_data_html:
# respect line breaks if <br> tag is added so as to mimic excel's parsing strategy
if each.name == 'br':
offset += 1
elif each.name == 'table':
data_rows = self._get_row(each, ["tr"])
for row_no, row in enumerate(data_rows, 1):
table_row = row_no """!!!!!!!!!!NEW LINE!!!!!!!!!!"""
row_no += offset
columns = self._get_row(row, ["th", "td"])
next_col = 1
for col in columns:
col_no = next_col
while not (valid_cols_for_rows[row_no][col_no] or valid_cols_for_rows[row_no][col_no] is None):
col_no += 1
attrs = col.attrs
cell_map_dict[row_no].append((col_no, col))
next_col = col_no + 1
colspan = int(attrs.get("colspan", 1))
rowspan = int(attrs.get("rowspan", 1))
self.set_neighbor_cells_to_valid(row_no, col_no, rowspan, colspan, valid_cols_for_rows)
self.set_parsed_cells_to_invalid(row_no, col_no, rowspan, colspan, valid_cols_for_rows)
offset += table_row """!!!!!!!!!!NEW LINE!!!!!!!!!!"""
#offset += row_num !!!!!!!!!!REMOVED LINE!!!!!!!!!!
return cell_map_dict
Current version has a compounding issue in files with more than two tables in ExcelParser.get_cell_value_map. Once you get to a third table the combined row count of the previous tables is added to the row value after each subsequent table causing a file with even small, 20 rows, tables to hit tens of thousands of rows or more, most blank, after just a handful of tables written and rapidly reach excels row limit causing a crash.
My fix tested on a markdown file with 178, 22 row, tables: