Skip to content

Fix for files with multiple tables #9

Description

@AHoopyFrood

Current version has a compounding issue in files with more than two tables in ExcelParser.get_cell_value_map. Once you get to a third table the combined row count of the previous tables is added to the row value after each subsequent table causing a file with even small, 20 rows, tables to hit tens of thousands of rows or more, most blank, after just a handful of tables written and rapidly reach excels row limit causing a crash.

My fix tested on a markdown file with 178, 22 row, tables:

def get_cell_value_map(self, all_data_html: Tag) -> Dict[int, List[Tuple[int, Tag]]]:
        """
            iterates over the html body and creates
            a cell value mapping
            Parameters
            ----------
            all_data_html: Tag
                    Html body
            Returns
            -------
            cell_map_dict : Dict[int, List[Tuple[int, Tag]]]
                    dictionary that maps a row to its corresponding columns and values
        """
        # cell_map_dict = {"1":[(1, 'a'), (2, 'b'), (3,'c'), (4,'d'), (5,'e'), (6,'f')], "2":[(3,'g'),(5,'h'), (6,'i')], "3":[(2,'j')]}
        cell_map_dict = defaultdict(list)
        valid_cols_for_rows: Dict[int, Dict[int, Optional[bool]]] = defaultdict(lambda : defaultdict(lambda: None))
        offset = 0
        table_row = 0 """!!!!!!!!!!NEW LINE!!!!!!!!!!"""
        for each in all_data_html:
            # respect line breaks if <br> tag is added so as to mimic excel's parsing strategy
            if each.name == 'br':
                offset += 1
            elif each.name == 'table':
                data_rows = self._get_row(each, ["tr"])
                for row_no, row in enumerate(data_rows, 1):
                    table_row = row_no """!!!!!!!!!!NEW LINE!!!!!!!!!!"""
                    row_no += offset
                    columns = self._get_row(row, ["th", "td"])

                    next_col = 1
                    for col in columns:
                        col_no = next_col
                        while not (valid_cols_for_rows[row_no][col_no] or valid_cols_for_rows[row_no][col_no] is None):
                            col_no += 1
                        attrs = col.attrs
                        cell_map_dict[row_no].append((col_no, col))
                        next_col = col_no + 1
                        
                        colspan = int(attrs.get("colspan", 1))
                        rowspan = int(attrs.get("rowspan", 1))
                        self.set_neighbor_cells_to_valid(row_no, col_no, rowspan, colspan, valid_cols_for_rows)
                        self.set_parsed_cells_to_invalid(row_no, col_no, rowspan, colspan, valid_cols_for_rows)

                offset += table_row """!!!!!!!!!!NEW LINE!!!!!!!!!!"""
               #offset += row_num !!!!!!!!!!REMOVED LINE!!!!!!!!!!

        return cell_map_dict

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions