Skip to content

Saving large arrays (>10GB) #104

@RonnyAI

Description

@RonnyAI

This bug is fixed by changing:
The Problem: In npy_save, the calculation of nels uses:
The initial value 1 is an int, not a size_t. With std::accumulate, the accumulator type is determined by the initial value. This means the multiplication 1 * 858403972 * 3 is performed using 32-bit integer arithmetic, which overflows to -1719755380. This negative value then gets converted to an unsigned size_t, resulting in 18446744071989796236.
This causes fwrite to attempt writing a massive amount of data, leading to heap corruption and the embedded breakpoint exception at fclose(fp).
The Fix: Change the initial value from 1 to (size_t)1 or 1ULL:
size_t nels = std::accumulate(shape.begin(), shape.end(), 1, std::multiplies<size_t>());
to
size_t nels = std::accumulate(shape.begin(), shape.end(), (size_t)1, std::multiplies<size_t>());

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions