[feat] Make total_storage_size optional to support unlimited storage capacity#130
Merged
ji-huazhong merged 1 commit intoJun 29, 2026
Merged
Conversation
CLA Signature Passhuniu20, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
huniu20
force-pushed
the
feat/optional-storage-capacity-limit
branch
from
c67fafe to
d5ab107
Compare
CLA Signature Passhuniu20, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
huniu20
changed the title
total_storage_size optional to support unlimited storage capacity
…e capacity When the caller does not know the expected data volume at TQ initialization time (e.g. storage is created before the dataset is loaded), there is no reasonable value to set for total_storage_size, and the hard-coded default of 100000 may cause 'Storage capacity exceeded' errors at runtime. This PR makes total_storage_size optional (nullable). When set to null, the capacity check is skipped and storage grows on demand. Since the underlying StorageUnitData uses a dict (no pre-allocation), this has zero performance or memory overhead. Changes: - StorageUnitData: storage_size is now Optional[int] (default None) - SimpleStorageUnit: storage_unit_size is now Optional[int] - simple_storage_bootstrap: handles null total_storage_size gracefully - config.yaml: default total_storage_size changed to null Fully backward compatible — passing an explicit value still enforces the limit. Signed-off-by: nexhu <nexhu@tencent.com>
huniu20
force-pushed
the
feat/optional-storage-capacity-limit
branch
from
d5ab107 to
af6e786
Compare
CLA Signature Passhuniu20, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
ji-huazhong
approved these changes
Jun 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
When the caller does not know the expected data volume at TQ initialization time (e.g. storage is created before the dataset is loaded), there is no reasonable value to set for
total_storage_size, and the hard-coded default of may causeValueError: Storage capacity exceedederrors at runtime.Solution
Make
total_storage_sizeoptional (nullable). When set tonull, the capacity check is skipped and storage grows on demand.Since
StorageUnitDatauses a dict internally (no pre-allocation), skipping the capacity check has zero performance or memory overhead.Changes
StorageUnitData:storage_size→Optional[int](defaultNone)SimpleStorageUnit:storage_unit_size→Optional[int]simple_storage_bootstrap: handlesnullgracefullyconfig.yaml: defaulttotal_storage_size→nullBackward Compatibility
Fully backward compatible — passing an explicit integer value still enforces the capacity limit as before.