Improved memory usage and linux support#120
Conversation
Removes Iteration and Timestep columns from LandscapeRasters datasheet Schema update
Adds conda-lock environment for windows and linux Updates conda env version OS conditional PROJ reset Schema update for LandscapeRasters removed columns
setting parameter for arrow 24.0 batch reading to avoid memory overload removes large tables from memory after saving
change batch process to output a single file per tile/batch id
adding notes to detail changes
shreeramsenthi
left a comment
shreeramsenthi
left a comment
There was a problem hiding this comment.
Have a couple questions about interoperability on Windows, but looks good for the most part! I think I'll separate out the schema changes for a later release since it's not mission critical and gives users more flexibility to roll back library versions until we roll out a single bigger change that requires a schema update.
| sharedDefinitionsPath <- paste0(ssimEnvironment()$PackageDirectory, "/shared.R") | ||
| return(sharedDefinitionsPath) | ||
| } | ||
| Sys.setenv("MALLOC_CONF" = "dirty_decay_ms:0,muzzy_decay_ms:0") # Disable jemalloc decay to free up memory immediately (Arrow 24.+. Must be done before loading Arrow) |
There was a problem hiding this comment.
Is this a linux-specific change? If so maybe should have a conditional based on .Platform$OS.type ? If not, has it been tested on Win yet?
There was a problem hiding this comment.
Actually, when searching if this was linux specific, I found that jemalloc reached its end of life. Linux is not planning to remove it from future distributions, but future versions of arrow will likely start defaulting to mimalloc (which is the standard memory allocator used by arrow in Windows). It should be harmless to set this environment variable even if it's not used, but it would be worth testing this changes in Windows, and include a mimalloc parameter to replicate this behaviour in Windows and future Linux distributions:
# For the current jemalloc default
Sys.setenv("MALLOC_CONF" = "dirty_decay_ms:0,muzzy_decay_ms:0")
# For the future mimalloc default
Sys.setenv("MIMALLOC_PURGE_DELAY" = "0")
There was a problem hiding this comment.
How does this conda lock file work with Windows? Does SyncroSim provide a way to specify different conda files for Linux and Windows?
There was a problem hiding this comment.
This conda-lock contains parallel environments for Windows and Linux, but it's not handled by SyncroSim yet.
2 participants
Changes updateResampleFireIDsParquet to reduce memory usage and prevent file handle limit crash in Linux
Adds immediate parquet memory decay (necessary for Arrow 24.0+)
Conda-lock parallel environment for Linux
Removes unnecessary columns from LandscapeRasters datasheet
Schema update after removed columns