feat: two-stage CI build with cached prereqs and TensorStore FetchCon…#46
Open
sameeul wants to merge 5 commits intoPolusAI:mainfrom
Open
feat: two-stage CI build with cached prereqs and TensorStore FetchCon…#46sameeul wants to merge 5 commits intoPolusAI:mainfrom
sameeul wants to merge 5 commits intoPolusAI:mainfrom
Conversation
…tent
Adds a prebuild job (one per OS/arch) that runs before the parallel
wheel-build jobs. The prebuild populates two caches:
- prereq_cache/: filepattern + pybind11 (keyed on install scripts)
- ts_fc_cache/: TensorStore FetchContent source + build state
(keyed on root CMakeLists.txt + tensorstore_prebuild/CMakeLists.txt)
Wheel-build jobs restore both caches and skip the heavy dependency
compilation on cache hits. FETCHCONTENT_BASE_DIR is wired into
CMAKE_ARGS for all platforms so cmake reuses the pre-fetched
TensorStore source tree.
New files:
ci-utils/tensorstore_prebuild/ — minimal cmake project for TS-only prebuild
ci-utils/prebuild_linux.sh — runs inside manylinux container
ci-utils/install_sys_deps_linux.sh — builds zlib/jpeg/png to /usr/local
ci-utils/before_all_{linux,macos}.sh / before_all_win.bat
— replace inline CIBW_BEFORE_ALL commands
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…cmake Without CALL, invoking a .bat file from within another batch context transfers execution permanently — the caller never resumes. This caused the xcopy and cmake TensorStore steps to be silently skipped after install_prereq_win.bat completed. Also adds --parallel to cmake --build on Windows to match the -j$(nproc) flag used on Linux/macOS, and explicitly copies DLLs to %TEMP%\argolid\bin in before_all_win.bat's cache-miss path rather than relying on the conditional copy inside install_prereq_win.bat. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously a single prebuild matrix job blocked all wheel-build jobs until every OS finished. Now each OS has its own named prebuild job (prebuild_linux, prebuild_macos_intel, prebuild_windows, prebuild_macos_arm64) and its own wheel-build job that depends only on its matching prebuild. Linux wheels start building as soon as the Linux prebuild is done, independently of macOS/Windows. The build_wheels matrix is also split into per-OS jobs (build_wheels_linux, build_wheels_macos_intel, build_wheels_windows, build_wheels_apple_arm64), each retaining its cibw_build matrix for the 4 Python versions. Cache keys are hardcoded per job rather than derived from matrix variables. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add ccache detection to CMakeLists.txt and tensorstore_prebuild/CMakeLists.txt via CMAKE_CXX/C_COMPILER_LAUNCHER so all compilation units go through ccache - Install ccache in prebuild_linux.sh and before_all_linux.sh via dnf - Set CCACHE_DIR=/project/ccache_dir in prebuild (env var) and wheel-build (CIBW_ENVIRONMENT_LINUX) so the cache is in the host-accessible workspace - Add ccache_dir GitHub Actions cache to prebuild_linux (saves after prebuild) and build_wheels_linux (restores from prebuild's key, saves per Python version) - Prebuild now also triggers Docker run on ccache miss, not just prereq/ts misses - Add ccache_dir/ to .gitignore Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
brew's curl dynamically links against openssl. Uninstalling openssl before before_all_macos.sh (which uses curl to download zlib/jpeg/png) caused curl to abort with a missing libssl dylib error. Reordered CIBW_BEFORE_ALL_MACOS so before_all_macos.sh (curl downloads) runs first, then openssl is removed before cmake runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a prebuild job (one per OS/arch) that runs before the parallel wheel-build jobs. The prebuild populates two caches:
Wheel-build jobs restore both caches and skip the heavy dependency compilation on cache hits. FETCHCONTENT_BASE_DIR is wired into CMAKE_ARGS for all platforms so cmake reuses the pre-fetched TensorStore source tree.
New files:
ci-utils/tensorstore_prebuild/ — minimal cmake project for TS-only prebuild
ci-utils/prebuild_linux.sh — runs inside manylinux container
ci-utils/install_sys_deps_linux.sh — builds zlib/jpeg/png to /usr/local
ci-utils/before_all_{linux,macos}.sh / before_all_win.bat
— replace inline CIBW_BEFORE_ALL commands