Skip to content

Use the shared fixed-path registry and preserve legacy cache behavior - #620

Merged
iskandr merged 4 commits into
mainfrom
refactor/589-shared-registry
Sep 29, 2026
Merged

iskandr merged 4 commits into
mainfrom
refactor/589-shared-registry

Conversation

@iskandr

@iskandr iskandr commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Replace hitlist's version resolution, path lookup, receipt hashing and manifest publication with datacache 1.15.0's shared VersionedFileRegistry. Hitlist retains a small adapter for its public exception default, progress/cache messages, 300-second socket timeout and literal-URL decompression policy.

Existing <root>/<name>/<version>/<filename> files and root manifests remain in place; Path results, status keys and silent ensure() cache hits are preserved. New receipts use bounded-memory hashing. Per-root writer serialization fixes the reproduced race where two successful downloads left one manifest entry.

The generation-bundle registry was not a compatible storage replacement, so the reusable fixed-path implementation was moved upstream in openvax/datacache#84 and published as 1.15.0. Its 764 tests and full Linux/Python 3.9–3.14 plus macOS matrix passed; the two-process regression fails when writer serialization is removed. Published wheel/sdist hashes match the clean-master build.

Hitlist validation: format/lint and focused adapter/download/inventory/public-API checks pass against the published datacache wheel. Direct comparison with released hitlist 1.63.14 matches legacy paths, return values, status dictionaries and offline reuse, without new metadata. The full Python 3.9–3.12 unit matrix, all 45 corpus integration tests, lint, documentation build and combined coverage pass on final head 52e20fa (CI run).

The release workflow budget increases from 45 to 90 minutes to address #612: a passing serial unit phase previously took 37 minutes before integration/setup. The one-worker limit, complete test phases, memory guards and artifact/source verification remain intact. Its PR release-build gate passed in 47m54s, beyond the old timeout: 2,413 unit passes, one optional-data skip, all 45 integration passes, successful distributions and provenance artifacts. Heavy local validation was stopped after the user reported memory pressure; the partial run (1,185 passes and an ENOSPC failure) is not a passing gate. Remaining release validation runs in CI. Merged as 96035cf. The final clean-main Release build 36618631704 passed format, lint, full test.sh (2,413 unit passes, one optional-data skip, all 45 integration passes), build and distribution checks. Source/run provenance and both artifact hashes were verified against clean main before upload. Wheel metadata requires datacache >=1.15.0. PyPI 1.63.16 is published, and both published SHA-256 hashes and sizes match the tested artifacts. This stage is merged and deployed. The normal workspace environment also passes check_dev_install.py at 1.63.16 with datacache 1.15.0.

Closes #589. Closes #618. Closes #612.

Builds on the preceding downloader/inventory stage in #617, now published as 1.63.15 with verified PyPI hashes. Version 1.63.16. The separately reported training-data issues #614–#616, #18, #593, #588 and #599 remain open for the requested discussion; no training labels or exports change here.

@iskandr
iskandr marked this pull request as ready for review September 29, 2026 15:50
@coveralls

coveralls commented Sep 29, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 36612670698

Coverage increased (+0.01%) to 87.367%

Details

  • Coverage increased (+0.01%) from the base build.
  • Patch coverage: No coverable lines changed in this PR.
  • 43 coverage regressions across 1 file.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

43 previously-covered lines in 1 file lost coverage.

File Lines Losing Coverage Coverage
downloads.py 43 79.38%

Coverage Stats

Coverage Status
Relevant Lines: 9554
Covered Lines: 8347
Line Coverage: 87.37%
Coverage Strength: 0.87 hits per line

💛 - Coveralls

@iskandr
iskandr changed the base branch from refactor/589-datacache-downloads to main September 29, 2026 17:31
@iskandr
iskandr merged commit 96035cf into main Sep 29, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants