Skip to content

Optimize SPFresh base-delta HBC maintenance - #214

Open
ajroetker wants to merge 427 commits into
mainfrom
spfresh-v2
Open

Optimize SPFresh base-delta HBC maintenance#214
ajroetker wants to merge 427 commits into
mainfrom
spfresh-v2

Conversation

@ajroetker

@ajroetker ajroetker commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add base-delta posting maintenance, bounded repair/debt reporting, thresholded folding, and overlay-cache instrumentation for SPFresh/HBC
  • add scalable centroid-directory comparison paths and optimized-gate benchmark scripts
  • wire DB/runtime maintenance scheduling, resource/elapsed budget checks, crash/recovery proof coverage, and public dense format/version config
  • harden dense catch-up posting maintenance so post-finalize maintenance is best-effort and still contributes to global maintenance stats with idle/catch-up source counters

Evidence

  • production proof PASS: zig build spfresh-production-test --summary all
  • optimized gate rechecked after the warm/cold summarizer fix against /private/tmp/antfly-spfresh-hbc-final-gate-evidence.jsonl; the default gate now FAILS only the read recall denominator checks: read_packed_hbc_hbc, read_flat_rabitq, and read_two_level_rabitq report recall_total=315, while the current gate requires >=1000
  • overlay cache admission-skip checks pass under the fixed warm-preferred summarizer (0 skips), so the original cold/warm concern is resolved; rerun read evidence with a >=1000 recall denominator before claiming a full optimized-gate PASS again
  • 1M VDBB-shaped write evidence still covers hot/random/semantic/append/mixed overwrite distributions; append-heavy deferred debt remains bounded at dirty/delta/overfull 63, max over-capacity 2

Validation

  • zig build lib-db-test --summary all (378/382 passed, 4 skipped)
  • zig build lib-data-runtime-test --summary all
  • zig build spfresh-production-test --summary all
  • bash -n zig/scripts/run_spfresh_hbc_comparison.sh
  • python3 -m py_compile zig/scripts/summarize_spfresh_hbc_comparison.py
  • git diff --check
  • python3 zig/scripts/summarize_spfresh_hbc_comparison.py /private/tmp/antfly-spfresh-hbc-final-gate-evidence.jsonl --optimized-gate check --gate-production-proof-file zig/bench/results/spfresh-production-test.log (expected FAIL on the three stale read recall-denominator checks above)

ajroetker added 30 commits June 19, 2026 14:04
# Conflicts:
#	go/pkg/antfly/src/store/db/indexes/openapi.gen.go
#	go/pkg/sdk/oapi/client.gen.go
#	zig/build.zig
#	zig/pkg/antfly/src/data/runtime.zig
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant