Skip to content

fix: resolve purge-test C++ compilation and pacing validation blockers #618

Closed

Description

@zackees

Context

Blocker extracted from #614 and draft PR #616. The CPU candidate V7 is published for review, not accepted or measured. Two validation defects need resolution before that work resumes.

1. Existing secure C++ compilation failure

test/test-purge-holes.c:test_owner_sweep_pacing uses early goto done paths across initialized scalar locals (stamped_at, before_in, after_in, elapsed, before_out, after_out). This is valid C but rejected when CMake intentionally sets this .c source to LANGUAGE CXX under MI_USE_CXX. The test calls internal allocator functions with C++ linkage in that configuration, so simply forcing this test to C is not a demonstrated solution.

Reproduced with MI_SECURE=5, MI_DEBUG=3, C++17 against byte-identical test source on frozen baseline 227eef06ff03f92966f38508441d6c2f60a75135. Diagnostics report jumps to done at line 1939 crossing initializations. A likely minimal repair is an inner scope for measurement locals, with cleanup outside; preserve the timing of clock/counter reads and all checks.

2. Unresolved purge-pacing assertion

Actual ASan + owner-gate Debug candidate run: 34 cases passed; new-thread-does-not-sweep-at-once failed at line 2313 with 6 discards, expected 0. The matching baseline build passed with 0 discards. Do not call this a baseline failure or an accepted flaky test.

The test measures a process-wide discard counter around thread C. Background scavenger work remains possible, so the delta is not necessarily attributable to C. This is a hypothesis needing thread/reason attribution. Diagnostics were enabled and unsampled; both runs recorded zero successful large repurposes, proving V7's specialized removal did not execute in the failing suite. That narrows the investigation without waiving the assertion or proving a test-only bug.

Evidence / reproduction

Draft PR #616 contains the evidence archive, digest and handoff. Archive SHA256: c1a9c4ff57f1694031ed38aeee4b1e1dc9f9f05802fe925b76afe642d06465e1 (46 members; 4,864,167 bytes).

The archive preserves actual candidate and baseline executables, CMake caches/flags, commands, failed and passing logs, current source patch/new headers, and per-member hashes. Relevant members include v7-baseline-cxx-command.json, v7-baseline-cxx-error.log, v7-cxx-build.log, v7-asan-tests.log, and baseline/Testing/Temporary/LastTest.log. Original local paths in command JSON must be adapted to the checkout location.

C candidate commit 2ec4d32e; generated Rust C 28ef6ad7; fixed baseline above. The focused large-span test passes in C, secure C++ and actual ASan + owner-gate builds. No V7 performance run has been dispatched.

Proposal

  1. Fix C++ scope compatibility without moving measurement calls earlier, removing const unnecessarily, changing cleanup, or weakening checks.
  2. Attribute each discard observed by the pacing scenario to its thread and trigger. Establish a focused, reproducible RED signal for the real defect.
  3. Fix the test if accounting is polluted, or allocator logic if pacing is genuinely wrong. Preserve separate coverage for background release and owner-side pacing; do not disable required behavior merely to make the test pass.
  4. Record the disposition on perf: reduce large-page reuse CPU after #611 while preserving RSS and p99 #614 and perf: large-page reuse CPU investigation and draft V7 (#614) #616 before resuming CPU measurement.

Acceptance criteria

  • Original C++ compiler repro is RED; repaired test compiles and runs in C and secure C++ with the same checks and read timing.
  • Pacing failure has concrete thread/trigger attribution and focused RED → GREEN evidence; an unchanged passing rerun is not a fix.
  • Original candidate failure and matching baseline pass remain in the record; no threshold relaxation, swallowed failure or waived assertion.
  • Validate affected ASan/owner-gate, ordinary C/C++, native MSVC and Windows GNU configurations, plus relevant scavenger/release tests.
  • Preserve C/Rust source separation and regenerate Rust if allocator C changes; pass the required local gate before push.
  • Link the fix/evidence and unblock decision on perf: reduce large-page reuse CPU after #611 while preserving RSS and p99 #614 and draft PR perf: large-page reuse CPU investigation and draft V7 (#614) #616. This issue does not claim completion of the CPU improvement.

Decisions

  • Priority P2; this is a validation blocker, not a demonstrated V7 memory-safety defect.
  • Keep this issue focused on the two purge-test blockers; CPU acceptance remains perf: reduce large-page reuse CPU after #611 while preserving RSS and p99 #614.
  • No new workflow files, unrelated policy changes, or relaxation of perf acceptance.
  • The current publication is a handoff, not authorization to merge the CPU candidate.

Open questions

Which thread and purge trigger produced the six observed discards? Does the deterministic fix belong in test attribution or allocator pacing?

Related

Activity

  1. zackees commented on Oct 23, 2026

    @zackees
    OwnerAuthor

    #618 disposition — fixed on #616 head f0433230 (C bd05be78, Rust vendor 75e4afc1)

    1. Secure C++ compile. RED: frozen baseline 227eef06 with MI_USE_CXX=ON, MI_SECURE=5 MI_DEBUG=3, C++17 fails with jump to label 'done' crosses initialization (line 1939). Fix: the measurement section of test_owner_sweep_pacing sits in an inner scope, so the early goto done skips it whole. Clock and counter reads keep their order, every check is unchanged, and cleanup is unchanged. GREEN: compiles.

    1b. Allocator defect found once the file compiled as C++ (also on baseline). The secure-C++ suite segfaulted every time (3/3 runs on baseline and candidate) in idle-thread-releases-large-page. Backtrace: scavenger → _mi_pages_release_retired → mi_page_discard_slack → _mi_os_discard → eager-zero memset on the MI_SECURE≥5 trailing guard page (PROT_NONE). The slack range ran to the end of the slices and included the guard page. Fix (src/page-holes.c): subtract _mi_os_secure_guard_page_size() from hi, matching page_noguard_size in arena.c. GREEN: secure C++ purge-holes 32/32, 3/3 runs.

    2. Pacing assertion: attributed to the background scavenger, not to thread C.

    build, 144 runs, 48 concurrent fails (discards)
    candidate, scavenger on 10/144 (8 discards)
    baseline 227eef0, scavenger on 33/144 (8 discards)
    baseline, MIMALLOC_SCAVENGER=0 0/144
    fix (--new-thread-pacing, own process, scavenger off) 0/144

    The test reads a process-wide counter, so the scavenger's sweep of thread A's abandoned page during C's life was charged to C. The defect predates V7: the baseline fails more often. V7's path was not exercised (0 repurposes). #478 is about the owner thread's first tick, so the case now runs in its own process with no scanner (test-new-thread-pacing, MIMALLOC_SCAVENGER=0;MIMALLOC_PURGE_HOLES=1). That mirrors test-reserved-page-reuse. The background-release cases stay in the default invocation with the scavenger on. No threshold changed. The original candidate failure and the baseline pass from the evidence archive stay on record. The baseline passing once was luck under low load.

    Also fixed: PR CI build (release) and the other Release rows failed with -Werror=unused-parameter on page in mi_page_queue_remove_retired, because the asserts compile out. Added MI_UNUSED(page).

    Local validation: owner-gate Debug (MI_DEBUG_FULL + MI_DIAGNOSTICS + MI_PPROF; no ASan headers on this host, CI's asan row covers ASan) 86/86 passed. Release MI_PPROF 70/70 passed. Secure C++: purge-holes passes. Local gate 45/45.

    Not fixed by this change:

    Native MSVC and win-gnu run in PR CI.

    Unblock decision: #618's validation blockers are resolved. The V7 CPU measurement under #614 may resume. No CPU, throughput or RSS gain is claimed. #616 stays a draft until #614's perf-ab campaign and platform gates pass.

  2. zackees commented on Oct 23, 2026

    @zackees
    OwnerAuthor

    Fixes split out of rejected V7 #616 and onto main in #622 (closes this issue on merge). #616's verdict: V7 rejected (confirmed CPU FAIL on large-class-ephemeral/8).

  3. added a commit that references this issue on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions