ci+deps+test(persistence): run the S3+ES composite suite on every agent, fix its latent defects, and bump past the crc-fast x86_64 panic

[aacruzgon]

Run the S3+ES composite suite on every agent, fix its latent defects, and remove the crc-fast panic at the root

Summary

crates/persistence/tests/s3_es_tests.rs self-skips unless RUN_MINIO_S3_ES_TESTS=1, and no workflow ever set it. The suite had never run in CI — every green run passed it vacuously (#390).

Un-skipping it uncovered two further layers:

1. The suite was born broken. 3 of its 10 tests fail deterministically anywhere they actually execute. Fixed here.
2. The AWS S3 SDK panicked on the self-hosted Windows agent. This is the same defect ci: fix Windows MinIO step crc-fast panic + add CI timeouts #345 hit, and it is an upstream crc-fast bug rather than a property of the runner. Rather than gate the suite to Linux, this PR fixes it at the root by bumping the SDK past the fixed crc-fast — which also lets ci: fix Windows MinIO step crc-fast panic + add CI timeouts #345's checksum workaround be removed.

Net result: the suite runs unconditionally on every agent, verified 10/10 on the Windows agent that previously failed 8/10, and the Windows MinIO leg regains the checksum coverage it has been forfeiting since #345.

Why

Three separate ways this was silently verifying nothing:

• A suite that never ran. RUN_MINIO_S3_ES_TESTS was unset everywhere, so s3_es_tests skipped in every CI run since it was written. The finished in 0.01s signature was there the whole time.
• A suite that couldn't have worked. Even when run, its harness never indexed anything into Elasticsearch, so every positive search assertion was failing — and its delete/isolation assertions could pass vacuously on timing.
• A production bug hiding as a CI quirk. The crc-fast panic was treated as a Windows-runner peculiarity. It is not: any HFS deployment writing to S3 on a CPU without SSE4.1+PCLMULQDQ panicked on every upload.

Changes

crates/persistence/tests/s3_es_tests.rs

1. The composite never indexed anything into ES. The harness used the default sync config — SyncMode::Asynchronous — but never called start_sync_workers(). With an S3 primary (no durable outbox), every secondary sync failed with "async worker was never started", and composite.create/update/delete swallow sync errors by design (primary succeeded, warn! only). Nothing ever reached ES, so all positive search assertions failed. The harness now builds the composite with SyncMode::Synchronous, which is also the semantics the tests assert. Verified by writing to the ES backend directly (worked all along) — the bug was purely the composite wiring in the harness.
2. Fixed 100ms sleeps → bounded polling. ES refresh visibility on a loaded runner can lag past any fixed sleep, so a search_until() helper polls (10s bound). The delete test now waits for the create to be searchable before deleting, so its absence assertion can't pass vacuously; the tenant-isolation test waits for tenant-a's visibility before asserting tenant-b sees nothing, making the isolation check meaningful rather than timing-dependent.

Cargo.lock, crates/sof/Cargo.toml

• aws-sdk-s3 1.119.0 → 1.122.0, pulling aws-smithy-checksums 0.63.12 → 0.64.4 → crc-fast 1.6.0 → 1.9.0, past the upstream fix.
• helios-sof's lru 0.12 → 0.16, following the SDK's own pin (^0.12 → ^0.16) so the lockfile keeps a single lru. The comment there already prescribed exactly this: "Bump to the latest once the AWS SDK updates its lru pin." Usage is new/get/put only; no source change needed.

.github/workflows/ci.yml

• RUN_MINIO_S3_ES_TESTS: "1" on the existing "Run tests with all features" step — no new step, no extra build, no OS gate.
• Removed AWS_REQUEST_CHECKSUM_CALCULATION / AWS_RESPONSE_CHECKSUM_VALIDATION from the Windows MinIO step. See Relationship to #345. ci: fix Windows MinIO step crc-fast panic + add CI timeouts #345's job/step timeouts are untouched.

Investigation

1. Reproducing the Windows failure

An earlier revision of this PR gated the suite to Linux on two assumptions: that the AWS SDK panics on the Windows agent, and that the ES container was unproven there. To replace conjecture with evidence, run 30303350382 (commit 2717104ce, reverted) removed the gate and pinned test-rust to [self-hosted, Windows]. The job failed as intended:

thread 's3_es_test_create_then_search' panicked at
  ...\crc-fast-1.6.0\src\arch\mod.rs:121:13:
x86 features missing (SSE4.1 && PCLMULQDQ)

test result: FAILED. 2 passed; 8 failed; finished in 13.07s

Assumption 1 confirmed. The only two passing tests are precisely the two that never PUT an object to S3.

Assumption 2 disproved — ES was never a blocker. s3_es_test_read_missing_returns_none passed, and it runs the whole make_harness() path: ES testcontainer start, ElasticsearchBackend construction, and initialize().await.expect("initialize ES backend"). The container works on that agent.

2. Root cause

crc-fast 1.6.0, src/arch/mod.rs:

ArchOpsInstance::SoftwareFallback => {
    #[cfg(target_arch = "x86")]
    crate::arch::x86_software_update(state, bytes, params);

    // This should never happen, but just in case
    panic!("x86 features missing (SSE4.1 && PCLMULQDQ)");
}

A table-based software fallback has existed since 1.2.0, but the call to it is gated #[cfg(target_arch = "x86")] — 32-bit x86 only. On x86_64 that line compiles to nothing and control drops into the panic!. (The helper it would have called is broken the same way: its inner software call is also cfg'd to x86, and it discards the return value, falling through to an identical panic.) The // This should never happen comment is the tell — the author assumed every x86 CPU advertises SSE4.1+PCLMULQDQ; the agent is a virtualized guest that does not. The SDK reaches this on every upload via aws-sdk-s3 → aws-smithy-checksums → crc-fast.

Upstream fixed it in 1.7.1, under "Ensure Miri passes on x86_64" (PR #26) — not advertised as a CPU-compat fix, which is why the changelog doesn't flag it:

version	panics	clean software fallback
1.6.0	yes	no
1.7.0	yes	no
1.7.1	no	yes
1.8.0 – 1.10.0	no	yes

3. Why the fix needs an SDK bump

Not reachable by cargo update alone: aws-smithy-checksums 0.63.12 requires crc-fast ~1.6.0 (>=1.6.0, <1.7.0), there is no 1.6.1, and a [patch.crates-io] override cannot satisfy a tilde requirement either. It has to arrive through the SDK.

Why 1.122.0 and not latest: aws-sdk-s3 1.140.0 needs aws-smithy-types 1.6.1 → serde ^1.0.228, and the workspace pins serde = { version = "=1.0.224" } (Cargo.toml:40). 1.122.0 is the earliest release carrying the fix and it resolves against that pin.

Relationship to #345

cc @smunini — #345 hit this exact panic in the Windows MinIO step and diagnosed it correctly down to the line number and dependency chain. Its workaround was the right call at the time: the fix genuinely wasn't reachable, for the tilde-pin reason above, and #345 said so explicitly, naming the root fix as "expose host CPU features (SSE4.1 + PCLMULQDQ) to the Windows runner's hypervisor."

What's new here is the mechanism. #345 described crc-fast 1.6.0 as having "no software fallback on x86_64" — true in effect, but the fallback exists and is merely cfg'd out, which makes this an upstream bug with an upstream fix rather than a hardware requirement. That opens a third option beyond "workaround" and "change the hypervisor": bump the SDK. With the panic gone, #345's env opt-out is unnecessary and is removed here.

This matters for coverage, not just tidiness. WHEN_REQUIRED doesn't merely route around a crash — it disables the SDK's default integrity protection, so the Windows leg was running a different SDK configuration than the Linux leg, and the checksum path (the thing that was broken) was the one part Windows never exercised. Both legs now run identical defaults and differ only by shell.

#345's remaining caveat still stands: "any other hardware-CRC/crypto path on that runner could still hit this." This fixes crc-fast specifically, not the general class of hardware-accelerated paths on a guest with a restricted CPU feature set.

Testing

CI, on the Windows agent that originally failed

Each run below was temporarily pinned to [self-hosted, Windows] to guarantee it landed on github-agent4; every pin has been reverted, and the branch ships runs-on: [self-hosted].

run	change under test	s3_es_tests	MinIO settings
30303350382	gate removed, no SDK bump	8 failed / 2 passed	not reached
30310519486	+ SDK bump	10 passed / 0 failed	opt-out still in place
30318227484	+ #345 opt-out removed, main merged	10 passed / 0 failed	5 passed / 0 failed

Final run (job), with the SDK's default CRC32 path active on the CPU that used to panic:

running 10 tests
test s3_es_test_create_then_search ... ok
test s3_es_test_delete_then_search ... ok
test s3_es_test_multi_tenant_isolation ... ok
test s3_es_test_history_from_s3 ... ok
test s3_es_test_read_from_s3 ... ok
test s3_es_test_read_after_delete_returns_gone ... ok
test s3_es_test_s3_search_returns_unsupported ... ok
test s3_es_test_read_missing_returns_none ... ok
test s3_es_test_update_then_search ... ok
test s3_es_test_vread_from_s3 ... ok

test result: ok. 10 passed; 0 failed; finished in 14.45s

running 5 tests
test test_minio_settings_create_only_single_winner ... ok
test test_minio_settings_concurrent_patches_never_lose_an_update ... ok
test test_minio_settings_delete_is_idempotent ... ok
test test_minio_settings_round_trip ... ok
test test_minio_settings_stale_if_match_conflicts ... ok

test result: ok. 5 passed; 0 failed; finished in 0.80s

14.45s of real container work, not the finished in 0.01s skip signature — the first genuine full pass of this suite on a non-Linux agent. Because the eight previously-panicking tests had never reached their ES assertions on Windows, passing them also proves ES indexing, search visibility, and the search_until() bound work there — not merely that the panic is gone. And because real uploads now compute CRC32 on a CPU lacking those instructions, the crc-fast 1.9.0 fallback is genuinely exercised rather than just present.

Local

• RUN_MINIO_S3_ES_TESTS=1 cargo test -p helios-persistence --test s3_es_tests --features s3,elasticsearch — 10/10 against real MinIO + ES containers.
• cargo clippy --all-targets --all-features with CI's exact -A flags — exit 0, re-run after merging main.
• cargo test -p helios-sof --lib — 127/127 (the lru bump).
• cargo check -p helios-persistence --features s3,elasticsearch --tests and cargo check -p helios-sof --all-targets — clean.
• cargo metadata exits 0 with no lockfile rewrite after the main merge, confirming the merged Cargo.lock is consistent.
• cargo fmt --all applied.

Notes

• This is a product fix, not just a CI fix. Any HFS deployment writing to S3 on a CPU without SSE4.1+PCLMULQDQ panicked on every upload. CI merely surfaced it.
• macOS is untested. Linux and Windows agents are both proven now; ci.yml has a macOS linker branch, so an agent may exist. crc-fast's aarch64 path needs NEON+AES, which Apple Silicon has, so the same failure is unlikely — but it has not been demonstrated.
• Future SDK bumps are bounded by two constraints worth remembering: the serde = "=1.0.224" workspace pin caps aws-sdk-s3 below 1.140.0, and helios-sof's lru must track whatever the SDK pins or the lockfile carries two versions whenever s3 is enabled.
• Possible product-level footgun surfaced (not addressed here): a composite configured with async sync but no started workers fails every secondary sync and only warn!s — a production misconfiguration would silently lose the entire search index. Flagged on Test suites can pass vacuously: env-gated tests silently skip in CI (sibling of #389) #390 for triage.
• History of the failure: earlier revisions of this PR appeared green because the Test Rust job landed on the Windows agent, where the env var was empty and the suite skipped — a live demonstration of the exact vacuous-pass failure mode Test suites can pass vacuously: env-gated tests silently skip in CI (sibling of #389) #390 describes.
• The anti-vacuous count assertion originally proposed here was dropped per the no-more-steps constraint; its guarantee moves to Test suites can pass vacuously: env-gated tests silently skip in CI (sibling of #389) #390 PR B (guards fail loudly in CI at the code level).
• Remaining Test suites can pass vacuously: env-gated tests silently skip in CI (sibling of #389) #390 items land separately: PR B (guard hardening, MongoDB testcontainer migration, #[ignore] for the AWS suite, MinIO container run_id label) and PR A2 (scheduled OIDC workflow for s3_tests.rs, pending IAM setup). The Redis twin-suite contract check rides with feat(cluster): make HFS cluster-capable behind a load balancer #269.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!