Skip to content

Fix flaky OU-Bridge tests: compare against exact moments - #335

Merged
ChrisRackauckas merged 1 commit into
SciML:masterfrom
ChrisRackauckas-Claude:fix-ou-bridge-statistical-test
Sep 13, 2026
Merged

ChrisRackauckas merged 1 commit into
SciML:masterfrom
ChrisRackauckas-Claude:fix-ou-bridge-statistical-test

Conversation

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member

Summary

The scalar, vector, and inplace OU-Bridge testsets in test/bridge_test.jl compared three independent Monte Carlo estimates to each other at atol = 1e-3 over a 500k-sample loop. The Random.seed!(1234) calls between the samplers were dead code: solve reseeds each NoiseProcess from system entropy (reseed = true is the default), so every realization is an independent draw.

A difference of two 500k-sample estimates carries ~4.3 standard errors of headroom at atol = 1e-3, so the test flaked on a few percent of CI runs on every Julia version. The Inplace OU-Bridge failure on Julia 1.13 CI was this tail event, not a 1.13 regression — the same failure mode has appeared on other versions.

The fix

Each sampler is now compared directly against the exact OU marginal moments — mean μ(1 − e^{−Θt}), variance σ²(1 − e^{−2Θt})/(2Θ) — at the same atol = 1e-3. That gives ~6 standard errors of headroom and is a strictly stronger test: a bias shared by all three samplers can no longer hide in a pairwise comparison.

This also matches the file's existing convention — the Brownian Bridge testset already compares timestep_mean to analytical moments.

The bridge estimator is compared against the unconditional OU moments, which is correct: an OU bridge pinned to an endpoint drawn from the forward process's own distribution reproduces the forward law when marginalizing over the endpoint.

Discriminating-test evidence

This is a flakiness fix, so "fails before / passes after" doesn't apply per-run — instead, evidence that the old formulation was noise-limited and the new one is not:

  1. Before (master): the three samplers were compared pairwise; instrumented on Julia 1.13, all three track the analytical moments within ~4×10⁻⁴ (~2.5 SE) — no bias — yet pairwise differences of two estimates reach ~1.5×10⁻³ in tail realizations, exceeding atol = 1e-3. Reproduced failure: Inplace OU-Bridge in https://github.com/SciML/DiffEqNoiseProcess.jl CI (master, Julia 1.13).
  2. After: against fixed analytical moments each estimate has ~6 SE of headroom at the same tolerance.

Verification

  • test/bridge_test.jl on Julia 1.13.0: all testsets pass — Brownian Bridge 80/80, Scalar/Vector/Inplace OU-Bridge 6/6 each.
  • Same file on Julia 1.12.7: all pass.
  • GROUP=QA equivalent (test/qa/qa.jl) on 1.13: 21/21 pass.
  • Runic format check: clean. typos: clean.

Docs build skipped — test-only change, no docstrings or public API touched.

🤖 Generated with Devin (model: SWE-2 Max)
Session: /home/crackauc/.local/share/devin/cli/summaries/history_d0e0e74010b34bc5.md

The scalar, vector, and inplace OU-Bridge testsets compared three
independent Monte Carlo estimates to each other at atol = 1e-3. The
Random.seed!(1234) calls between the samplers were dead code: solve
reseeds each NoiseProcess from system entropy (reseed = true), so every
realization is an independent draw. A difference of two 500k-sample
estimates carries ~4.3 standard errors of headroom at atol = 1e-3, so
the test flaked on a few percent of CI runs on every Julia version --
the Inplace OU-Bridge failure on 1.13 was this tail event, not a
regression.

Each sampler is now compared directly against the exact OU marginal
moments, mean mu(1 - e^{-Theta t}) and var sigma^2(1 - e^{-2Theta t})/(2Theta),
which gives ~6 standard errors at the same tolerance and is a strictly
stronger test: a bias shared by all three samplers can no longer hide in
a pairwise comparison. Verified on Julia 1.12 and 1.13 (all testsets
pass; the bridge estimator tracks the unconditional moments because its
endpoints are drawn from forward samples of the same process).

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Devin <noreply@cognition.ai>
Agent-Harness: Devin CLI
Agent-Model: SWE-2 Max
Agent-Session: /home/crackauc/.local/share/devin/cli/summaries/history_d0e0e74010b34bc5.md
@ChrisRackauckas
ChrisRackauckas merged commit 8339b36 into SciML:master Sep 13, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants