Codex/ivashchenko filter

.github/workflows/ci.yml

@@ -177,6 +177,14 @@ jobs:
             os: ubuntu-latest
             arch: x64
             test_set: "particle_filter"
+          - version: '1'
+            os: ubuntu-latest
+            arch: x64
+            test_set: "quadratic_kalman"
+          - version: '1'
+            os: ubuntu-latest
+            arch: x64
+            test_set: "ivashchenko_kalman"
     steps:
       - uses: actions/checkout@v7
       - uses: julia-actions/setup-julia@v2
@@ -207,14 +215,18 @@ jobs:
             Project.toml
           rm -f Project.toml.bak
 
-      # - name: Restrict DynamicPPL to 0.40 for pigeons runs
-      #   if: contains(matrix.test_set, 'pigeons')
-      #   shell: bash
-      #   run: |
-      #     sed -i.bak \
-      #       -e '/^\[compat\]/,/^\[/ s/^DynamicPPL[[:space:]]*=.*$/DynamicPPL = "0.40"/g' \
-      #       Project.toml
-      #     rm -f Project.toml.bak
+      - name: Restrict DynamicPPL to 0.40 for pigeons runs
+        if: contains(matrix.test_set, 'pigeons')
+        shell: bash
+        run: |
+          # Pigeons 0.4 and FlexiChains require disjoint DynamicPPL ranges.
+          # FlexiChains is removed below for these jobs; keep the compatible
+          # DynamicPPL line explicit so a future resolver update cannot select
+          # the incompatible 0.41/0.42 branch.
+          sed -i.bak \
+            -e '/^\[compat\]/,/^\[/ s/^DynamicPPL[[:space:]]*=.*$/DynamicPPL = "0.40"/g' \
+            Project.toml
+          rm -f Project.toml.bak
 
       - name: Remove Mooncake from pigeons runs
         if: contains(matrix.test_set, 'pigeons')

AGENT_PROGRESS.md

@@ -0,0 +1,38 @@
+# Agent progress
+
+## Current task
+
+Implement Ivashchenko's unpruned Gaussian moment-closure filter as a separate filter for raw second- and third-order solutions, and resolve the CI dependency isolation issue.
+
+## Status
+
+- Repository progress file was absent at task start; this file records the current task.
+- Quadratic and cubic Kollmann-style conditional covariance corrections remain implemented.
+- A separate `:ivashchenko_kalman` filter now evaluates raw second-/third-order polynomial
+  solution maps and closes Gaussian moments through fourth/sixth order, respectively.
+- The Ivashchenko filter has coupled theoretical mean/covariance initialization, optional
+  measurement error, partial and fully missing observation support, an RTS smoother, and
+  analytical reverse-mode rules for both raw second- and third-order solutions.
+- CI now isolates the Pigeons/DynamicPPL resolver branch and has a dedicated Ivashchenko test row.
+
+## Verification
+
+- The scalar Gaussian moment reproduction passes.
+- `test/test_ivashchenko_kalman.jl` passes 35/35, including Monte-Carlo moments, both orders,
+  initialization modes, partial/fully missing observations, RTS smoothing, standard deviations,
+  and reverse-vs-forward gradient checks.
+- The isolated cubic tensor reverse check matches ForwardDiff to `8.9e-16`; the public third-order
+  likelihood reverse check differs from ForwardDiff by `3.9e-9` with theoretical initialization
+  and `2.8e-7` with the diagonal prior at the largest parameter gradient.
+- `test/test_quadratic_kalman.jl` passes 33/33 and `test/test_cubic_kalman.jl` passes 30/30.
+- CI YAML parsing and both non-Pigeons and Pigeons resolver probes pass; the direct root
+  `Pkg.test()` remains intentionally unsatisfiable because it includes incompatible optional
+  targets together, which the workflow-pruning steps resolve.
+- `git diff --check` passes; module loading succeeds in the isolated test environment.
+
+## Implementation decision
+
+Ivashchenko's non-pruned Gaussian QKF is a separate algorithm rather than a switch on the
+pruned augmented-state recursion. Its fourth-moment closure and unpruned state-product
+dynamics require separate initialization and moment contractions. The cubic implementation
+is an explicit extension of that idea; it is not attributed to Ivashchenko's second-order paper.

benchmark/quadratic_kalman_sw07_benchmark.jl

@@ -0,0 +1,31 @@
+include("/private/tmp/claude-501/-Users-thorekockerols-GitHub-MacroModelling-jl/c2294a5c-2537-47e2-8f6d-68d07bd438d9/scratchpad/pfenv/qkf_filter.jl")
+cd("/Users/thorekockerols/GitHub/nonlinearisties")
+include("/Users/thorekockerols/GitHub/nonlinearisties/sw07_common.jl")
+
+m    = SW07_MODEL
+obs  = SW07_OBSERVABLES
+data = SW07_DATA(obs)
+pars = sw07_full_parameters(SW07_INITIAL_FREE_PARAMETERS)
+println("model=", m.model_name, "  observables=", obs)
+println("data ", size(data), "  algorithm=", SW07_ALGORITHM)
+
+# --- inversion filter at pruned second order (the reference) ---
+t0 = time(); inv2 = get_loglikelihood(m, data, pars; algorithm = :pruned_second_order,
+                                      filter = :inversion, presample_periods = 4); t_inv = time()-t0
+println("\ninversion  pruned_2nd = ", round(inv2, digits=3), "   [", round(t_inv, digits=3), " s]")
+
+# --- quadratic Kalman filter ---
+opts = MacroModelling.merge_calculation_options()
+MacroModelling.solve!(m, parameters = pars, algorithm = :pruned_second_order, dynamics = true, opts = opts)
+_,_,𝐒,_,_ = MacroModelling.get_relevant_steady_state_and_state_update(Val(:pruned_second_order), pars, m, opts = opts)
+println("max|S2| = ", round(maximum(abs, Matrix(𝐒[2])), digits=3))
+ssn = m.constants.post_complete_parameters.SS_and_pars_names
+oi  = convert(Vector{Int}, indexin(obs, ssn))
+NSSS = get_steady_state(m, parameters = pars, derivatives = false)
+Y = collect(data) .- [NSSS(v) for v in obs]
+t0 = time(); sys = build_qkf(m, 𝐒[1], 𝐒[2], oi); t_build = time()-t0
+println("augmented dim nz = ", sys.nz, "   [build ", round(t_build, digits=2), " s]")
+for mev in (1e-3, 1e-4, 1e-5, 1e-6)
+    t0 = time(); q = run_qkf(sys, Y; me_var = mev, presample = 4); t1 = time()-t0
+    println("  QKF ME var=", rpad(mev,7), " = ", rpad(round(q, digits=3),12), " [", round(t1, digits=2), " s]")
+end