Skip to content

Add selectable flat, fused and strided secondary-variable scheduling - #404

Merged
moyner merged 5 commits into
sintefmath:mainfrom
moyner:perf/ka-secondary-tiles
Oct 22, 2026
Merged

moyner merged 5 commits into
sintefmath:mainfrom
moyner:perf/ka-secondary-tiles

Conversation

@moyner

@moyner moyner commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

CPU secondary-variable updates can revisit large worker ranges for each dependent property or distribute spatially uneven property costs poorly. Add secondary_scheme to KernelAbstractionsContext, DefaultContext, and ParallelCSRContext, keeping :flat as the default:

  • :flat: retain the existing contiguous worker batches.
  • :fused: group variables by their associated entity and evaluate dependent properties on cache tiles sized using minbatch.
  • :strided: incorporate Secondary variables: strided batches, up to four per thread #385's distribution of up to four interleaved batches per worker.

Evaluation calls the exported secondary_scheme(model) function, which defaults to the context setting and can be specialized by applications. Preserve the setting in adjoint/linear-solver contexts and multimodel backend adaptation. Non-default schemes require CPU host evaluation.

Fused evaluation preserves dependency order within each entity and uses one worker launch per entity group. Entity groups come from the variable definitions, including domains whose generic entity enumeration omits their actual variable entities. Typed-key dispatch keeps heterogeneous state lookups specialized without per-variable tuple definitions. The production-source diff is +130/-9 lines, including documentation and the shared context plumbing. The public selector is included in the manual so the context docstring cross-references resolve.

Validation

  • Focused scheme/context tests and existing backend tests: 3,224 assertions at one thread; 8,894 at 15. Covers all three schemes and contexts, model overrides, serial policy, empty and unequal cell/face counts, interleaved entity-local dependencies, scalar/matrix properties, Float32/Float64, seeded AD values, mutable definitions, and simulator/multimodel propagation.
  • All fixed-input Norne secondary arrays and AD partials match the flat reference bitwise.
  • 18 refined Float64 application samples preserve complete final pressure/saturation fields bitwise and solver work exactly, with zero timed compilation or wasted Newton work. No GPU execution was tested.

Performance

Julia 1.13.2 on a Ryzen 7950X under WSL; 15 workers pinned to physical cores, BLAS=1, Float64/Int32, fixed GMRES/GS/KA system ILU. Norne z8 and Olympus z4 use three original report entries capped at one day each. Each policy warms separately; each measured sample starts with fresh storage/hierarchies; three rounds rotate policy order. These are short diagnostic schedules, not new first-20%-schedule scaling results.

Fixture Secondary phase: flat / fused / strided, s Total: flat / fused / strided, s
Norne z8 0.4889 / 0.3155 / 0.3714 9.5932 / 9.2996 / 9.5569
Olympus z4 0.4904 / 0.3273 / 0.3494 18.7348 / 18.4208 / 18.4543

Fused reduces the secondary phase by about 33-35%, with total median reductions of 3.06% / 1.68%. Strided gives 0.38% / 1.50% total median reductions and increases allocation. Total differences are small relative to setup/solver noise: an earlier fused-only screen gave 1.90% for Norne and -0.55% for Olympus. Both schemes remain opt-in.

The branch and measurements start from 3b90948f; the branch merges cleanly with current main dc6e630c. This PR is independent of the sorted-ILU traversal change and includes no ILU scaling or worker-policy changes.

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 98.27586% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 58.54%. Comparing base (789732f) to head (221164f).
⚠️ Report is 10 commits behind head on main.

Files with missing lines Patch % Lines
src/variable_evaluation.jl 97.43% 1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #404      +/-   ##
==========================================
+ Coverage   57.48%   58.54%   +1.05%     
==========================================
  Files         224      224              
  Lines       30333    30384      +51     
==========================================
+ Hits        17437    17788     +351     
+ Misses      12896    12596     -300     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@moyner
moyner merged commit 4d10c75 into sintefmath:main Oct 22, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants