Skip to content

Hash embedded objects by type and content, keeping ids session-stable - #157

Merged
ChrisRackauckas merged 1 commit into
masterfrom
claude/vibrant-euler-6k5jvw
Sep 24, 2026
Merged

ChrisRackauckas merged 1 commit into
masterfrom
claude/vibrant-euler-6k5jvw

Conversation

@ChrisRackauckas

Copy link
Copy Markdown
Member

Please ignore until reviewed by @ChrisRackauckas. Opened by an AI agent; not yet reviewed by a human.

Fixes #155. This is an alternative to #156, which fixes the same bug but loses compilation caching.

What changed and why

expr_to_id hashed every non-Expr leaf by string(leaf) (src/RuntimeGeneratedFunctions.jl:456 on master). The id is a type parameter of the RGF (:73-75) and keys both the body cache (_cache_body) and the generated_callfunc specialization (:401). Leaves that print alike therefore collided, and the second RGF silently ran the first one's body. This affects more than objects with a summarizing show. On master all of these collide: Scale(2.0) vs Scale(3.0) with a custom show, make_scaler(2) vs make_scaler(3) (same closure type, different captures), x -> x vs x -> "x", and () -> Int32(1) vs () -> 1.

This PR serializes the expression to an unambiguous tagged byte stream and hashes that with SHA-1. Symbols, strings, bit numbers, modules, GlobalRefs and types (by module fullname, type name and parameters) have dedicated encodings. Other objects are encoded by type plus fields recursively, Array/Memory by size plus elements (primitive eltypes as a raw byte block), BigInt/BigFloat by value. Mutable objects are tracked in an IdDict so cycles become back-references. Objects with no fields and opaque storage fall back to objectid, which is safe but uncached.

Why not objectid (PR 156)

objectid of any type, function or struct instance mixes in the defining module's build_id, which is new each session for non-precompiled modules and each time a package is re-precompiled or baked into a sysimage. Mutable leaves hash by address. So under PR 156, the same expression gets a different id in a new session. That reintroduces #117 (precompiled/sysimage RGF signatures must match ids computed at runtime). It also means two equal-content mutable leaves (e.g. [1.0, 2.0] twice) no longer share a compiled body. Measured with a module defined in a script, two fresh processes:

objectid(M.T(1.0)): 7025260776328602294 vs 7025260778881564268
objectid(M.f):      18446744072981704109 vs 18446744072637911334
objectid(GlobalRef(M, :f)): 5534674955889929608 vs 17123887598007707521

The new encoding uses only names, module paths and content, so ids are identical across processes.

Tests

Added to test/core_tests.jl (Core group):

  • Embedded objects are identified by type and content: the collision cases above, for immutable and mutable structs.
  • Equal expressions share one compiled body: distinct-but-equal leaves (immutable and mutable structs, BigInt, BigFloat, arrays, Vector{Any} with undef slots, tuples, types, UnionAll) yield the same RGF type and cached body. It also runs test/shared/expr_id_stability.jl in two fresh processes and asserts the printed ids match.
  • RGFPrecompTest.h embeds a package-defined struct and an Int32 at precompile time. The test rebuilds the expression at runtime and asserts the same RGF type (pkgimage cache hit).

Failing before (master src, new tests), Julia 1.13.0

Embedded objects are identified by type and content: Test Failed at test/core_tests.jl:275
  Expression: f3(1.0) == 3.0
   Evaluated: 2.0 == 3.0
...
  Expression: (g2(1), g3(1)) == (2, 3)
   Evaluated: (2, 2) == (2, 3)
  Expression: (sym(1), str(1)) == (1, "x")
   Evaluated: (1, 1) == (1, "x")
  Expression: i64() isa Int64
   Evaluated: 1 isa Int64
Embedded objects are identified by type and content |    3     7     10

On master the caching testset passes on its own (13/13); master already caches correctly.

PR 156's src with the same tests

The collision testset passes (10/10). The caching testset fails 7 of 13: typeof(f_again) === typeof(f) fails for the mutable struct, the arrays, Vector{Any}, BigInt and BigFloat, and the two-process id comparison fails:

  Expression: ids[1] == ids[2]
   Evaluated: "(0x05a0b7c6, ...)\n(0x77dba16a, ...)\n(0xb63fbac3, ...)\n" == "(0x8cc62eb2, ...)\n(0xc4a1b048, ...)\n(0xd9c636c3, ...)\n"
Equal expressions share one compiled body |    6     7     13

Passing after

GROUP=Core and GROUP=QA with julia --project -e 'using Pkg; Pkg.test()', Julia 1.13.0 (release channel here), x86_64 Linux:

PrecompileTools 1.2 workload                        |    1      1
Embedded objects are identified by type and content |   10     10
Equal expressions share one compiled body           |   13     13
     Testing RuntimeGeneratedFunctions tests passed      (Core)
QA                                                  |   20     20
     Testing RuntimeGeneratedFunctions tests passed      (QA)

GROUP=Core also passes on Julia 1.10.12 (LTS) and 1.11.9 with the same counts. runic --check . and typos are clean.

Hashing cost

expr_to_id timings, minimum of repeated runs. The expression is a 2000-statement block, and the array is a 100,000-element Vector{Float64} leaf:

master this PR
2000-statement expr 4.2 ms 2.7 ms
1e5 Float64 array leaf 26.3 ms 2.2 ms

Push back on

  • All ids change once. Any RGF type stored in an existing pkgimage or sysimage is invalidated by upgrading, as with any hashing change. This needs a new release, but after that ids are stable again.
  • Equal mutable leaves still share a body, as on master. Two equal but distinct arrays embedded in otherwise identical expressions get one id, so the second RGF references the first array. That is a caching-vs-aliasing tradeoff. I kept master's behavior because switching to identity loses cross-session caching for every expression with an embedded array. It is a separate decision from Embedded objects that print alike give two RGFs the same id, so the second runs the first's body #155, which is about different values colliding.
  • Content hashing of internals. Objects reached through fields are hashed by their raw field contents. So something like a Dict hashes by its slot layout, and any Ptr field hashes by address. That can only cause extra cache misses, never collisions.
  • Struct redefinition (1.12+) with an identical name and content yields the same id, as it did with string.

Not verified

  • Downstream packages (ModelingToolkit, Symbolics), and a PackageCompiler sysimage. The cross-process stability test stands in for these.
  • QA on 1.10/1.11, julia pre, macOS, Windows and 32-bit. Those run only in CI.
  • Docs build was not run. The PR does not touch docstrings, docs/ or public API.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UjNooJk1j2y5NpVaJ9MAp8


Generated by Claude Code

`expr_to_id` hashed every non-Expr leaf by `string(leaf)`, so distinct
leaves that print alike (objects with a summarizing `show`, closures with
different captures, `:x` vs "x", `Int32(1)` vs `1`) got the same id and
the second RGF silently ran the first one's body (issue #155).

Serialize leaves into an unambiguous tagged byte stream by type and
content instead. Unlike `objectid`, this stays independent of module
build ids and addresses, so equal expressions still share one cached
body and ids still match across sessions, pkgimages and sysimages.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UjNooJk1j2y5NpVaJ9MAp8
@ChrisRackauckas
ChrisRackauckas marked this pull request as ready for review September 24, 2026 17:32
@ChrisRackauckas
ChrisRackauckas merged commit ddf352d into master Sep 24, 2026
19 checks passed
ChrisRackauckas referenced this pull request Sep 26, 2026
Agent-Harness: Claude Code
Agent-Model: claude-opus-5-5[1m]
Agent-Session: local Claude Code session 3cd6500a-1f81-46b5-ac0b-c466e15b6a53 on Chris's Mac (session ID, no URL)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Embedded objects that print alike give two RGFs the same id, so the second runs the first's body

2 participants