Skip to content

cue leaves placeholder mug/weight caches on decoded cells, breaking Noun equality and jam dedup #30

Open

Description

@thomas-kroes

Summary

cue_bitslice in crates/iris-ztd/src/noun.rs builds each cell's children as placeholder handles and then writes the decoded noun through a raw pointer into the Arc's contents:

// 10 tag: cell
let head = HashNoun::from(atom(0));
let head_ptr = Arc::as_ptr(&head.0) as *mut _;
let tail = HashNoun::from(atom(0));
let tail_ptr = Arc::as_ptr(&tail.0) as *mut _;
*dest_ptr = Noun::Cell(head, tail);

The write through head_ptr/tail_ptr replaces only the noun field of HashNounContents — the cached mug and weight keep the values computed for atom(0). Every cell produced by cue therefore carries degenerate hash caches (mug = mug(0), weight = 1).

Consequences

  1. Noun::eq is inconsistent across provenances. Cell equality short-circuits on the cached mug/weight of the children. Two value-equal nouns compare unequal when one was cue'd (placeholder caches) and the other was built via constructors (correct caches). We observed value-equal trees where the parents compare unequal while all leaves compare equal.

  2. jam misses backrefs on mixed-provenance nouns, producing non-canonical (longer) encodings. find_backref keys candidates by (weight, mug); a duplicate subtree is not found when its cache state differs from the first occurrence's. Concretely: in iris-blocks, jam(raw_tx.to_noun()) for a v1 tx whose typed value embeds cue'd Noun fragments (witness hax_map) came out 482 bits larger than the same value jammed with uniform caches — which broke reconstructed total_size against nockchain consensus (+compute-size-jam). See Fix sync stalling on tx-bearing blocks from current nodes iris-blocks#8 for the full analysis; the indexer now works around it by jamming the noun exactly as received.

  3. jam/equality on cue'd data degrade to quadratic probing. All cue'd subtrees share the single (1, mug(0)) bucket, so every node linearly probes (with deep equality checks) every previously seen node. Jamming a cue'd ~275 KB transaction takes minutes in debug builds; a sweep over ~75k stored transactions did not finish in 10 hours.

Fix sketch

Compute the real mug/weight for each cell after its children are decoded (e.g. process a fix-up pass when the BackRef stack entry completes the cell, or restructure cue to build bottom-up), so that HashNounContents invariants hold for all construction paths. That restores consistent equality, canonical jam output, and hash-bucket dispersion in one go.

Verified at checkout 583ffa5.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions