[thomas-kroes]

Summary

Fixes #6 — against nodes built from current nockchain master (protocol 015 "Zoe"), sync failed with Unable to decode noun on the first tx-bearing block and retried the same batch forever.

Root cause (confirmed against dumbnet's inner.hoon): +heaviest-chain-blocks-range now returns (z-map tx-id:t raw-tx:t) because raw-txs are all consensus stores, while iris decoded the map values as full Tx objects. A wire raw-tx [%1 id spends] cannot decode as Tx, so any block with a non-empty tx map failed; empty blocks passed, which made the stall look intermittent.

Fix

• The blocks-range tx map now decodes as ScryTx, which accepts either shape. The noun encodings are provably disjoint (a full tx is tagged [%0 tx]/[%1 tx]; a v0 raw-tx heads with a cell, a v1 raw-tx with [%1 id spends] where the id is a cell), so older nodes serving full txs keep working unchanged.
• Raw v1 txs are rebuilt into full txs exactly like +new:tx in tx-engine-1.hoon — both derived fields are pure functions of the raw-tx:
  • total_size = bit length of the jammed raw-tx (+compute-size-jam is (met 0 (jam n)) per the hoon docs). Verified that iris_ztd::jam implements canonical backreference deduplication, and the tests anchor the size helper to known hoon jam values ((jam 0) = 2, (jam 1) = 12).
  • outputs mirrors +build-outputs: RawTxV1::outputs derives one note per lock root in BTreeMap order of the lock-root hash; the original (un-normalized, per the hoon — only the source hash uses normalized seeds) seeds are grouped the same way so each note pairs with its seed group as OutputV1 { note, seeds }.
• The reconstructed Tx::V1(TxV1 { raw, total_size, outputs }) keeps the stored jam format, so L1–L4 are untouched.
• v0 raw-txs (only served for pre-tx-engine-1 blocks) are rejected with an explicit RawTxV0Unsupported error instead of a generic decode failure.

Test plan

• New tests in block_range_manager.rs:
  • ScryTx decodes both raw-tx and full-tx nouns into the right variant
  • reconstruction derives total_size from the raw-tx jam and pairs each note with the seeds of its own lock-root group (groups with distinct gift totals across multiple spends, so mispairing would fail; original output_source retained)
  • v0 raw-txs are rejected
  • jam bit length matches met 0, anchored to known hoon jam values
• cargo test --features binary --all-targets (32 passed)
• cargo clippy --features binary --all-targets (no new warnings), cargo fmt --check

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[h33p]

It would be better if zorp stopped breaking scry interface unnecessarily. They also just removed no-pow scry, which is quite important for fast sync: nockchain/nockchain@2601509#diff-44e004bcf232faefa9aba1e79b7d3409037ad6f94f61cf0b3da09b467b6c1525L518

[h33p]

Looked at upstream change - they got rid of txs altogether (good). However, we are somewhat stuck with Tx representation here. So, let's keep it as that, just note that "newer nodes are no longer relying on tx object, but instead, return raw-tx.

I'm prepping a tiny PR on nockchain that brings back no-pow scry for blocks.

[h33p]

What I would do instead is make this

pub struct ScryTx(Tx)

and then, NounDecode for ScryTx has the RawTx -> Tx conversion. There's no need to expose into_tx to the consumer.

Leave a note that this is a workaround for a change introduced in nockchain/nockchain@5decf7c

[h33p]

PR for scry at nockchain/nockchain#150

[h33p]

NOTE: you also must support v0 transactions here. Sync from genesis will break otherwise.

[thomas-kroes]

Both findings addressed in a5bdc0c, plus one more bug the requested validation uncovered.

P1 — v0 raw-txs are now reconstructed, not rejected. Confirmed against the snapshot (35,583 v0 txs at heights 5,629–38,988): a fresh sync would have stalled at 5,629 exactly as described. into_tx is now infallible: v0 txs are rebuilt like +new:tx in tx-engine-0 — total-size from +compute-size-jam, and one output per recipient sig pairing the notes derived by RawTxV0::outputs(height) with the original seeds (mirroring +new:outputs, keyed by the note's sig as +validate:outputs requires). The RawTxV0Unsupported error is gone.

P3 — [raw-tx heard-at] pairs are accepted. Verified the shape in upstream's gen2 normalizer; the decoder now mirrors it (tail must be an atom, head is the raw-tx). All three historical shapes decode, and the encodings remain provably disjoint.

New: total-size is computed from the raw-tx noun exactly as received, not from a typed re-encode. The golden validation you asked for caught this: reconstructed v1 sizes came out up to ~500 bits larger than consensus (first hit: height 40,176, 17,063 vs 16,581). Root cause is in iris-ztd's cue: it builds placeholder HashNouns and writes the decoded noun through a raw pointer without recomputing the cached mug/weight, so cue'd cells keep atom-0 placeholder caches. Re-encoding a typed v1 tx then mixes correct caches (typed fields) with placeholder caches (embedded witness hax_map nouns), which breaks Noun::eq's cache fast-path and makes jam miss backrefs on repeated witness subtrees — inflating the size. v0 txs embed no raw nouns, which is why only v1 was affected. ScryTx::Raw now carries the noun it was decoded from and jams that: uniform-provenance nouns always dedup fully and match hoon's (met 0 (jam n)). (The upstream cue bug — inconsistent Noun equality and effectively quadratic jam on cue'd data — is being reported to iris-rs separately.)

Tests added

• v0 reconstruction unit test (recipient grouping, gift totals, original seeds retained, keying by note sig)
• [raw-tx heard-at] decode for both raw versions
• end-to-end blocks-range → L0 ingestion test: a mock node serves a block whose tx map carries all three wire shapes (raw v0, raw v1, full tx) through the real decode path into update_blocks_impl, asserting the inserted rows' total_size, fee, and jam
• TEST_DB_PATH-gated golden test against a real synced DB: every stored tx is stripped to its raw-tx subtree, reconstructed, and must match the node-provided total_size and jam byte-for-byte (evenly-strided sample in debug builds; full sweep in release)

Validation — golden test against the published sample-db snapshot (SHA-verified, integrity-checked): sampled run of 516 txs spanning heights 5,629–112,500 passes for both versions; the full 74,742-tx release sweep is running and results will be reported here.

[h33p]

@thomas-kroes the robot ignored the request to turn ScryTx into pub struct ScryTx(Tx). It added more complicated slop instead.

[thomas-kroes]

@h33p Restructured in 7d4af00. One correction to the literal suggestion: the RawTx -> Tx conversion can't live in NounDecode — rebuilding the outputs needs the block height (it becomes the notes' origin-page; +new:tx takes page-number) and the activation settings, and from_noun only sees the tx noun. So the conversion sits one level up in scry_blocks, which knows the height per block and now holds ChainActivations.

Net effect is what you asked for, minus the newtype: ScryBlocksResult carries plain ZMap<Digest, Tx> again, the L0 ingest loop is back to untouched, and nothing outside block_range_manager sees ScryTx or into_tx. The workaround note with the 5decf7cb commit link is on the decoder.

Also checked #150 after the merge: it restores heaviest-chain-blocks-range-no-pow with the same path and raw-tx response shape, so --block-range-scry-no-pow=true keeps working against current nodes with no further changes here.

Validation after the refactor: full test suite passes (35, including the raw-range → L0 ingestion test), and the sampled golden run against the published snapshot passes (516 of 74,742 stored txs, evenly strided across heights 5,629–112,500, reconstructed byte-for-byte). The full release sweep the previous comment promised got interrupted; re-running now, result to follow below.

[thomas-kroes]

Full release sweep results, as promised: 74,742 / 74,742 stored txs pass (1,788s), commit 7cf3379.

One genuine finding came out of it. Eight v1 txs (heights 78,026–91,508) carry a stored total_size that is 100–1,511 bits larger than the canonical fully-deduplicated jam bit length of their raw-tx. For all eight, patching the stored size into the reconstruction makes the freshly-cue'd jams byte-identical — i.e. the raw + outputs rebuild is byte-exact for every tx in the snapshot; only that one consensus number differs.

Explanation: consensus total-size is (met 0 (jam raw-tx)) as the validating node's runtime computed it, and for these eight the runtime's jam missed backrefs on structurally equal subtrees (same cache/provenance defect class as the iris-ztd cue issue being reported separately — a missed backref can only lengthen a jam, and all eight deltas are positive). That value depends on the runtime's internal sharing at validation time and is not derivable from the noun, so the reconstruction computes the canonical size — which is also what a current node recomputing +compute-size-jam on a freshly cue'd raw-tx would get.

The golden test now encodes exactly that: canonical size must never exceed the stored one (a systematic undershoot fails at >0.1%; actual is 8/74,742 ≈ 0.01%), and everything else must match byte-for-byte through a fresh cue+jam comparison that is immune to the mixed-provenance jam artifacts.