HyperBEAM Mainnet & Native AO Migration - #186
Open
jim-toth wants to merge 34 commits into
Open
Conversation
Migrate operator-registry off the dead AO legacynet onto self-hosted
HyperBEAM as a native lua@5.3a process. This is the validated pilot for
the D26 native contract shape; the reward contracts follow.
Runtime (runtime/native.lua): infallible compute trampoline (atomic
revert, outbox, runtime-owned ACL roles, Owner set-once, dispatch);
base-addressed state (now/state/<key>); computed views (now/as/<view>).
Migrate-on-spawn seeds both base.state and base.acl.roles from the
module's declared initial state/acl. Carries the A16 strip-on-write fix
(silent write-loss on string-valued state maps grown across slots).
Contract (src/contracts/native/operator-registry.lua): behavior frozen
from the legacynet contract, reshaped to { state, actions, views }.
Addresses now stored as EIP-55 (was 0x+ALLCAPS): on-chain keccak-256 +
EIP-55 validate-and-reject at every untrusted ingress (common/eip55.lua,
proven bit-identical to ethers.getAddress on luerl and the device VM).
common/bigint.lua replaces .bint for luerl-safe 256-bit math; the legacy
relay/staking contracts are updated to match (.common.bigint,
AnyoneUtils.parseInt).
Tests: 3-tier harness (spec/) — Tier-1 busted/Lua-5.3 92/0, Tier-2 luerl
1.3.0 141/0, Tier-3 live v0.9-FINAL node 66/0. Real-seed migrate-on-spawn
validated on-node against the 2026-07-09 dump (~1MB: 7932 verified / 2940
claimable / 1088 hardware): full seed-diff byte-identical, roles seeded,
EIP-55 lookups + post-seed writes hold at scale.
Tooling (scripts/): native HB client (hb-client.ts), by-id module
publish, seed builder (build-seed.ts), tier3 validators + A-series repro
probes, and the d6-conformance harness.
Second contract of the migration ported to the D26 native shape, with the frozen reward math proven byte-identical to legacynet all the way to Tier-3 on a live node. Contract (ao/src/contracts/native/relay-rewards.lua): - Complete-Round math copied verbatim from legacynet; only the wrapper reshaped (Handlers -> native actions, RelayRewards.X -> ctx.state.X, msg -> ctx). - EIP-55 at every untrusted address ingress; PreviousRound.Details off-persist (served from the Complete-Round output, never re-serialized into base.state) -> persistent state drops from ~4MB to ~740KB. - Staged Add-Scores + Complete-Round kept verbatim; delegate feature retained. Two on-device bugs the spike caught (invisible to Tier-1/2), both fixed: - A17 (luerl): a large positive-int table key (13-digit ms round timestamp) makes the device VM array-allocate and hang. PendingRounds is now keyed by tostring(timestamp). The legacy contract has this latent too. - A18 (runtime/native.lua): action handlers were handed metadata-laden base.state; a handler iterating a persisted map hit HB ao-types/commitments as data. Strip at compute start, symmetric with the view read path. Verification (all pass): - Tier-2 golden 12/0 (reproduces the exact bint golden under luerl). - Tier-3 on-node: 719KB real seed materializes byte-identical; a 300-fingerprint realistic round onto migrated balances -> 300/300 rewards + accumulation. - Legacy cross-check (tier2-relay-legacy-crosscheck.ts): the same realistic round through both the legacy Handlers contract and the native port yields 300/300 Details + Summary + TotalFingerprintReward byte-identical across every branch (hardware pool split, exit distribution, multi-fp proportional division, delegate split). Built-in negative control confirms the diff has teeth. - Durability measured: verdict GO for our sizes (moderate staged batches).
Per-behavior regression net (38 tests) on the native runtime under Lua 5.3, mirroring the legacynet WASM harness (test/spec/contracts/relay-rewards/*), re-expressed native: state read from base.state, a write's reply is the compute output, reads are native.view, an assert failure surfaces as output 'error: ...' and reverts atomically. Covers: ACL gating per action (owner/admin/named-role/denied), Update-Configuration + Modifiers/Multipliers/Delegates validation, Add-Scores per-field validation and validate-before-mutate atomicity, the A17 string-keyed PendingRounds, Set-Delegate set/clear, Cancel-Round, Complete-Round lifecycle (old-round pruning, summary persisted / Details off-persist on the output), reward RELATIONSHIPS (ratings, period, accumulation by fingerprint + EIP-55 address, delegate split identity), Claim-Rewards, all six views, and the D8 runtime-safety axes (unsigned/unknown action/atomic revert). Reward MAGNITUDES stay a Tier-2 concern by design: common/bigint is a native-integer wrapper, exact only where ints are arbitrary-precision (luerl/device); real Lua 5.3 is 64-bit and overflows at token scale. Tier-1 shrinks TokensPerSecond to assert relationships; the exact bint golden is reproduced at Tier-2 (spec/luerl/scenarios/native-relay-rewards.lua) and cross-checked vs legacy.
Recreate every legacynet WASM+TS harness case (test/spec/contracts/relay-rewards/*, 78 it() across 15 files) against the native port, split across tiers. Tier-1 busted (spec/native/relay-rewards_spec.lua): grown to 79 tests covering all validation (add-scores per-field, configuration, modifiers, multipliers, delegates), ACL (admin + named role per gated write), round lifecycle (cancel/complete pruning + last-round metadata), and the reward MAGNITUDE cases recreated at the harness's small TokensPerSecond so the exact expected integers (ratings 110/70, uptime 100/700/675, hardware 65-35 split 186/373, exit 660, delegate 2240/3360, accumulation 558/930/3631/3259/5808 -> 3918/6530/5871, claim tracking) hold in 64-bit. Init import/reimport -> migrate-on-spawn seeded-base view round-trips. Full spec/native/ = 171/171 (op-registry 92 + relay 79). Tier-2 luerl (scripts/tier2-relay-datasets.ts): the 3 data-driven cases run the real captured datasets at real token scale through the native contract under luerl (arbitrary-precision) -- staging1 330fp / staging2 353fp / test1 420fp, all process to OK with full Details. Reward MAGNITUDES stay a Tier-2 concern by design (common/bigint overflows 64-bit at real token scale); the exact bint golden (12/0) and the legacy cross-check (300/300) already cover full scale. Two intentional deviations, asserted as such: Set-Delegate trusts the node-verified committer (ctx.from) as operator, not re-validated in-contract; no Init action (state import is migrate-on-spawn). Minor luerl finding noted: its parser rejects 4-equals long brackets ([====[); use [==[.
Third and final contract ported to the D26 native shape, with full parity against the legacynet WASM harness (108 it() across 13 files). Runtime — ctx.timestamp (ms): Contracts that record message time (staking Set-Share) need a native clock. The scheduler stamps the assignment with two DIFFERENT units: `timestamp` = erlang:system_time(millisecond), and `block-timestamp` = Arweave block time in unix SECONDS (literally 0 when the node runs mode: debug). hyper-aos maps os.time to the latter. We take `timestamp` (ms), matching legacynet msg.Timestamp exactly, and document the trap at the source. Verified on a live node with scripts/probe/timestamp-probe.lua: timestamp=1784989059945 (inside the host's own Date.now() window), block-timestamp=0. This resolves the open "os.time — verify" item in D8; the answer was that it is NOT fine. Contract (src/contracts/native/staking-rewards.lua): - Frozen reward math copied verbatim; only the wrapper reshaped. - Two-level maps throughout (Rewarded/Claimed/Details[hodler][operator]); the operator's own cut lives at the self-key Rewarded[operator][operator]. - Details ARE persisted here (state is ~322KB total vs relay's 4MB), so Last-Snapshot/Last-Round-Data stay plain views. - A17: PendingRounds keyed by the string timestamp. - EIP-55 at every untrusted ingress, wrapped so legacynet error strings survive. Share-change delay unit fix (deliberate, not a math change): legacynet compared `RequestedTimestamp + ChangeDelaySeconds <= roundTimestamp` where both timestamps are MILLISECONDS but the delay is denominated in SECONDS, so the 7-day (604800s) default elapsed in 604800ms ~= 10 minutes -- with hourly rounds, at the very next round. Fixed by converting the delay to ms. Safe to fix in place: the feature is dormant (live config SetSharesEnabled=false, Shares and PendingShareChanges both empty, no Set-Share in the live message tail, and no Set-Share caller in any consumer repo). Tests: - Tier-1 busted spec/native/staking-rewards_spec.lua: 114 tests, all 108 WASM cases recreated (the suite's largest asserted value is 7.5e13, inside 64-bit, so unlike relay nothing needs Tier-2 for magnitude) + D8 safety axes. Full spec/native/ = 285/285, stable over 10 consecutive runs. - Tier-2 luerl spec/luerl/scenarios/native-staking-rewards.lua: 29/29 on the real device VM using REALISTIC 13-digit ms timestamps, which is what exercises A17. - New realistic-millisecond delay cases at both tiers. The legacynet harness only ever used toy integers (request@1000, round@2000, delay 1000) which pass under BOTH the buggy and fixed arithmetic -- that is precisely how the unit bug survived. The new cases fail on the old arithmetic and pin the fix. Also fixes a latent flaky assertion in both reward specs: comparing json.encode(map) strings is order-dependent because Lua pairs() order is not stable; compare deep copies structurally instead.
Units were only half the question. Traced where req.timestamp actually comes
from: it is the SCHEDULER'S LOCAL WALL CLOCK -- HyperBEAM's own comment reads
"Note: Local time on the SU, not Arweave" (dev_scheduler_server.erl:231). It is
not the data item's timestamp (ANS-104 items carry none, so a sender cannot set
or forge it) and not Arweave block time.
Guaranteed: committed into the signed assignment, so it is fixed at assignment
time and identical for every CU replaying the schedule -- execution stays
deterministic.
NOT guaranteed: monotonicity. erlang:system_time follows the OS clock and an NTP
correction can step it backwards, and the scheduler performs no ordering
validation on it. So message timestamps do not reliably order messages.
This matters because staking's share-change delay gates RequestedTimestamp (SU
wall clock) against Round-Timestamp (the controller's clock) -- two independent
clocks, so host skew shifts the effective delay. That is inherited from legacynet,
which compared msg.Timestamp against Round-Timestamp the same way, and is kept
deliberately to keep the port faithful. Round-Timestamp is by contrast
contract-enforced monotonic via the backdating assert, and the alternative design
(start the delay at the first Complete-Round after the request and measure purely
in round time) is documented for revisit before SetSharesEnabled is ever turned on.
Documented at every point of use: runtime/native.lua where ctx.timestamp is
derived, the staking contract header ("ACCEPTED LIMITATION") and its Set-Share
site, and the D8 port-safety checklist.
No behaviour change: comments only. Tier-1 285/285, staking Tier-2 29/29.
The staking counterpart of tier2-relay-legacy-crosscheck.ts, closing the same
weak link. The native reward math is a verbatim frozen copy of legacynet
Complete-Round, so the residual risk is a transcription typo -- and the busted
and luerl specs cannot see it, because they assert the port against itself.
This drives the same round through BOTH contracts under luerl (legacy in `run`
mode against the StakingRewards global, native in `native` mode against
base.state) and diffs the parsed results.
Inputs are real, from the 2026-07-09 live dump: the deployed Configuration, 300
(hodler, operator) pairs drawn from Rewarded, and the full 562-hodler Rewarded
and 400-hodler Claimed maps as priors -- so the cumulative bigint add lands on
migrated balances rather than zeros, and every restaked branch fires from real
data (Rewarded-Claimed, Rewarded alone, and no prior at all).
Three runs:
A live configuration verbatim. 373 pairs -- 92 self-pairs (the operator
own-cut key), 92 multi-operator hodlers, 215 with a real Claimed prior, 73
fresh, 34 below the Running gate and 34 exactly at it.
373/373 pair Details identical, Summary/Period/Rewarded/Claimed identical,
309 hodlers accumulated off their seeded prior.
B SetSharesEnabled with per-operator Shares {0, 0.1, 0.5, 1.0}. The live
config has share-setting disabled, so run A cannot reach this branch at
all. 373/373 identical; shares 0, 0.05 (the Default fallback), 0.1, 0.5 and
1.0 all exercised in-round, including both edges (139 pairs at a 100%
operator cut, 65 at 0%).
C the one INTENDED deviation, asserted rather than assumed: legacy compares
ms + SECONDS against ms, so a configured delay elapses ~1000x early; the
port converts to ms. Legacy applies a 1-hour delay after 60 seconds of
round time, native holds. C fails in BOTH directions -- if native applies
early the fix has regressed, and if legacy holds the fixture has stopped
reproducing the bug and proves nothing.
Negative control (PERTURB=1, native TokensPerSecond +1 wei) catches 114
mismatches across Details, Summary and Rewarded, so the diff has teeth.
Address encoding is the other deliberate deviation (legacy 0x+ALLCAPS, native
EIP-55): priors are seeded to each side in its own encoding and keys are
lowercased before diffing, so the math is compared apples-to-apples. Encoding
fidelity is covered separately by validate-address-migration.ts.
Timestamps stay small (roundLength 60) to keep the legacy contract clear of its
own A17 large-int-table-key hang; roundLength is a scalar multiplier and changes
no branch, and intermediates still exceed 64-bit by ~20 orders of magnitude.
Run C picks a 1-hour delay for the same reason -- the live 7-day value would
need 13-digit ms timestamps to straddle the two readings. The realistic-ms cases
live in the native Tier-1/Tier-2 specs, where no legacy contract is present to
hang.
Compares parsed structures, never json bytes: Lua pairs() order is not stable.
Closes the last gap for the third contract: the native staking-rewards port now
materializes the real 322KB legacynet state on a live HyperBEAM node and settles
a realistic round byte-identically against an independent luerl oracle.
build-staking-seed.ts -- live dump to native seed (migrate-on-spawn). EIP-55 at
BOTH levels of the two-level maps (Rewarded/Claimed/PreviousRound.Details) and
in Shares/PendingShareChanges; 3511 addresses converted, none rejected. Unlike
relay, PreviousRound.Details is KEPT: staking's whole state is 322KB, so there
is no size pressure and seeding the final legacynet round preserves
Last-Snapshot/Last-Round-Data across the migration. The transform asserts it is
total -- a dropped key would silently shrink the migration, so hodler and pair
counts must match the dump exactly on the way out.
build-staking-oracle.ts -- the parity oracle. Relay needed a minimal config-only
oracle because its 800KB bundle times out in luerl; the staking bundle is 402KB
and parses in under 2s, so the oracle runs the SAME full seed, with the SAME
562-hodler Rewarded and 400-hodler Claimed priors, that the node spawns from.
The comparison is whole-state rather than "fresh addresses accrue the same". It
also refuses to emit a degenerate round (wrong Period, zero Rewards, or a Details
pair count that does not match the input).
util/staking-round.ts -- the single definition of the round, imported by both the
oracle and the node driver, with a timestamp equality guard so the two provably
cannot drift. 250 real pairs drawn from the seed plus 25 fresh ones, so restaked
is computed against real migrated balances (182 pairs with a Claimed prior ->
Rewarded-Claimed, the rest -> Rewarded alone, fresh -> 0), 74 self-pairs, and 56
pairs at or below the Running gate. Timestamps are realistic 13-digit ms
(seeded PreviousRound.Timestamp + 1h): a large integer round timestamp is what
hung the device VM under A17, so Tier-3 must use one.
tier3-staking-validate.ts -- ALL PASS on hb-tier3:
* seed materializes: 562/400 hodler counts, config, roles, PreviousRound
* full seed-diff pair-by-pair -- Rewarded 865 pairs, Claimed 594, and the
persisted Details 636, all identical
* the legacynet read surface answers off migrated state: last_round,
last_snapshot (carrying all 636 Details pairs), last_round_data, rewards,
claimed, plus a base-addressed point read
* the round settles identical to the oracle: Period, Summary, all 275
per-pair Score/Rating/Reward, and the whole 587-hodler/915-pair cumulative
Rewarded map; Claimed untouched; PendingRounds cleared; 215 hodlers moved
off their migrated priors, and Rewarded grew by exactly the 25 fresh
operators' own-cut self-keys
* Add-Scores 7.1s / Complete-Round 4.0s for a 33KB payload
PERTURB=1 shifts the oracle by one wei and is caught on all three surfaces, so
the whole-map comparison is not a no-op.
Comparisons are structural, never JSON strings. The node re-encodes state from
Lua tables and pairs() order is not stable, so byte-different encodings
routinely carry identical data -- the first run of this script reported four
false failures for exactly that reason. Same trap as the reward specs and the
relay cross-check; noted at the helper so it does not return a fourth time.
Tier-1 spec/native/ still 285/285.
scripts/spawn.ts was written for the genesis-wasm era and carried exactly the defaults that caused this migration: HB_URL fell back to https://push.forward.computer (FR-operated), MODULE to a hardcoded genesis-wasm id, and DEPLOYER_PRIVATE_KEY to a hardhat test key. It spawned, slept 5s, then Eval'd the contract source as aos. Moved to scripts/legacy/ and kept runnable as `deploy:legacy-genesis-wasm`; `npm run deploy` is now the native path. scripts/deploy.ts has no fallbacks anywhere. HB_URL, DEPLOYER_PRIVATE_KEY, MODULE_ID and --seed are each required, and --seed has no default because "which state does this process start from" is not a question to guess at. It also refuses outright to deploy against a forward.computer / ao-testnet / arweave.dev host rather than trusting whatever HB_URL says -- the no-FR-infra constraint checked in code, not just written down. Two phases, because they need different credentials. --publish-cmd emits the node-host command that commits the module signed by the NODE wallet: that writes it into the node's local cache so the spawn resolves immediately, rather than waiting on Arweave gateway propagation as a deployer-wallet Turbo upload would. The spawn phase then runs anywhere, signed by the deployer wallet that must be in the node's faff allow-list, with scheduler-location and authority explicit (H1/H2/H3 from the blast-radius audit are all "a default nobody set"). Seeds come from the existing build-{,relay-,staking-}seed.ts builders, invoked rather than reimplemented -- they are what the Tier-3 validations ran against and they assert their own totality. The deploy VERIFIES rather than trusting a 200. Migrate-on-spawn happens on first compute, so a broken seed cannot surface any earlier: it polls the status view and compares the materialized counts against what the seed builder emitted. Writing that caught a gap -- each contract's status view names its counts differently (op-registry claimable/verified/blocked vs the reward contracts' addresses/fingerprints/rewardedHodlers), and the first version would have run ZERO checks on a seeded op-registry deploy and reported success, since undefined === undefined passes. The mapping is now explicit per contract, and a deploy that can verify no count at all fails rather than passing quietly. Verified end-to-end on hb-tier3, all three contracts from the real live dump: operator-registry claimable 2940, verified 7932, hardware 1088, blocked 0, credits 0 -- all 5 counts matched relay-rewards addresses 763, fingerprints 9750, claimed 530 staking-rewards rewarded 562, claimed 400 Owner came back as the deployer wallet in every case. Guards confirmed firing: missing HB_URL, an FR host, and a missing --seed each abort with an actionable line rather than a stack trace. Transport is @anyone-protocol/ao-client (git dep, v0.1.1), its first real consumer -- and the only one that exercises spawnProcess.
scripts/verify-migration.ts checks the chain from manifest hash to live state: dump files still hash to the recorded sha256, the seed transform is deterministic, on-node state deep-equals the transform output, and roles are restored. D12 asks for a hash of post-init state against the manifest. That predates the EIP-55 decision and is unsatisfiable now -- native state is a deliberate transform of the dump, so the bytes cannot match. Checking every link instead; rationale is in the script header. D13: "no replay required" was a human determination in a doc. Now machine- checked -- every tail message must predate the dump, and none may have failed except known-benign ones. Also wires D10's negative fixtures: the staking tail's three "No rewards for 0xFB5BA08D..." failures, replayed against the live process, must still fail closed. All three processes, 49 checks, ALL PASS on hb-tier3.
The score generator varies hardware/exit/uptime-tier, but only the delegate
branch was ever asserted to have fired. "Both sides agree" stays true even if a
branch stops firing entirely -- a config change (Modifiers.Hardware.Enabled=
false), a different tier table, or an edit to the generator would silently shrink
what the test proves while still reporting 300/300.
Same trap that made the staking share run look green while using only half its
share values.
Now reports and requires: hardware 100, exit 60, uptime 300, tiers {0,3,14,45}.
Verified it fails when a branch cannot fire (K=3 yields 3 of 4 tiers -> exit 1
with zero math mismatches).
Reads back the policy the node actually loaded rather than reviewing the jobspec, probes the nginx edge from outside, and asserts faff denies a non-allow-listed signer. 74/74 across all three; PERTURB=1 self-test.
Closes D3's rate-limiting/body-cap clause. --dos is opt-in and dev-only by default since it generates real load. Asserts the probe actually exceeded 30 req/s first: an earlier form reached only ~17 req/s against a large payload and would have 'passed' without exercising the limiter.
An arbitrary (non-allow-listed) wallet is admitted past faff for the whitelisted ids and refused for a foreign one — by faff on dev, at the edge on stage/live. 80/80.
Legacy asserted 'timestamp is integer'; the native specs only fed non-numeric values, so parseInt's integer-only strictness (the A12 workaround) was untested. Relay also lacked the missing-tag and empty cases staking had. Verified to fail when fed a valid integer.
Tier-2 invocations are non-uniform (three runner modes, different arities) and scenarios/ also holds a Tier-3 oracle that is not a test and spins for hours under A17 if run wrong. Encode all of it in spec/run-tier2.sh rather than leaving it tribal; a scenario producing no result line is an error, never a pass. ao-test.yml runs Tier-1 and Tier-2 as separate jobs so a failure names the layer. The legacy mocha suite runs non-blocking: it is a coverage reference for the native specs, not a gate.
Fork PRs execute contributed code (two image builds + a bun install that runs their lifecycle scripts). The read-only token and absent secrets mean that is not a credential risk, but it is free compute now and RCE if this ever moves to a self-hosted runner — so gate every job on the PR originating from this repo. Also state permissions explicitly rather than inheriting them, and pin bun instead of tracking latest.
The aos-WASM harness is obsolete under the Lua device and the legacynet contracts now carry luerl workarounds their specs never got. Keep the specs in the repo as a coverage reference; stop executing them.
The E2E assertions already existed across the tier3-* verticals; what was
missing was a way to run them as a suite. Each vertical needed a different
module id from a manual publish, in an order written down nowhere — the same
problem run-tier2.sh solved for Tier-2, with the same failure mode of a
mis-invoked or skipped stage reading as success.
scripts/run-e2e.ts: preflight -> artifacts -> module registration -> the four
verticals -> verify-migration x3 -> aggregate. Skipped stages are reported as
skipped rather than counted, failing stages dump full output to dist/e2e-logs/,
and a vertical that exits 0 without spawning is an ERROR, not a pass.
19/19 green from a clean tree against a local v0.9-FINAL node.
scripts/build-relay-probe.ts: dist/relay-oracle-probe.json was read by
tier3-relay-validate.ts and written by nothing, so a fresh checkout could not
run the relay vertical at all. Staking's equivalent already self-automated.
Guards that the oracle actually computed its three fingerprints with non-zero
rewards — otherwise the parity check passes vacuously against a node that also
did nothing.
hb-client.ts: correct the moduleIdFor comment. It claimed an inline spawn
leaves the module child in the node's cache so a later by-id spawn resolves it.
It does not, and it is expensive to believe: the by-id spawn is ACCEPTED and
only fails at compute time with {case_clause,{error,not_found}}. Verified
refuted on v0.9-FINAL at both ~200B and 806KB, including after forcing a
compute on the inline process first. A spawnable module must be a signed ans104
module message written to the node cache via bin/hb eval.
verify-migration gets a FRESHLY spawned process rather than its vertical's:
the verticals mutate state by design (relay and staking drive a full round),
so reusing one fails looking like a migration defect — extra reward keys, a
moved PreviousRound.Timestamp — which is the kind of red that gets "fixed" by
weakening an assertion.
CI: an e2e job on push + workflow_dispatch, not pull_request. It needs
bun install, which re-introduces the lifecycle-script surface that dropping
legacy mocha removed from the PR path, and Tier-1/Tier-2 already gate contract
logic there.
The stage-node run stays re-homed to D14.
publish-native-module.ts's bin/hb eval flow cannot target a self-hosted bundler: `bin/hb eval` starts a separate VM that never boots the hb application, so it loads NO config and always resolves bundler-ans104 to the compiled-in default. Verified — with config set to http://127.0.0.1:9999 an eval still reported <<"https://up.arweave.net:443">> and uploaded there. So publishing moves client-side over HTTP: sign the ans104 item here, POST to <bundler>/~bundler@1.0/tx. The bundler becomes a URL rather than a deployment assumption, and the same script works against up.arweave.net or our own node. The verification is the point. A bundler answers 200 with a signed receipt as soon as it has QUEUED an item, and settlement takes hours — so a receipt proves nothing, and the gateway DATA endpoint is actively misleading (it serves 200 from the optimistic cache before settlement). Only the GraphQL index distinguishes queued from persisted, which is also exactly what node-side resolution needs: hb_store_gateway -> hb_gateway_client:read/2 goes via /graphql. --verify-spawn additionally proves a cold node can resolve the id, which settlement alone does not establish. Three ids are in play and confusing them turns a successful publish into an apparent silent failure: - signed ans104 item id = what is uploaded, indexed, and spawnable - hb_util:id(Msg) = local-cache id, NEVER uploaded - bundle tx id = the L1 tx carrying the item, not the module This script tracks the first and says so in its output. BUNDLER has no default: publishing to up.arweave.net should be a deliberate choice, not the consequence of an unset variable.
… probe
Settlement takes hours, so blocking on it is the wrong workflow. --manifest
records {file, id, bytes, publishedAt} and --recheck re-reads it later,
reporting largest-settled vs smallest-pending so a free-tier boundary shows up
as the size at which settlement stops. The manifest is written BEFORE the wait:
if the wait is interrupted the ids must still be recoverable, or the upload is
unverifiable and effectively lost. --wait 0 publishes without waiting at all.
build-size-ladder.ts emits inert, self-describing lua probes at 100KB..10MB.
They are permanent if they settle, so each carries a header saying it is a
capacity probe and not a contract.
Already answered by publishing the ladder: up.arweave.net caps requests at
EXACTLY 5 MiB (5,242,880 accepted, 5,242,881 -> 413, nginx client_max_body_size
5M), measured across the whole ans104 item. That is a TRANSPORT limit, not a
billing one. Bisected with deliberately invalid payloads so nothing was
persisted to find it: over the cap nginx 413s on size before parsing, under it
the bundler rejects the malformed item.
Our largest module (operator-registry-seed, 1,077,072 B) is at 20% of the cap.
Whether items settle for FREE at these sizes is still open and needs the
recheck; the ids are recorded outside dist/ because dist/ is gitignored.
GraphQL indexing only means a gateway saw the item. It does not mean anything was mined, and the gateway data endpoint returns 200 from the bundler's optimistic cache the whole time, so neither signal distinguishes queued from persisted. ANS-104 data items are L2: they live inside a bundle and L1 knows nothing about them, so /tx/<dataItemId>/status returning Not Found is CORRECT and says nothing either way. Bundles also nest -- bundledIn can point at an intermediate bundle that is itself a data item. Only the ROOT of that chain is an L1 transaction, and only its confirmation count is evidence of mining. finality() walks bundledIn to the root and reports the root's L1 confirmations. Validated against two references: legacynet item -> depth 2, root oYcUSTh3..., 151,436 confirmations our 66KB module -> depth 1, root O9K653PJ..., 96 confirmations Caveat kept in the code: a confirmed L1 tx carries a data_root, but chunks are uploaded separately (dev_bundler's third phase, post_proof). A tx can be mined while its chunks were never seeded, so confirmations plus repeated successful retrieval over days is the real bar -- a single check right after publishing proves neither. Hence --recheck against a manifest rather than a one-shot wait. Baseline at 0.2h after publishing: 0/8 indexed, all showing the misleading data-200.
The publish phase had a worse problem than publishing to the wrong place.
MODULE_ID came from hb_util:id(Msg) after a node-host hb_cache:write — a
NODE-LOCAL id. The id Arweave indexes is the signed ans104 item id, a different
value. So a process spawned this way had a module living in exactly one node's
cache: a rebuilt node, a fresh alloc, or the second node of the redundancy pair
could never resolve it, and the process could never compute another slot.
Nothing about the process reveals this; it surfaces at cold start, which is
precisely what D5 is meant to guarantee against.
The eval also ran hb_client:upload "for durability". It did not: `bin/hb eval`
starts a VM that never boots the hb application, so it loads no config and
always resolves bundler-ans104 to the compiled-in default regardless of what
the node is configured with (verified 2026-07-27). Removed rather than left as
a false comfort.
Now:
--publish delegates to publish-module.ts — signed client-side, posted to
BUNDLER, settlement verified. One implementation of publishing
rather than a second copy that can drift. BUNDLER has no
default.
--publish-cmd still available for fast local iteration, but prints the
node-local caveat instead of implying durability.
And the spawn phase now refuses to deploy against a module that is not indexed
on Arweave, since that is the last point where the mistake is cheap to fix.
--allow-unpublished-module opts out for throwaway test deploys.
Verified: node-local id refused with actionable guidance; settled id passes
("module durability on Arweave (indexed)"); escape hatch warns and proceeds.
A long-lived branch with an open PR fires both `push` and `pull_request` for the same commit, so tier1 and tier2 were running twice on every push to dev-hyper-aos-migration. Deduped with a concurrency group keyed on the commit rather than the event: github.sha on a push and pull_request.head.sha on a PR are the same commit, so both runs land in one group and the second cancels the first. That dedup only works if every job behaves the same for both event types, so e2e loses its blanket `!= 'pull_request'` and takes the same fork guard as the tiers. Otherwise coverage would depend on which run won the race — if the PR run survived, e2e would silently never run, which is worse than running twice. Dropping the blanket exclusion is also the better policy on its own. The concern was that `bun install` re-introduces a contributor-controlled lifecycle-script surface, but that is a FORK problem and the fork guard addresses it precisely; excluding all PRs additionally meant same-repo PRs got no integration coverage at all.
Simpler than the concurrency-by-commit trick in 899c960, and Jim's call. Listing the working branch under `push` meant every commit fired both `push` and `pull_request` once a PR was open. Deduping that let both runs start and then cancelled one -- burning runner minutes and leaving cancelled runs in the UI that read like failures. Not firing the second event at all is strictly better. Branch coverage now comes from the PR, which is where the result is needed anyway. Keeping e2e's fork guard from 899c960, which this makes necessary rather than merely tidier: with `push` main-only, a PR is the only event a working branch fires, so the old blanket `!= 'pull_request'` would have meant e2e never ran until merge. Concurrency stays, retargeted to its actual job: superseding an in-flight run when new commits land on the same PR. It is no longer what prevents the double-run.
The e2e job failed on run 30292049064 while tier1/tier2 passed. Root cause was
a missing step, not a test regression:
ERROR relay parity oracle — Unable to find image 'anyone-luerl:1.3.0' locally
pull access denied for anyone-luerl, repository does not exist
Both oracle builders shell out to anyone-luerl:1.3.0 to run the frozen round
under luerl and produce the parity expectations. Tier-2 builds that image, but
jobs get separate runners with separate docker daemons, and the image is
local-only so it cannot be pulled. One missing build step produced four
reported failures: two oracle stages, then both parity verticals ENOENT-ing on
the probe JSON they consume.
Also make that cascade legible. A vertical whose prerequisite artifact is
missing now reports "prerequisite artifact missing: <file> — see the artifact
stage above" instead of an ENOENT stack trace from deep inside the vertical,
which read like a new problem rather than the downstream consequence of a
failure already reported one stage earlier.
Verified by reproducing the CI condition locally (LUERL_IMAGE pointed at a
nonexistent image): the oracle stage errors, the vertical names the missing
artifact, and with the image present the slice is green again.
Worth noting the suite behaved correctly throughout -- it failed loudly and
exited 1 rather than reporting a pass, and stage 4 still passed because
verify-migration spawns fresh rather than reusing a vertical's process.
The e2e job failed on `staking parity oracle`: build-staking-oracle.ts hardcoded `podman` and the podman-only `:Z` mount flag, while run-e2e.ts was passing it CONTAINER_ENGINE=docker. GitHub's runners ship podman too, so this did not fail with "not found" — podman tried to PULL anyone-luerl:1.3.0 from docker.io and got `requested access to the resource is denied`. That image is local-only (the workflow builds it into docker), so the pull could never succeed, and the error named a registry rather than the engine mismatch. Adds scripts/util/luerl.ts as the single place that knows how to invoke the runner image, and routes all six callers through it. Five of them were carrying the identical hardcoding; only build-relay-probe.ts read the env, which is why relay's oracle passed in the same run that staking's failed. The prerequisite-artifact check added in d6f88ff worked as intended here: the staking vertical reported "prerequisite artifact missing: dist/staking-oracle-probe.json — see the artifact stage above" instead of an opaque ENOENT. Verified: full suite 19/19 locally under podman, and CONTAINER_ENGINE=echo confirms the engine is honoured and `:Z` dropped for non-podman engines.
Published modules are now PURE SOURCE. The seed travels as spawn-message data
and the runtime consumes it at slot 0 (runtime/native.lua, where
`base.state == nil`) and never again.
The embedded-seed module was costly in a way that is not visible from the
contract: the module is reloaded into a fresh luerl VM on EVERY READ, while the
declared initial `state` is consumed exactly once. So a seeded module re-ran
json.decode over the entire dump for every read and threw the result away.
operator-registry-seed.lua was 1,076,898 B, 93.7% of it embedded JSON.
Measured on a node holding the real migrated registry:
embedded seed at spawn
published module 1,076,898 B 68,177 B
status view 2.60 s 0.43 s
dump view 2.94 s 0.67 s
State is byte-identical between the two, verified field by field including the
ACL roles. Isolating further: keeping the 1 MB literal in the source but NOT
decoding it reads in 0.17 s, so lexing that long bracket is ~140 ms and the
json.decode is ~2.44 s.
Second, less obvious win: the module stops being a per-migration artifact. One
published module per contract now serves dev/stage/live and every reseed. The
E2E demonstrates it — the same module id backs both the seeded and the unseeded
operator-registry vertical, and registration dedupes by file so identical source
cannot mint two ids.
Envelope: { "ao-migration-seed": 1, state, acl: { roles } }. Marker-gated so
ordinary spawn data is never mistaken for a seed. A marked-but-malformed seed
yields an EMPTY state rather than a half-initialized one, which deploy tooling
catches immediately by diffing against <contract>-seed.expected.json.
buildSeedBundle is RETAINED, deliberately: the Tier-2/Tier-3 luerl fixtures load
a .lua file directly under the runner and have no spawn message to carry a seed.
There the bundle loads once per invocation, so the per-read cost does not apply.
Needs @anyone-protocol/ao-client v0.1.2 (adds `data` to SpawnOptions); the pin
bump is not in this commit because the tag is not pushed yet.
Green: Tier-1 298/0, Tier-2 205/0, E2E 21/21 from a clean tree. The suite is also
markedly faster now every read is — the operator-registry vertical went 40.6s to
13.4s, verify-migration staking 21.8s to 14.8s.
OPEN (deferred, settle before D14): durability. The module on Arweave used to be
enough to rebuild a process on a cold node; the migration state now lives in the
spawn message, so that must be retrievable too.
Picks up `data` on SpawnOptions, which 146d03d needs to carry the migration seed in the spawn message. That commit was validated against a locally-built copy of the package; this pins the published tag and re-runs the suite against it. E2E 13/13 green (8 artifact stages skipped via --keep-artifacts).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.