feat: in-depth defensive/validation improvements #1048
Open
charmful0x wants to merge 19 commits into
Open
Conversation
Caching an L1 transaction header (every mode) and caching a fully-parsed bundle item (full mode) used the same scope-convert-write sequence with different codecs. Share it as `cache_item/3'; behaviour is unchanged (the header path keeps its log-and-skip guard, the full-mode path still asserts the write).
…to lib_scheduler `~scheduler@1.0' and the `~arweave-scheduler@1.0' device added later in this branch present the same consumer interface, so everything about interpreting a scheduling request -- and storing and serving its results -- is identical regardless of how the schedule is sourced. Move that shared logic into a new `lib_scheduler' device library: - `find_target_id/4,3' and `find_message_to_schedule/3' (request parsing), `load_message_to_schedule/3' (find + fully load, 404 on failure), and `only_committed/2' (reduce to committed components, 400 on failure). - `parse_slot_range/2' (the `from'/`to' slot parse; `from' is now clamped non-negative after coercion, which also catches a negative binary input) and `read_assignment_range/5' (bounded range read + truncation flag), with the single `?MAX_ASSIGNMENT_QUERY_LEN' definition and `max_assignment_query_len/0' for callers that bound remote requests by it. - The assignment cache: `cache_opts/1', `write_assignment/3' and `read_assignment/4', parameterized by each device's pseudo-path prefix. The read restores unsigned commitment IDs and converts legacy `ao.TN.1' assignments, exactly as `dev_scheduler_cache:read/3' did. - `base_assignment/4', the assignment keys every scheduler shares -- `dev_scheduler_server:do_assign' merges its block fields and commits; other schedulers merge their own position fields. - `at_slot/2', `slot_unavailable/0' and `format_opts/1' (the no-cache/no-await formatting options, previously inlined in several places). `dev_scheduler', `dev_scheduler_cache', `dev_scheduler_formats' and `dev_scheduler_server' delegate to the library; no behavioural change.
`dev_arweave' and the `~arweave-scheduler@1.0' device both broadcast a request to several Arweave nodes and take the most successful reply. Move `dev_arweave''s status-sorting `best_response/1' (with its `response_status/1' helper) into `lib_arweave_common', which both devices already depend on, and drop the local copy. No behavioural change for `dev_arweave'.
A transaction node returned from the local ~query@1.0 GraphQL endpoint resolved `block' to null, because a cached tx carries no block metadata of its own. Carry each match's weave offset onto its node and resolve `block' by locating the cached block whose byte range contains that offset, distinguished from the top-level block(id/height) query by the offset key on the node. Resolution is local but scans the cached blocks, so callers that only need on-chain ordering should use the offset directly. The cached-heights enumeration and the block byte-bounds arithmetic are shared as `cached_heights/1' and `block_bounds/2' with the module's pre-existing `latest_cached_block/1' and `block_range_to_offset_range/2', which previously inlined the same code separately.
A drop-in ~scheduler@1.0 replacement that sources a process's schedule directly from Arweave layer-1 transactions. Slot 0 is the process message (its tx id is the process id); slot N is the Nth confirmed L1 tx whose `target' is the process id, in ascending weave-offset order. Assignments are deterministic and uncommitted -- Arweave itself is the authority -- so every node converges on identical assignment ids without trusting a scheduler wallet. The on-chain position recorded on each assignment is the weave offset. Discovery is local-first, with no gateway attribution: a single `~copycat@1.0/arweave' shallow pass over the range records each tx's weave offset and caches its (data-free) header, which populates the local `field-target' match; the device then resolves the node's own ~query@1.0 (via hb_ao:resolve, not a remote call) for the recipient set, ordered by that offset index. `arweave_scheduler_query_source => remote' opts into a gateway query instead. The block index is node-shared and `ensure_offsets' passes `reindex=false', so N processes on one node index each block once rather than once per process. The device is header-only end to end. Bodies are read as tx headers, so the schedule never depends on data availability; the header verifies and its committed id is the tx id. POST /schedule likewise relays the header only, rejecting a message that carries data (422) rather than uploading chunks, and rejecting any non-tx@1.0 commitment (422). The misleading tip-derived timestamp/block-height/block-hash that the standard formatter stamps are omitted from a chain-derived schedule. Synchronization tracks a `synced-to' block high-water mark: the attestable range `[spawn-height, synced-to]' the node can vouch for. A read with no new blocks does no network work; otherwise only the delta is indexed, in `arweave_scheduler_sync_chunk'-block passes whose progress is persisted per chunk (resumable first sync). /status reports each tracked process's range. The device is built on `lib_scheduler': request parsing, the assignment cache, the range read, and the base assignment schema are the same code paths ~scheduler@1.0 uses. The full consumer interface (status/next/schedule/slot/init/checkpoint + router) is implemented so ~process@1.0 drives it unchanged. Includes a permanent mainnet test fixture (a lua@5.3a process whose three scheduled messages transform state 1000 -> 1337, one posted directly to arweave.net to prove foreign-tx indexing) and device unit tests.
A process message may now carry `scheduler-mode: all', and its schedule becomes every base-layer transaction on the network rather than only those whose `target' is the process. This is what lets a process observe value moving between two other addresses -- a payment it is owed but is not a party to -- which no recipient-filtered schedule can see. The mode is read from the process message itself at first contact and pinned into the persisted sync state. The process message is an L1 transaction, so the mode is chain data like the rest of the schedule, and every node derives the same one; a node option would have made slot -> body a function of node config, producing two indistinguishable, equally `valid' schedules for one process id. An unrecognized mode falls back to `target' rather than erroring: a spawn tag cannot be corrected once it is on-chain. `all' mode does not query. Dropping `recipients:' from the GraphQL would have silently returned nothing -- `dev_query_arweave' only builds candidate sets from height/id/ids/tags/owners/recipients, and `block:' is a post-filter over a set that must already exist -- so the range's block headers are walked directly instead, taking each block's own `txs' list. That is the same source `~copycat@1.0/arweave' enumerates, and `ensure_offsets' has already cached those blocks, so it is a local read. Nor does it re-sort by weave offset. Offset is not a total order: `TXStartOffset = TXEndOffset - data_size' and the accumulator only advances by `data_size', so every data-free transaction in a block shares the offset of the one before it -- and a data-free header is exactly what an AO message is. Block order (heights ascending, then each block's own transaction order) is a total order and is already canonical. The process's own transaction is skipped, as the first sync deliberately re-scans the spawn block and slot 0 is already the process. Assignments in this mode also record the `block-height' that sequenced them, so a process reading the whole chain has a deterministic clock; until now an assignment carried only its offset, and a block-denominated deadline was inexpressible. `target'-mode assignments are untouched, so existing schedules keep their assignment ids. Two things this mode cannot do the way `target' mode does. It must not drop a transaction the block lists but the local index does not hold: indexing a block is all-or-nothing and reports success either way, so dropping them would silently shorten the schedule and -- slots being positional -- shift every later slot on that node alone, with both nodes serving 200s and claiming the same attestable range. The range fails instead, leaving `synced-to' for the next pass to retry. And it sequences bundle transactions, whose headers the shallow indexing pass deliberately does not cache, so a header that is not held locally is fetched data-free on demand rather than stopping the sync. Tested: mode selection and its fallback, slot-0 detail per mode, block enumeration order and process-tx exclusion, offset annotation without re-sorting, failing closed on an unindexed transaction, the assignment cache round trip carrying block-height, and the mode's round trip through the sync state. The existing live-weave fixture tests still pass unchanged.
The device promises that assignment bodies are transaction headers only, so that a schedule never depends on the availability of anybody's data. Reading one through the whole store chain broke that promise: the read falls through to a gateway, which answers with the entire transaction -- its data, or, for a bundle, its items decoded into submessages whose contents this node does not hold -- and `lib_scheduler:write_assignment' then forces the assignment to load. The slot dies on a link to a chunk nobody fetched. In `target' mode this never came up, because the only messages in a schedule were the header-only ones addressed to the process. In `all' mode every transaction on the network is a message and most of them carry data, so it came up immediately: replaying a five-block range failed on a stranger's bundle. Headers are now read from the node's own stores, where the indexing pass has cached a data-free header for every plain transaction it walked, and anything else is fetched as a header in its own right with `exclude-data'. Loading whatever happened to be available locally would have been worse than either: two nodes would mint different assignments for the same transaction. Found by replaying a real sale off mainnet; the live-weave fixture tests for `target' mode are unchanged and still pass.
A process device that sells its own token for Arweave's. The two halves of the trade live in different places, and everything here follows from that asymmetry: the token half is in the process, where it can be held in escrow and paid out with certainty, and the AR half is not -- it moves directly between two user addresses, and no process can hold, redirect or refund it. So the process is sequenced by `~arweave-scheduler@1.0' in its `all' mode. The payment leg is an ordinary transfer to the seller, tagged with an `order-id'; the process is not addressed by it and sees it only because it is sequenced by every transaction on the network. The protocol is four messages. `make-offer' escrows the seller's `offer-quantity + deposit' out of the `balances' submessage that a token implementation keeps in the same base, recording an `asking' price in winston and a `deadline' in blocks. The offered amount cannot be called `quantity': that is a transaction's own value field, so the codec would carry it as winston of AR sent to the process -- an address with no key, which would destroy it. `register-interest' costs nothing and moves nothing, but makes the order exclusively one buyer's and freezes the seller out of cancelling it -- that window is the whole point, since without it a buyer races the seller's cancellation having already sent AR that nobody can claw back. `cancel-order' returns the goods. The payment settles the order, crediting the goods to the payer and returning the bond to the seller. The `deposit' is the seller's bond against a payment the protocol cannot honour, so it outlives the goods: cancelling or expiring an order releases them at once but holds the `deposit' until `deadline + swap-cancel-grace'. A payment landing in that window against goods that are gone or spoken for is paid the bond instead. A seller who strands nobody always gets it back. Neither the order's creator nor its recipient may pay it: paying oneself costs only a network fee, and would otherwise let a seller settle their own order, taking back the goods and the bond and leaving a real payment to arrive against an order already spent. Deadlines are read from the `block-height' the assignment carries, which is the device's only clock: reading the chain tip during a compute would be non-deterministic, and a slot result is cached forever. Expiry is swept lazily before each dispatch, guarded by a single `next-deadline' integer, because in this mode the device runs on the order of a thousand times per block and almost every slot is unrelated traffic. Being sequenced by the whole chain also means every message it computes is a stranger's, and a slot that cannot be computed can never be skipped: the schedule is immutable chain data, so one bad slot stops the process on every node forever. The device therefore exports no key list -- the key a slot resolves comes from the sender's own `path' tag, and a key outside an export list would answer `not_found' and fail the slot -- and excludes none either, since an excluded key is handed to `~message@1.0', whose `set' would let a passer-by write the process's own balances and whose `keys' would replace the state with a list of key names. `info' is arity 0 for the same reason: a device's `info' is always exported, so at arity 1 it would answer the `info' key and hand back its own map as the process state. Figures a stranger wrote are read with `hb_util:safe_int/1', because a tag like `deadline: tomorrow' would otherwise raise out of the enclosing `maybe', which catches mismatches and not exceptions. An `order-id' is looked up as a key of the order book rather than resolved as a path, or `<id>/creator' would reach an address and be read as an order. And the state is read as a message, because during a compute it carries this device, and a plain read of one of its keys would resolve that key back through `compute'. Tested: whole trades played out against the device -- settlement, underpayment, payment to the wrong address, cancellation and bond retirement, cancellation by a stranger, reservation blocking cancellation, reservation exclusivity and lapse, expiry, a late payment taking the bond, the bond paying out only once, double settlement, a seller unable to buy or compensate their own order, unknown and hostile order ids, a non-numeric figure, an offer that misnames its amount `quantity', stray and reserved `path' tags, numbers that came back from the cache encoded as binaries, and unrelated traffic moving only the clock.
Reserving an order costs the seller something: it freezes their bond and their right to cancel behind somebody who may never pay. The only thing making that costly to abuse was the seller's `deposit', which protects the buyer, not the seller -- and an order with no bond had no protection at all. An order may now declare a `minimum-fee' in winston, which a `register-interest' transaction pays as its own transaction `reward' -- the fee the network is already charging to accept it. Overpaying the reward sends the difference where Arweave sends rewards, to miners and the endowment, so the deterrent funds the weave. The alternative was to have the registration send value to the process, but a process address is a transaction id with no key behind it: that would stranded the fee there forever. Value sent to the process is therefore explicitly not the fee and does not clear it. Being denominated in AR, the fee asks nothing of a buyer who holds none of the token they are trying to buy. That is what makes a name sellable: a name's whole supply is the single unit being offered, so its seller has nothing left to bond with, and a zero-deposit order is exactly the case that needed defending. `make-offer' asks such an order only for the goods. Tested: an order that charges turns away a registration that pays nothing, one that underpays, and one that sends value to the process instead; it admits a registration whose reward covers the fee. An order that charges nothing still registers free, and the holder of a single unit can offer it.
A name is a token with a single unit and a message attached. Whoever holds
the unit decides what that message says; transferring the unit hands over
that right with it. Selling the unit -- with `~arweave-swap@1.0', which
settles in the same balances -- sells the name.
The device is `token-1.0''s shape carrying `~reference@1.0''s payload. From
the token: the state layout, the `transfer' action and its `Debit-Notice' /
`Credit-Notice' pair, and the rule for who may `set' -- the holder of the
whole supply, which `token-1.0' already calls `supply-threshold-owner'. From
the reference: what `set' writes, a linked message the name resolves through
to, given as a `reference-value' or inherited from the `set' message's own
keys. There is no mint path, so a name cannot be diluted into existing twice.
Ordering is simply the schedule's. `~reference@1.0' must reconstruct which of
several competing `set's won, from a signer-declared timestamp tie-broken by
weave position, because it is assembled out of loose layer-1 messages. A
process is handed that order already.
Two things are shaped by the chain rather than by taste. A process spawned as
an Arweave transaction may only carry scalars -- the codec turns a submessage
or a list into a `+link' to content the weave does not hold, and keys are
lowercased on the way, which would quietly rewrite an address. So the single
unit is not written into the spawn: the spawn names its first holder in a
value, `initial-holder', and the unit appears the first time the process
computes. And the selling device is named by a scalar, `swap-device', with
this device handing each message to it before applying its own actions --
`~stack@1.0' written out by hand, because `device-stack' is a list and a list
does not survive the spawn.
Reads are paths into the state, as `token-1.0' reads too. The device exports
no key list and answers every key with `compute', because a name that is for
sale must be sequenced by every transaction on Arweave, which means a
stranger's `path' tag chooses the key each slot resolves.
Tested against three names that were really sold, transferred and withdrawn
on mainnet, covering every path and refusal:
* a name that had to be paid for -- a registration paying only what the
network charges is refused, so is one that sends the fee to the process
instead of paying it as its reward, and the one that overpays the reward
reserves it; a payment from somebody who did not reserve it buys nothing;
the registrant's payment settles; the seller can no longer speak for the
name and the buyer can; a stranger routing a slot at `info' breaks
nothing and an offer whose deadline is `tomorrow' opens nothing.
* an offer withdrawn -- the unit comes back, a later registration paying
the fee in full is refused, a later payment moves nothing in either
direction, and a stranger cannot withdraw somebody else's order.
* a name handed over -- a plain transfer, its notices, and the authority
moving with the unit.
Each story pins its transaction ids, resolves each to the slot it landed at,
and reads the state forward one slot at a time
(`<process>~process@1.0/compute&slot=<n>/orders/<id>/status'), so what the
schedule did is visible in the order it happened. The driver that seeded them
is here too, gated behind HB_LIVE_SUITE because it spends real AR, reporting
through `?event'.
A name minted with nothing in it says nothing until somebody sends it a `set`, and the first message addressed to a process pays Arweave's new-account fee -- so every name would cost that before it could resolve at all. Neither the holder nor the value can be written into the spawn directly: `balances` and the linked message are submessages, and a submessage does not cross the chain. What does cross is a scalar. `initial-holder` is an address and `initial-value` the id of whatever the name should resolve to, and the device turns them into the balance and the value the first time it computes. Also here: the live suite that exercises this against mainnet, reading state slot by slot through `hb_ao:resolve/2` so the reader can see how each message moved it.
`~arweave-scheduler@1.0' in `all' mode hands a process every transaction on Arweave. That is the point -- it is how a swap sees a plain transfer between two other addresses as the payment for a name -- but it means every message these devices read was written by somebody else, and the `device' key of each is the author's to choose. `hb_ao:get/4' dispatches on that key. A device answers only for the keys a message does not carry, so the reads that could be attacked are exactly the ones with a default: leave the key out, and the answer comes from whatever device the author named. `lua@5.3a' answers with anything, and both its `device' and `module' tags survive a signed Arweave transaction intact, so the payload is deliverable by publishing it. Both devices already had the pinned idiom for reading their own state, for the mirror-image reason -- a plain read of the state would resolve back into `compute'. `field/4' is the same move pointed outwards, and the seven reads in `~name-token@1.0' and eleven in `~arweave-swap@1.0' that took untrusted input now go through it. Two regressions, one per device, both signed transactions carrying a real `tx@1.0' commitment and a real Lua module. Unpinned, the name-token moves a unit on a message addressed to nobody, and the swap pays a seller's escrow to an address the seller never named. Pinned, neither does. Identity was never at risk: `hb_message:signers/2' resolves through `hb_ao:raw/5', which forces the message device, so the signer of a message cannot be forged. What this closes is the gate that decides whether a message was addressed to the process at all, the values the actions then run on, and the execution of a stranger's code inside every slot -- which is also a way to wedge any `all'-mode process for good.
charmful0x
commented
Jul 26, 2026
| field(Key, Msg, Default, Opts) -> | ||
| hb_ao:get(Key, {as, <<"message@1.0">>, Msg}, Default, Opts). | ||
|
|
||
| address(Address) when is_binary(Address), byte_size(Address) =:= 32 -> |
Author
There was a problem hiding this comment.
raw 32 canonicalization clause
|
|
||
| address(Address) when is_binary(Address), byte_size(Address) =:= 32 -> | ||
| {ok, hb_util:human_id(Address)}; | ||
| address(Address) when ?IS_ID(Address); byte_size(Address) =:= 44 -> |
Author
There was a problem hiding this comment.
the 44 byte size OR is to support solana address size. with IS_ID macro balances can support AR/ETH/SOL and excludes short weird addresses that may clash with ~reserved keys e.g. balances/path/set etc.
i know we always lean to keep the protocol neutral / open design by default, but this is good for in depth- defense for arbitrary user input.
samcamwilliams
force-pushed
the
feat/name-token
branch
from
July 27, 2026 00:54
717e937 to
824816e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.