feat: add ~arweave-scheduler@1.0, an Arweave base layer-sequenced scheduler - #1022
Open
samcamwilliams wants to merge 7 commits into
Open
feat: add ~arweave-scheduler@1.0, an Arweave base layer-sequenced scheduler#1022samcamwilliams wants to merge 7 commits into
~arweave-scheduler@1.0, an Arweave base layer-sequenced scheduler#1022samcamwilliams wants to merge 7 commits into
Conversation
samcamwilliams
force-pushed
the
feat/arweave-scheduler
branch
2 times, most recently
from
July 16, 2026 19:52
f0a93f5 to
fa9ece6
Compare
speeddragon
reviewed
Jul 23, 2026
speeddragon
left a comment
speeddragon
left a comment
Collaborator
There was a problem hiding this comment.
Review part of the code with help of AI. I will continue with the reaminig PR review later.
Comment on lines
+290
to
+292
| case hb_store:resolve(Store, Path, Opts) of | ||
| {ok, ResolvedPath} -> | ||
| case hb_cache:read(ResolvedPath, Opts) of |
Collaborator
There was a problem hiding this comment.
hb_cache:read already calls hb_store:resolve. We can delete the previous call.
Comment on lines
+432
to
+433
| "&from=", (hb_util:bin(To))/binary, | ||
| "&to=", (hb_util:bin(From))/binary |
Collaborator
There was a problem hiding this comment.
This logic is a bit weird. I understand that when we think about this, we think in ascending order (from=1 to=10), but the logic inside copycat is the other way around.
Maybe we should add a comment explaining this so that if someone looks at it, they don't think it's a typo or something similar.
Comment on lines
+63
to
+79
| write_state(ProcID, State, RawOpts) -> | ||
| Opts = opts(RawOpts), | ||
| Store = hb_opts:get(store, no_viable_store, Opts), | ||
| #{ | ||
| <<"next-slot">> := NextSlot, | ||
| <<"spawn-height">> := SpawnHeight, | ||
| <<"synced-to">> := SyncedTo | ||
| } = State, | ||
| hb_store:write( | ||
| Store, | ||
| #{ | ||
| state_path(ProcID, <<"next-slot">>) => hb_util:bin(NextSlot), | ||
| state_path(ProcID, <<"spawn-height">>) => hb_util:bin(SpawnHeight), | ||
| state_path(ProcID, <<"synced-to">>) => hb_util:bin(SyncedTo) | ||
| }, | ||
| Opts | ||
| ). |
Collaborator
There was a problem hiding this comment.
Should I assume we don't care if we successfully stored the state? This is normally the default behaviour on other write-cache implementations, but only because the code isn't properly handling when it fails to write.
| ensure_offsets(From, To, Opts) -> | ||
| maybe | ||
| {ok, _} ?= | ||
| hb_ao:resolve( |
Collaborator
There was a problem hiding this comment.
A call to dev_copycat_arweave, fetch_blocks doesn't produce an error tuple, making this always be {ok, _}. We want to check the output of process_block and make it return an error if it cannot process it.
The main issue is synced-to property of the state can be updated with something that didn't happen. We also have the case where a TX couldn't be processed (was skipped). I'm not sure if this should be signaled as an error for this use case.
We probably want to keep the current behaviour to not break the indexing of other blocks, but for this use case we want it to return an error if that happens.
AI Agent issue description (below)
Impact: a transient gateway/header failure for one block can cause transactions in that block to be omitted permanently for that process, while /status still reports the range as attestable.
Proposed fix: make bounded copycat indexing return an error or highest contiguous fully-indexed height when any block is skipped. The scheduler should only advance synced-to over fully verified contiguous ranges. Add a mocked regression test where one tx header fetch fails and assert the scheduler does not advance.
Comment on lines
+354
to
+356
| {ok, State} ?= ensure_initialized(ProcID, Opts), | ||
| {ok, Upper} ?= confirmed_tip(Opts), | ||
| do_sync(ProcID, State, Upper, Opts) |
Collaborator
There was a problem hiding this comment.
Slot 0 will always be filled even if the confirmed_tip is below the current block. We should always check SpawnHeight to reach synced-to before initializing the process slot 0.
AI explanation:
High — Spawn initialization does not enforce confirmation depth
sync/2 initializes the process before computing the confirmed tip:
- ensure_initialized runs before confirmed_tip: src/preloaded/process/dev_arweave_scheduler.erl:352
- initialize/2 accepts any /tx/status block_height: src/preloaded/process/dev_arweave_scheduler.erl:650
- confirmation depth is only applied later: src/preloaded/process/dev_arweave_scheduler.erl:746
Impact: slot 0 can be cached for a process transaction that is mined but not yet under the configured confirmation depth. If that block changes, the node has already materialized state.
Proposed fix: compute confirmed tip first and reject/defer initialization unless SpawnHeight =< ConfirmedTip. Add a test with mocked current height/status.
Caching an L1 transaction header (every mode) and caching a fully-parsed bundle item (full mode) used the same scope-convert-write sequence with different codecs. Share it as `cache_item/3'; behaviour is unchanged (the header path keeps its log-and-skip guard, the full-mode path still asserts the write).
…to lib_scheduler `~scheduler@1.0' and the `~arweave-scheduler@1.0' device added later in this branch present the same consumer interface, so everything about interpreting a scheduling request -- and storing and serving its results -- is identical regardless of how the schedule is sourced. Move that shared logic into a new `lib_scheduler' device library: - `find_target_id/4,3' and `find_message_to_schedule/3' (request parsing), `load_message_to_schedule/3' (find + fully load, 404 on failure), and `only_committed/2' (reduce to committed components, 400 on failure). - `parse_slot_range/2' (the `from'/`to' slot parse; `from' is now clamped non-negative after coercion, which also catches a negative binary input) and `read_assignment_range/5' (bounded range read + truncation flag), with the single `?MAX_ASSIGNMENT_QUERY_LEN' definition and `max_assignment_query_len/0' for callers that bound remote requests by it. - The assignment cache: `cache_opts/1', `write_assignment/3' and `read_assignment/4', parameterized by each device's pseudo-path prefix. The read restores unsigned commitment IDs and converts legacy `ao.TN.1' assignments, exactly as `dev_scheduler_cache:read/3' did. - `base_assignment/4', the assignment keys every scheduler shares -- `dev_scheduler_server:do_assign' merges its block fields and commits; other schedulers merge their own position fields. - `at_slot/2', `slot_unavailable/0' and `format_opts/1' (the no-cache/no-await formatting options, previously inlined in several places). `dev_scheduler', `dev_scheduler_cache', `dev_scheduler_formats' and `dev_scheduler_server' delegate to the library; no behavioural change.
`dev_arweave' and the `~arweave-scheduler@1.0' device both broadcast a request to several Arweave nodes and take the most successful reply. Move `dev_arweave''s status-sorting `best_response/1' (with its `response_status/1' helper) into `lib_arweave_common', which both devices already depend on, and drop the local copy. No behavioural change for `dev_arweave'.
A transaction node returned from the local ~query@1.0 GraphQL endpoint resolved `block' to null, because a cached tx carries no block metadata of its own. Carry each match's weave offset onto its node and resolve `block' by locating the cached block whose byte range contains that offset, distinguished from the top-level block(id/height) query by the offset key on the node. Resolution is local but scans the cached blocks, so callers that only need on-chain ordering should use the offset directly. The cached-heights enumeration and the block byte-bounds arithmetic are shared as `cached_heights/1' and `block_bounds/2' with the module's pre-existing `latest_cached_block/1' and `block_range_to_offset_range/2', which previously inlined the same code separately.
A drop-in ~scheduler@1.0 replacement that sources a process's schedule directly from Arweave layer-1 transactions. Slot 0 is the process message (its tx id is the process id); slot N is the Nth confirmed L1 tx whose `target' is the process id, in ascending weave-offset order. Assignments are deterministic and uncommitted -- Arweave itself is the authority -- so every node converges on identical assignment ids without trusting a scheduler wallet. The on-chain position recorded on each assignment is the weave offset. Discovery is local-first, with no gateway attribution: a single `~copycat@1.0/arweave' shallow pass over the range records each tx's weave offset and caches its (data-free) header, which populates the local `field-target' match; the device then resolves the node's own ~query@1.0 (via hb_ao:resolve, not a remote call) for the recipient set, ordered by that offset index. `arweave_scheduler_query_source => remote' opts into a gateway query instead. The block index is node-shared and `ensure_offsets' passes `reindex=false', so N processes on one node index each block once rather than once per process. The device is header-only end to end. Bodies are read as tx headers, so the schedule never depends on data availability; the header verifies and its committed id is the tx id. POST /schedule likewise relays the header only, rejecting a message that carries data (422) rather than uploading chunks, and rejecting any non-tx@1.0 commitment (422). The misleading tip-derived timestamp/block-height/block-hash that the standard formatter stamps are omitted from a chain-derived schedule. Synchronization tracks a `synced-to' block high-water mark: the attestable range `[spawn-height, synced-to]' the node can vouch for. A read with no new blocks does no network work; otherwise only the delta is indexed, in `arweave_scheduler_sync_chunk'-block passes whose progress is persisted per chunk (resumable first sync). /status reports each tracked process's range. The device is built on `lib_scheduler': request parsing, the assignment cache, the range read, and the base assignment schema are the same code paths ~scheduler@1.0 uses. The full consumer interface (status/next/schedule/slot/init/checkpoint + router) is implemented so ~process@1.0 drives it unchanged. Includes a permanent mainnet test fixture (a lua@5.3a process whose three scheduled messages transform state 1000 -> 1337, one posted directly to arweave.net to prove foreign-tx indexing) and device unit tests.
A process message may now carry `scheduler-mode: all', and its schedule becomes every base-layer transaction on the network rather than only those whose `target' is the process. This is what lets a process observe value moving between two other addresses -- a payment it is owed but is not a party to -- which no recipient-filtered schedule can see. The mode is read from the process message itself at first contact and pinned into the persisted sync state. The process message is an L1 transaction, so the mode is chain data like the rest of the schedule, and every node derives the same one; a node option would have made slot -> body a function of node config, producing two indistinguishable, equally `valid' schedules for one process id. An unrecognized mode falls back to `target' rather than erroring: a spawn tag cannot be corrected once it is on-chain. `all' mode does not query. Dropping `recipients:' from the GraphQL would have silently returned nothing -- `dev_query_arweave' only builds candidate sets from height/id/ids/tags/owners/recipients, and `block:' is a post-filter over a set that must already exist -- so the range's block headers are walked directly instead, taking each block's own `txs' list. That is the same source `~copycat@1.0/arweave' enumerates, and `ensure_offsets' has already cached those blocks, so it is a local read. Nor does it re-sort by weave offset. Offset is not a total order: `TXStartOffset = TXEndOffset - data_size' and the accumulator only advances by `data_size', so every data-free transaction in a block shares the offset of the one before it -- and a data-free header is exactly what an AO message is. Block order (heights ascending, then each block's own transaction order) is a total order and is already canonical. The process's own transaction is skipped, as the first sync deliberately re-scans the spawn block and slot 0 is already the process. Assignments in this mode also record the `block-height' that sequenced them, so a process reading the whole chain has a deterministic clock; until now an assignment carried only its offset, and a block-denominated deadline was inexpressible. `target'-mode assignments are untouched, so existing schedules keep their assignment ids. Two things this mode cannot do the way `target' mode does. It must not drop a transaction the block lists but the local index does not hold: indexing a block is all-or-nothing and reports success either way, so dropping them would silently shorten the schedule and -- slots being positional -- shift every later slot on that node alone, with both nodes serving 200s and claiming the same attestable range. The range fails instead, leaving `synced-to' for the next pass to retry. And it sequences bundle transactions, whose headers the shallow indexing pass deliberately does not cache, so a header that is not held locally is fetched data-free on demand rather than stopping the sync. Tested: mode selection and its fallback, slot-0 detail per mode, block enumeration order and process-tx exclusion, offset annotation without re-sorting, failing closed on an unindexed transaction, the assignment cache round trip carrying block-height, and the mode's round trip through the sync state. The existing live-weave fixture tests still pass unchanged.
samcamwilliams
force-pushed
the
feat/arweave-scheduler
branch
from
July 25, 2026 03:21
e444060 to
1fd8d85
Compare
The device promises that assignment bodies are transaction headers only, so that a schedule never depends on the availability of anybody's data. Reading one through the whole store chain broke that promise: the read falls through to a gateway, which answers with the entire transaction -- its data, or, for a bundle, its items decoded into submessages whose contents this node does not hold -- and `lib_scheduler:write_assignment' then forces the assignment to load. The slot dies on a link to a chunk nobody fetched. In `target' mode this never came up, because the only messages in a schedule were the header-only ones addressed to the process. In `all' mode every transaction on the network is a message and most of them carry data, so it came up immediately: replaying a five-block range failed on a stranger's bundle. Headers are now read from the node's own stores, where the indexing pass has cached a data-free header for every plain transaction it walked, and anything else is fetched as a header in its own right with `exclude-data'. Loading whatever happened to be available locally would have been worse than either: two nodes would mint different assignments for the same transaction. Found by replaying a real sale off mainnet; the live-weave fixture tests for `target' mode are unchanged and still pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
~arweave-scheduler@1.0: a drop-in~scheduler@1.0replacement that sourcesa process's schedule directly from Arweave base layer transactions instead of from a
scheduling authority. Slot 0 is the process message; slot N is the Nth confirmed L1 tx
whose
targetis the process id, in ascending weave order. Assignments are deterministicand uncommitted — Arweave itself is the authority, so every node converges on
identical assignments without trusting a scheduler wallet.
Functionality
~copycat@1.0/arweaveshallow pass store each
~tx@1.0message and its Arweave offset (populatingfield-targetfor matching andoffset-based ordering); the device then resolvesthe node's own
~query@1.0for the recipient set, ordered by that offset index.arweave-scheduler-query-source: remotein a node'sOptsallows it to delegateto another GraphQL source if preferred. Default is fully decentralized, local indexing.
The block index is shared between all processes (and other users of the data) at the
node level.
reindex: falseis passed to~copycat@1.0(see impr:
copycat@1.0Arweave indexing features #1021), so N processes onone node index each block once.
never depends on chunk seeding.
POST /schedulerelays the header only,rejecting messages that carries data (422) or non-
tx@1.0commitments (status: 422).synced-toblockhigh-water mark: the range
[spawn-height, synced-to]the node can vouch for.A read with no new blocks does no network work; otherwise only the delta is
indexed, in resumable chunks. The
/statuskey reports each tracked process's range.~scheduler@1.0interface (status/next/schedule/slot/init/checkpoint)compatibility, such that any
~process@1.0message may utilize the device seamlessly.Commit guide
refactor(scheduler): extract shared request-parsing intolib_scheduler(dedups
dev_scheduler). This standard scheduler device library may be used byother implementation of the standard in the future.
refactor(arweave): sharebest_responsevialib_arweave_common(dedups functionality previously in
dev_arweave).fix(query): resolveblockon~query@1.0transaction nodes (standalone).feat: the device itself.