Add catnix gateway adaptors and Hydra coverage - #2
Closed
georgewhewell wants to merge 29 commits into
Closed
Conversation
Replace the prefix-snapshot caching architecture with a single
commitment-keyed exact-replay cache plus a content-addressed receipt
store.
Architecture shift:
- The previous design cached intermediate session state at every
prefix length, with a `CausalStepper` trait abstracting the decode
loop so a fake stepper could substitute for the catgrad session in
property tests of "split stability" — the invariant that
`prefill(P + S)` == `prefill(P) ; advance_one(s_i for s_i in S)`
bit-for-bit. The whole approach is gone: it imposed a strict
determinism contract on backend kernels that's hard to honor across
hardware, and the cache hit ratio in practice didn't justify the
complexity.
- New design: each `ExecutionContext` keeps two caches.
* Continuation cache, keyed by `Cid<TextExecution>` — the request
commitment over (program, parameter tensor CIDs, prompt tokens,
policy). Same hash ⇒ byte-identical ask ⇒ stream stored tokens
without touching the model.
* Receipt store, keyed by `Cid<TextReceipt>` — names a particular
`(commitment, final state, output tokens, position)` tuple.
Populated at bind time with the genesis receipt (cold-start
anchor) and at end of every real execution with that execution's
final receipt. Anchored requests look up their incoming
`initial_receipt_id` to find the live state to start from.
Code consequences:
- `runner.rs`: drop the generic-over-stepper `decode<S: CausalStepper>`
function; the runner is now a concrete prefill-then-decode flow over
`TextDecoder`. `build_text_execution` builds the request commitment
for the quote path.
- `runner/tests.rs`: deleted (526 lines of split-stability proptests
testing an invariant we no longer pursue).
- `programs/context.rs`: drop `CATCH_UP_THRESHOLD` and the suffix
teacher-force path; `ExecutionStart` no longer carries a
`transcript`. Genesis receipt computed at bind time and exposed via
`genesis_receipt_id`.
- `executor/actor/{quote,execution}.rs`, `worker.rs`: track only the
cached-output-tokens count for stats, not cached prompt tokens.
Docs:
- `docs/PREFIX.md` documents the prefix-cache decision and why we
retired the split-stability approach.
- `docs/DISCOVERY_E2E.md` covers the discovery flow.
Cargo:
- Enable the local `[patch]` for `../catgrad/{catgrad,catgrad-llm}` to
iterate against the catgrad runtime-primitives branch in tandem.
…--pi flag
Collapses the two-step Execute → ExecuteStream RPC pair into a single
streaming Execute, deleting the entire late-subscriber path:
LocalExecutionStream, SubscriptionSet, broadcast channel, close-monitor,
handle_subscribe/handle_status/SubscriptionsClosed, and the buffered
output + status fields on ExecutionRecord. The per-execution
mpsc::Receiver returned by Execute IS the subscription; dropping it
closes the worker's sender, which fires the runner's CancellationToken
between decode steps. End-to-end drop-cancellation with no token
plumbing on the CLI side.
CLI execution layer is now stream-shaped end to end (ExecutionRequest::
stream, Outcome { Completed { receipt_cid, stop_reason }, Failed }).
Gateway handlers rewritten with async_stream::stream! around prepared.
stream(); sse_response is a one-line wrapper (no spawn, no channel).
Verify shadow strategy compares 32-byte receipt CIDs instead of full
output bytes.
inputs::Error variants now carry the originating HuggingFaceLocator;
adding `impl From<inputs::Error> for ExecutorError` lets every callsite
use `?` instead of map_err helpers. Three duplicate map_weights_error /
map_program_cache_error helpers deleted.
Numerous smaller dedups: state/ collapsed to state.rs, runner helpers
inlined, dead ExecutorError/ModelAssetsError variants removed, dead
ModelAssets methods deleted, build_text_execution moved onto
ExecutionContext, quote-prompt/chat-prompt assets-load deduped.
Also adds --pi flag to gateway: spawns pi-coding-agent against the
just-bound listener, exits when pi exits. Requires --force-model.
Net: −552 lines across 44 files; executor crate ~30% smaller.
- rpc: add From<ModelAssetsError> for Status and From<TokenBytesError>
for Status. Factor model_assets_status_code / executor_status_code
helpers so the impls compose without duplicating the per-variant
code-mapping table.
- rpc: add ExecutorError::TokenBytes(#[from] TokenBytesError) so
decode_token_ids can use ? from executor code.
- executor handle.rs: decode_tokens collapses a 4-deep nested match
(7x duplicated Status::internal(format!(...))) into a flat ? chain
inside a small decode closure. The first-message path uses .next()..??
to chain Option/Result unwrapping; gRPC-stream Err is propagated
without wrapping (it's already a Status).
- executor state.rs: drop the InvalidTokenPayload(error.to_string())
wrap; the new From impl propagates via ExecutorError::TokenBytes.
- cli text_output.rs / monitor.rs: swap two anyhow!("...: {err}")
flatten sites for .context("..."); preserves the error chain.
…ers + SSE events Threads catgrad execution provenance from the executor to HTTP gateway clients across two boundaries. **Executor → gateway** (tonic): mirrors the existing OTel W3C trace-context propagation pattern. Server-side attaches commitment_id + program_id to `Response::metadata_mut()` for both unary `get_quote` and the streaming `execute` initial metadata; client-side `RemoteExecuteDriver` extracts them before `into_inner()`. The local mpsc path (`ExecutorHandle`) carries the same struct alongside its event receiver. Receipt continues to flow via the existing `Completed.receipt_cid` proto field. **Gateway → HTTP client**: a new `tower::Layer` (`ProvenanceLayer`) wraps the axum router. Handlers insert `ExecutionProvenance` and (for buffered responses) `Cid<TextReceipt>` into `response.extensions_mut()`; the layer lifts both into `x-hellas-commitment-id` / `x-hellas-program-id` / `x-hellas-receipt-id` headers on the way out. Cross-cutting; no per-handler header-attachment boilerplate. For SSE responses the receipt is unknown at header-flush time, so the three SSE handlers (openai, anthropic, plain) also emit named in-band events: `event: hellas-provenance` first, `event: hellas-receipt` immediately before each protocol's terminal frame. This covers browser `EventSource` consumers which can't read response headers or HTTP trailers. New module `hellas_rpc::provenance` owns the wire-format primitives: `ExecutionProvenance` struct, header/metadata key constants, hex encoding/decoding, and `MetadataMap` round-trip helpers. Bytes-based so the rpc crate doesn't pull catgrad into its `client` feature; callers reconstitute typed `Cid<T>` at their boundary. `ExecutionRequest` now exposes `prepare()` separately from `stream()` so the gateway can read pre-flight provenance off the resulting `PreparedExecution` before any stream events flow. Tests: provenance round-trip (encode/decode, missing/malformed key handling), `ProvenanceLayer::apply_provenance_headers` unit coverage, plus an end-to-end Router test using `tower::ServiceExt::oneshot` that confirms extensions become headers through the full axum stack.
…rogram The commitment CID hashes (program, parameter CIDs, prompt tokens, policy), so it transitively identifies the program. Exposing `x-hellas-program-id` as a separate header was redundant. Removed: - `PROGRAM_HEADER` constant and the `program_id` field on `ExecutionProvenance`. - The `x-hellas-program-id` header attachment in `ProvenanceLayer`. - The `program_id` field in the `hellas-provenance` SSE event payload. - Server-side population in the executor's quote/execute handlers. The receipt header (`x-hellas-receipt-id`) and the commitment header (`x-hellas-commitment-id`) remain.
The gateway was emitting `event: hellas-receipt` on the wire but never
`info!`-ing it. So under `--pi`, when a client request finishes, we had
no server-side trace of which receipt was produced — making it
impossible to tell from logs whether the executor's terminal Outcome
actually reached the gateway or was lost to drop-cancellation.
Added an `info!` at each `Outcome::Completed` observation site in the
three protocol handlers (buffered + SSE = 6 sites total), recording
receipt_cid, provenance, total_tokens, and stop_reason.
To make the log fields readable: replaced the derived
`Debug for ExecutionProvenance` (which printed
`ExecutionProvenance { commitment_id: [171, 171, ...] }`) with a manual
impl that delegates to Display, so `?provenance` and
`?Option<ExecutionProvenance>` render as `Some(deadbeef...)` / `None`.
…e_tool_calls
Replaces the post-hoc model.parse_tool_calls(text, arch) dispatch +
has_tools-buffered streaming with an event-driven loop driven by
catgrad-llm's new ChatTurn / IncrementalToolCallParser API.
crates/rpc/src/model/assets.rs
drops parse_tool_calls and
prepare_chat_with_tools. Adds
chat_turn(wire_tools, options) -> ChatTurn
doing wire-shape -> typed ToolSpec
conversion + ToolDirectory build (compiles
JSON schemas) + protocol lookup. Empty
wire-tools list normalized to None at the
edge. Arc-wraps tokenizer / chat_template /
tokenizer_config / stop_token_ids so
chat_turn() clones cheaply per request.
crates/rpc/src/model/mod.rs
adds two ModelAssetsError variants used
by the gateway to classify request errors:
InvalidToolDirectory (bad schemas) and
ToolsUnsupportedForModel (caller asked for
tools but the architecture has no
registered protocol). Both surface as HTTP
400, never 502.
crates/cli/src/commands/gateway/state.rs
PreparedGeneration drops `has_tools` and
carries `chat_turn: Option<ChatTurn>`
instead. prepare_openai / prepare_anthropic
build the ChatTurn first, render via it,
then finalize generation. classify_chat_turn_error
maps tool-config errors to 400 request
errors.
crates/cli/src/commands/gateway/openai.rs
ground-up rewrite. Both respond (non-
streaming) and stream_response use
chat_turn.make_parser() and walk
DecodeEvents. apply_event accumulates
content + tool_calls and returns Terminal
(HTTP 502) for UnknownTool / InvalidArgs /
ParseError. stream_apply_event maps each
event to OpenAI SSE chunks per the wire
convention (Start carries name+id;
ArgsDelta carries arguments fragment with
no id repeat; End emits no separate frame;
terminal events emit error frame and close
WITHOUT [DONE]). saw_tool_call drives the
final finish_reason: tool_calls wins over
stop whenever any call was emitted.
crates/cli/src/commands/gateway/anthropic.rs
ground-up rewrite. Same pattern with
content_block_start / delta / stop
bracketing. Maintains its own block-index
counter (distinct from the parser's
tool-call index per the P6 contract).
Errors close with the anthropic `error`
event, no message_stop. Defensive
close_any_open_block before terminal
frames.
Plain endpoint stays passthrough — no chat template, no tool
contract; left untouched.
16 new unit tests covering apply_event / stream_apply_event /
events_to_blocks across both surfaces (97 total node tests pass).
End-to-end validated against Qwen3-0.6B on CUDA via curl: plain chat,
OpenAI/Anthropic non-streaming tool calls, OpenAI streaming tool
call, unknown-tool 502, invalid-args 502, bad-schema 400 — all match
the contract.
Pi has a rich TUI that mixes stdout and stderr; without redirection it clobbers the parent terminal. Adds --pi-log <path>; both streams go to the file when set, otherwise pi keeps inheriting the parent tty for interactive use.
Replaces bracketing 'event: hellas-provenance' / 'event: hellas-receipt'
SSE events with a namespaced "hellas" field on the existing
protocol-native JSON envelopes. Browser EventSource and many WASM
HTTP wrappers swallow response headers, so an in-band JSON extension is
the only carrier that reliably reaches those clients.
Wire shape per surface:
- OpenAI streaming: first chunk (role:assistant) carries
hellas.commitment_id; the SEMANTIC TERMINAL chunk -- the last data
event before [DONE] -- carries hellas.receipt_id. Without
include_usage that's the finish-reason chunk; with it, the trailing
usage chunk. Receipt placement is the testable invariant: "receipt
on terminal event," not "receipt on a finish-reason chunk that may
have other chunks after it."
- Anthropic streaming: hellas.commitment_id lives inside
message_start.message (on MessageResponse, same JSON path as
non-streaming so clients have one extraction path).
hellas.receipt_id rides message_stop, the structural terminator.
- Plain streaming: same shape as the chat surfaces.
- Non-streaming: the JSON body gets the same hellas extension on top
of the existing x-hellas-* headers (additive; headers stay).
Receipt injection is fenced strictly inside Outcome::Completed.
Transport / timeout / parser / Outcome::Failed branches emit cleanup
frames unwrapped -- no receipt leaks on any failure path.
Adds gateway/hellas_ext.rs with HellasExt + WithHellas<T> via
serde(flatten); catgrad-llm wire types stay protocol-neutral.
provenance_layer.rs is unchanged (still header-only; mutating bodies
in middleware would force buffering streams).
Anthropic streaming gets a build_anthropic_sse_stream extraction
parallel to OpenAI's testable seam, yielding AnthropicSsePayload
{name, json}.
Tests:
- 5 unit tests for the WithHellas wrapper (skip-empty, hex
rendering, flatten merge).
- OpenAI streaming_done_tests extended with positive-path coverage
(commitment on first, receipt on terminal, no commitment when
provenance is None, include_usage routes receipt to usage chunk)
and error-path receipt-leak guards on transport / timeout /
Outcome::Failed.
- Anthropic streaming_tests mirror the same coverage:
message_start.message commitment, message_stop receipt,
message_delta carries no receipt, errors emit no message_stop and
no receipt.
cargo test --workspace: 96 tests pass.
…teway edge, run_decode in executor
Repins catgrad/catgrad-llm to hellas-ai/catgrad@grw/feat/megatooler
(62aa3b1), which lands the WireMapper / AssistantTurnAccumulator /
run_decode refactor + typed tool wire shapes. Atomic with the
downstream code changes because catgrad-llm's exports widened
non-additively (ChatTurn::new return type, types::{openai,anthropic}
Tool fields).
Downstream:
- crates/rpc/src/model/assets.rs: ChatTurn now takes
Option<Arc<ToolDirectory>> and returns ChatTurnConfigError.
Wire-shape conversion (Vec<Value> -> ToolDirectory) moved out;
the gateway surfaces own it now via
ToolDirectory::from_openai_tools / from_anthropic_tools. Sheds
wire_tools_to_specs and the LLMError remapping shim.
- crates/rpc/src/model/mod.rs: ModelAssetsError collapses
InvalidToolDirectory + ToolsUnsupportedForModel into one
ChatTurnConfig variant carrying the typed catgrad-llm error.
Wire-shape errors are caught at the gateway edge and never reach
here.
- crates/cli/src/commands/gateway/state.rs: prepare_openai /
prepare_anthropic call the typed conversion helpers and pass the
resulting Option<Arc<ToolDirectory>> straight to assets.chat_turn.
Deletes the bespoke anthropic_tool_to_openai shim + its test --
Anthropic's input_schema vs OpenAI's parameters is folded in
catgrad-llm now.
- crates/executor/src/runner.rs: bespoke peek-stop-or-commit decode
loop replaced by catgrad_llm::runtime::run_decode. The new contract
is cancel-AFTER-commit (the in-flight token always reaches the
sink), one-extra-token in cancelled output vs the old loop.
Documented inline.
- nix/package.nix: bumps catgrad sha256 to match the new rev.
Adds gateway-multi-model: two executors (qwen + lfm2), one gateway
running in discovery mode (no --node-id/--node-addr), two pi processes
in parallel. Verifies that mDNS routing finds the right executor for
each requested model and that distinct requests produce ≥2 distinct
receipt_cid + commitment values in the gateway journal.
Plumbing:
- baseNode firewall: enable filter; allow 5353/udp (mDNS) and disable
reverse-path filtering (Linux drops multicast on bridged interfaces
by default).
- mkExecutorNode: openFirewall = true on the iroh listen port.
- mkGatewayNodeDiscovery: same shape as mkGatewayNode minus the
--node-id/--node-addr pinning, with iroh/pkarr/dns logs tightened
so structured fields stay legible.
- gatewayLauncherDiscovery: stripped command line for the discovery
case.
- hfHomeBoth: symlinkJoin of qwen + lfm2 caches so one gateway can
resolve config/tokenizer for both models.
Tool-use tests: pi output now goes to /tmp/pi.log via --pi-log;
gateway stdout/stderr lands in /tmp/gateway.log. Both are dumped
separately into the build log on success or failure. Same change
applied to gateway-tool-use-{openai,anthropic}.
|
Hydra reported failures for this PR.
Failed builds:
Update: hellas/node-pr-2, eval #969, head |
georgewhewell
added a commit
that referenced
this pull request
Sep 11, 2026
Every tonic-generated server registered on the node transport is now wrapped in `ManagedServer<S, …>` with `IrohPeerExtractor`. The wrapper handles peer observation, admission, rate-limit denials, and stashing the resulting `InboundAdmission` on the request extensions so handlers that need it (just `GetKnownPeers` for now) can read disclosure_limit without re-billing the rate bucket. Removals: - `ExecutePeerInterceptor` (used to hardcode `methods::RunTicket` for every Execute/Symbolic/Opaque call — review finding #2 + #3). - In-method `peer_directory.observe_inbound_request` calls in `get_node_info` and `get_known_peers`; the wrapper does it once with the typed policy from `RpcServiceSpec::inbound_policy(path)`. - `peer_observation(&request)` and `duration_ms` helpers — the extractor does both. - `tonic::service::interceptor::InterceptedService` import. - `tonic_iroh_transport::IrohContext` and `PathId` imports — code that needs the peer id falls through the typed extractor instead. `courtesy_service` is now wrapped too, closing the inbound-admission bypass flagged in the earlier review. Every inbound RPC on every service goes through one typed admission code path. The `service_alpn`-too-long check in `get_known_peers` stays in-handler since it's an application-level invariant on the request body (not a transport-level admission concern).
georgewhewell
added a commit
that referenced
this pull request
Sep 11, 2026
mux/transport.rs (generic ws driver): prepare_credit_updates now runs BEFORE the outbound drain, matching ws/cf_do.rs. Previously credit was prepared AFTER drain, so any Credit frame stayed queued in the slot's send_queued until the NEXT inbound activity — defeating the whole point of replenishment. mux/state.rs Body inbound: drop the eager local_recv_credit refill. Credit only refills via prepare_credit_updates' threshold check, which is the only path that ALSO queues a Credit frame back to the peer. The eager refill meant local_recv_credit never dropped below threshold so the Credit frame was never queued; the peer's peer_recv_credit would drain to 0 and stall. Both fixes from codex review #2.
georgewhewell
added a commit
that referenced
this pull request
Sep 11, 2026
Closes codex review #2 finding (snapshot poisoning). VALIDATION added to deserialize_mux: - Magic byte must match (already had this). - free_mask word count must equal (N+63)/64 — mismatched N rejects with SnapshotCorrupt("free_mask word count mismatch"). - free_mask bits for the peer's parity rejected (\"sets a bit owned by peer parity\") — prevents poisoning our slot allocator into handing out peer-domain indices. - count > N rejected. - slot_idx >= N rejected. - Duplicate slot_idx in the occupied list rejected. - A slot listed as occupied AND also in the free_mask rejected (the same-slot-dispatched-twice attack). - local_recv_credit > local_credit_high_water rejected. - peer_recv_credit > config.initial_credit rejected (peer can't have advertised more credit than they could ever receive). - has_pending=1 with len > 16 KiB rejected (attachment-budget enforcement). - Trailing bytes after the snapshot rejected. ADVERSARIAL TESTS (14 new + 2 existing + 1 proptest fuzz, total 17): - drop_and_fresh_start_is_a_fresh_mux - worker_dies_mid_callback_partial_attachment (every truncation prefix from 0..len rejects) - bit_flipped_magic_byte_rejected - future_magic_byte_rejected_with_named_error (version-upgrade case) - inflated_occupancy_count_rejected (count=33 vs N=32) - duplicate_slot_index_rejected - slot_in_both_free_mask_and_occupied_list_rejected - peer_parity_in_our_free_mask_rejected (poisoned allocator attack) - credit_exceeds_high_water_rejected - pending_write_oversized_rejected (1 MiB vs 16 KiB budget) - trailing_bytes_appended_rejected - n_word_count_mismatch_rejected (snapshot for N=32 into N=128 mux) - generation_survives_round_trip - random_bytes_never_panic_or_oom (proptest fuzz, 1000 cases, all re-serializable on success) cargo test -p hellas-wire (mux,iroh,ws,ws-cf-do): 25 tests pass.
georgewhewell
added a commit
that referenced
this pull request
Sep 11, 2026
`hellas rpc <id>` was broken: it dials Node::ALPN and calls
get_node_info, but `hellas serve node` only bound Execute/Symbolic/
Opaque/Courtesy ALPNs and dropped Node connections with a warn. The
"Node is pending" comment was the only mention.
This phase wires Node end-to-end with a real impl backed by the
existing peer state, so `hellas rpc` returns actual version, build,
graffiti, uptime, OS, and known-peers fields.
Changes:
- New `crates/cli/src/commands/serve/node_handler.rs` (~150 lines).
`NodeHandlerImpl: Clone` (fields are `Arc<str>` / `Arc<[u8]>` /
Copy) so we clone per-connection rather than wrap in `Arc<dyn>`.
Implements `hellas_rpc::services::node::NodeHandler`:
- `get_node_info` returns the exact proto fields (`node_id`,
`uptime_seconds`, `version`, `build`, `os`, `graffiti`).
Version = `env!("CARGO_PKG_VERSION")`; OS = `ARCH-OS`.
- `get_known_peers` uses `PeerDirectory::ranked_known_peers`
with a 64-peer disclosure limit. Requester defaults to
`PeerId::default()` (anonymous) — the generated trait doesn't
surface inbound peer identity to handlers. Phase F's
`AdmittingDispatcher` will fix that without a trait change by
enriching the request body, or via a thread-local. Until then,
`min_disclosed_auth_level` on the directory config remains the
gate.
- `serve/node.rs`:
- Added `Node::ALPN` to the bound ALPN list.
- Construct a `PeerDirectory` keyed on the local node id; pass an
Arc to the handler (and, in Phase F, to AdmittingDispatcher).
- Stopped discarding `build` and `graffiti` (previously `let _ =
(preload_weights, build, graffiti);`); they now feed the
handler. `preload_weights` stays parked.
- `serve_connection` takes a `NodeHandlerImpl` and grew a Node
ALPN arm: `let server = NodeServer(node_handler); serve_loop(...)`.
- `serve/mod.rs` registers the new `node_handler` submodule.
Tests:
- `commands::serve::node_handler::tests::get_node_info_populates_self_identity`
— checks version, build, OS shape, graffiti round-trip, uptime ≤ 1s.
- `commands::serve::node_handler::tests::get_known_peers_returns_empty_on_fresh_directory`
— fresh directory → empty list (verifies the disclosure path
compiles and returns the proto shape).
Verification:
- `cargo test -p hellas-cli --features hellas-executor` → 58 passed
(was 56, +2 new in the new module).
- `cargo test -p hellas-wire --all-features` → 57 passed.
- `cargo test -p hellas-rpc --all-features --lib` → 68 passed.
- `cargo build --workspace` clean.
Diffstat: 3 files, +184 / -7. Net: +177 (genuinely new functionality
— restoring a missing service). Tests: +51 of that. Net production
code: +126.
georgewhewell
added a commit
that referenced
this pull request
Sep 11, 2026
… rule Adversarial panel (Codex/gpt-5.3-codex-spark, Grok, nemotron-3-ultra) reviewed 8b5cd03..HEAD. Confirmed findings fixed: - StakeBondTerms commits challenge_margin (window + inclusion + finality, blocks); JobAcceptanceContext::covered_by is the admission rule (1 <= p_j <= max_job_price, terminal_deadline + margin <= bond timeout) and FraudArtifact::binds re-checks it — closes the stall-past-timeout escape's policy gap (Grok #3, Codex #1). - Kernel open rejects party-controlled treasuries (TreasuryIsParty): treasury == provider would collapse the slash penalty from S to A (Grok #2). - Kernel open rejects max_job_price == 0 (JobPriceCapZero): kills the A = C_disp strict-incentive degeneracy (Grok #7). - CloseKindSet::decode refuses no-exit sets (defense in depth, Grok #12). - Bond e2e uses STAKE_BOND_PROTOCOL, not the payment code (Grok #13). - preverified-seals feature doc: process-local cache, single-node dev only, never multi-validator (Grok #1 — by design, now explicit). Rejected with evidence: cross-bond artifact replay (binds checks bond_edge == public.edge_id), seal-omits-treasury (acceptance digest commits bond_terms which commits it), SealPublicInputs malleability (terms hash-checked against edge pre-verify), VoucherBook stale race (kernel payload hash binds the edge; older voucher pays provider less), stake==net-value as fee bug (plan defines S net).
georgewhewell
added a commit
that referenced
this pull request
Sep 11, 2026
Every tonic-generated server registered on the node transport is now wrapped in `ManagedServer<S, …>` with `IrohPeerExtractor`. The wrapper handles peer observation, admission, rate-limit denials, and stashing the resulting `InboundAdmission` on the request extensions so handlers that need it (just `GetKnownPeers` for now) can read disclosure_limit without re-billing the rate bucket. Removals: - `ExecutePeerInterceptor` (used to hardcode `methods::RunTicket` for every Execute/Symbolic/Opaque call — review finding #2 + #3). - In-method `peer_directory.observe_inbound_request` calls in `get_node_info` and `get_known_peers`; the wrapper does it once with the typed policy from `RpcServiceSpec::inbound_policy(path)`. - `peer_observation(&request)` and `duration_ms` helpers — the extractor does both. - `tonic::service::interceptor::InterceptedService` import. - `tonic_iroh_transport::IrohContext` and `PathId` imports — code that needs the peer id falls through the typed extractor instead. `courtesy_service` is now wrapped too, closing the inbound-admission bypass flagged in the earlier review. Every inbound RPC on every service goes through one typed admission code path. The `service_alpn`-too-long check in `get_known_peers` stays in-handler since it's an application-level invariant on the request body (not a transport-level admission concern).
georgewhewell
added a commit
that referenced
this pull request
Sep 11, 2026
mux/transport.rs (generic ws driver): prepare_credit_updates now runs BEFORE the outbound drain, matching ws/cf_do.rs. Previously credit was prepared AFTER drain, so any Credit frame stayed queued in the slot's send_queued until the NEXT inbound activity — defeating the whole point of replenishment. mux/state.rs Body inbound: drop the eager local_recv_credit refill. Credit only refills via prepare_credit_updates' threshold check, which is the only path that ALSO queues a Credit frame back to the peer. The eager refill meant local_recv_credit never dropped below threshold so the Credit frame was never queued; the peer's peer_recv_credit would drain to 0 and stall. Both fixes from codex review #2.
georgewhewell
added a commit
that referenced
this pull request
Sep 11, 2026
Closes codex review #2 finding (snapshot poisoning). VALIDATION added to deserialize_mux: - Magic byte must match (already had this). - free_mask word count must equal (N+63)/64 — mismatched N rejects with SnapshotCorrupt("free_mask word count mismatch"). - free_mask bits for the peer's parity rejected (\"sets a bit owned by peer parity\") — prevents poisoning our slot allocator into handing out peer-domain indices. - count > N rejected. - slot_idx >= N rejected. - Duplicate slot_idx in the occupied list rejected. - A slot listed as occupied AND also in the free_mask rejected (the same-slot-dispatched-twice attack). - local_recv_credit > local_credit_high_water rejected. - peer_recv_credit > config.initial_credit rejected (peer can't have advertised more credit than they could ever receive). - has_pending=1 with len > 16 KiB rejected (attachment-budget enforcement). - Trailing bytes after the snapshot rejected. ADVERSARIAL TESTS (14 new + 2 existing + 1 proptest fuzz, total 17): - drop_and_fresh_start_is_a_fresh_mux - worker_dies_mid_callback_partial_attachment (every truncation prefix from 0..len rejects) - bit_flipped_magic_byte_rejected - future_magic_byte_rejected_with_named_error (version-upgrade case) - inflated_occupancy_count_rejected (count=33 vs N=32) - duplicate_slot_index_rejected - slot_in_both_free_mask_and_occupied_list_rejected - peer_parity_in_our_free_mask_rejected (poisoned allocator attack) - credit_exceeds_high_water_rejected - pending_write_oversized_rejected (1 MiB vs 16 KiB budget) - trailing_bytes_appended_rejected - n_word_count_mismatch_rejected (snapshot for N=32 into N=128 mux) - generation_survives_round_trip - random_bytes_never_panic_or_oom (proptest fuzz, 1000 cases, all re-serializable on success) cargo test -p hellas-wire (mux,iroh,ws,ws-cf-do): 25 tests pass.
georgewhewell
added a commit
that referenced
this pull request
Sep 11, 2026
`hellas rpc <id>` was broken: it dials Node::ALPN and calls
get_node_info, but `hellas serve node` only bound Execute/Symbolic/
Opaque/Courtesy ALPNs and dropped Node connections with a warn. The
"Node is pending" comment was the only mention.
This phase wires Node end-to-end with a real impl backed by the
existing peer state, so `hellas rpc` returns actual version, build,
graffiti, uptime, OS, and known-peers fields.
Changes:
- New `crates/cli/src/commands/serve/node_handler.rs` (~150 lines).
`NodeHandlerImpl: Clone` (fields are `Arc<str>` / `Arc<[u8]>` /
Copy) so we clone per-connection rather than wrap in `Arc<dyn>`.
Implements `hellas_rpc::services::node::NodeHandler`:
- `get_node_info` returns the exact proto fields (`node_id`,
`uptime_seconds`, `version`, `build`, `os`, `graffiti`).
Version = `env!("CARGO_PKG_VERSION")`; OS = `ARCH-OS`.
- `get_known_peers` uses `PeerDirectory::ranked_known_peers`
with a 64-peer disclosure limit. Requester defaults to
`PeerId::default()` (anonymous) — the generated trait doesn't
surface inbound peer identity to handlers. Phase F's
`AdmittingDispatcher` will fix that without a trait change by
enriching the request body, or via a thread-local. Until then,
`min_disclosed_auth_level` on the directory config remains the
gate.
- `serve/node.rs`:
- Added `Node::ALPN` to the bound ALPN list.
- Construct a `PeerDirectory` keyed on the local node id; pass an
Arc to the handler (and, in Phase F, to AdmittingDispatcher).
- Stopped discarding `build` and `graffiti` (previously `let _ =
(preload_weights, build, graffiti);`); they now feed the
handler. `preload_weights` stays parked.
- `serve_connection` takes a `NodeHandlerImpl` and grew a Node
ALPN arm: `let server = NodeServer(node_handler); serve_loop(...)`.
- `serve/mod.rs` registers the new `node_handler` submodule.
Tests:
- `commands::serve::node_handler::tests::get_node_info_populates_self_identity`
— checks version, build, OS shape, graffiti round-trip, uptime ≤ 1s.
- `commands::serve::node_handler::tests::get_known_peers_returns_empty_on_fresh_directory`
— fresh directory → empty list (verifies the disclosure path
compiles and returns the proto shape).
Verification:
- `cargo test -p hellas-cli --features hellas-executor` → 58 passed
(was 56, +2 new in the new module).
- `cargo test -p hellas-wire --all-features` → 57 passed.
- `cargo test -p hellas-rpc --all-features --lib` → 68 passed.
- `cargo build --workspace` clean.
Diffstat: 3 files, +184 / -7. Net: +177 (genuinely new functionality
— restoring a missing service). Tests: +51 of that. Net production
code: +126.
georgewhewell
added a commit
that referenced
this pull request
Sep 11, 2026
… rule Adversarial panel (Codex/gpt-5.3-codex-spark, Grok, nemotron-3-ultra) reviewed 8b5cd03..HEAD. Confirmed findings fixed: - StakeBondTerms commits challenge_margin (window + inclusion + finality, blocks); JobAcceptanceContext::covered_by is the admission rule (1 <= p_j <= max_job_price, terminal_deadline + margin <= bond timeout) and FraudArtifact::binds re-checks it — closes the stall-past-timeout escape's policy gap (Grok #3, Codex #1). - Kernel open rejects party-controlled treasuries (TreasuryIsParty): treasury == provider would collapse the slash penalty from S to A (Grok #2). - Kernel open rejects max_job_price == 0 (JobPriceCapZero): kills the A = C_disp strict-incentive degeneracy (Grok #7). - CloseKindSet::decode refuses no-exit sets (defense in depth, Grok #12). - Bond e2e uses STAKE_BOND_PROTOCOL, not the payment code (Grok #13). - preverified-seals feature doc: process-local cache, single-node dev only, never multi-validator (Grok #1 — by design, now explicit). Rejected with evidence: cross-bond artifact replay (binds checks bond_edge == public.edge_id), seal-omits-treasury (acceptance digest commits bond_terms which commits it), SealPublicInputs malleability (terms hash-checked against edge pre-verify), VoucherBook stale race (kernel payload hash binds the edge; older voucher pays provider less), stake==net-value as fee bug (plan defines S net).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification