Add paid HTTPS Fetch and HTTP gateway account routing - #30
Merged
Merged
Conversation
Share channel setup, recovery, verified delivery and payment between SDK hosts, the CLI and the paid gateway. Keep Evaluate streaming and chain recovery while adding bounded caller-signed HTTPS Fetch with per-connection App Attest checks. Fetch providers journal accounting metadata without persisting request or result bodies, and never repeat an invocation after losing its in-memory payload. Validation: 1,029 passing tests across CLI, client, executor, providers, RPC, SDK, work journals and chain settlement; Fetch-only SDK check and formatting pass. Real Apple enrollment requires a provisioned signed host and was not exercised.
Share provisioning logic between frontends, remove redundant session and offer state, and replace the one-method backend adapter with a callback. Remove the unused Evaluate compatibility alias and require explicit assurance when verifying Fetch results. Keep Gate's one-job client entry point. Condense historical commentary and document payload-digest behavior precisely. Canonical records and wire encodings are unchanged. Validation: 691 tests passed across CLI, SDK, RPC and work; formatting and diff checks pass. The cleanup removes 637 net lines.
Centralize account admission and session binding, normalize backend setup, and represent conflicting continuation IDs explicitly. Snapshot account load before sorting while keeping first-session assignment atomic. Remove duplicate stored configuration, connection-cache lookup logic, metric cloning and paid-provider RPC wrappers. Preserve opaque routes, strict session affinity, byte forwarding and best-effort archives. Validation: 334 Rust tests; 63 gateway tests without telemetry; 149 HTTP comparisons; 20 two-provider routing checks; native Kimi, Codex and Claude tool workflows against owned fixtures. Strict clippy, format, CLI check and release build passed.
Remove synchronous validator refreshes from admission and streaming. Give each channel an independent observer and share journal state only across short synchronous operations, preserving payment ordering and restart recovery. Expire local readiness when finalized progress stalls, bound reconnects, and keep retained certificates retransmittable. Replace observation and work-config errors with typed causes and remove the competing gateway follower. Validated with 709 passing tests, strict Clippy, formatting, and two live Kimi/Codex/Claude tool-loop passes. Established request traces contain no validator RPCs.
Hydra: passedHead All 39 builds passed. |
Keep authoritative peer error trailers when an early rejection closes the request writer. Bound recovery and cover both unary and server-streaming calls. Refresh the Work and StreamResult wire-ID pins for the existing Fetch terminal schema, document coordinated upgrades, and apply dependency ordering and TOML formatting checks.
georgewhewell
marked this pull request as ready for review
September 26, 2026 04:16
The rewritten serve loop wrapped every RPC dispatch in a 30-second wall-clock timeout and dropped the failure logging the loop used to have. StreamResult drives the whole response stream inside dispatch and legitimately produces no frame for minutes while a job executes, so any paid job slower than 30 seconds had its stream cancelled mid-delivery and the connection torn down; the client refuses to retry once a prefix was emitted. Streaming methods already enforce their own application-level deadlines, so serve goes back to waiting without a total deadline and to logging dispatch and transport failures.
Two validation gaps let bad configuration past startup checks. The retained-transcript-capacity and route cross-checks ran only after Executor::spawn_configured had already created quota and transcript stores, so a config error aborted startup with the side effects committed; they now run before the spawn, and the Fetch-policy check the match already performs is no longer repeated. And an empty validator list reached connect_chain, which reduced a candidate index modulo zero inside the clock task: WorkRunner::discover now rejects it as configuration, the rotation counter is per runner rather than a process-global static, and ConsensusVerifier is built from the normalised network id the channel binds.
Client Fetch journals are metadata-only, so after a restart every open Fetch job has an empty prepared input, and the recovery scan aborted with MissingPayload for all of them. For a job whose result was delivered, verified and committed before the crash that wedged two ways: the provider, which did the work, was never paid even though the retained result and authorization are everything signing payment needs, and needs_recovery stayed set so every later request failed again until the payment deadline filtered the job. Recovery now pays delivered jobs straight from their retained evidence, logs and skips jobs that can neither execute nor settle, and no longer aborts the whole pass on one unrecoverable job. Adjacent recovery fixes: a stream recovery with no progress callback marks nothing emitted, so retryable delivery failures actually retry instead of replaying prefixes into a user stream that does not exist; the unary collect loop checks the payment window between retries like the stream path does; declining recovery no longer disarms the flag; settle documents its missing deadline; the dead MissingPayload and NoValidators error variants go away.
The ZDR store:true rejection parsed the raw wire bytes as JSON, so a gzip or zstd request body sailed past it and reached the upstream with retention enabled, exactly what an operator-enforced --zdr promises to forbid. The routing body decoder now lives in one shared helper, the archive middleware checks the decoded bytes and rejects bodies it cannot inspect under ZDR, and a regression test pins the compressed store:true refusal before anything is forwarded or archived.
Four small correctness fixes. An empty body event no longer kills an otherwise valid paid response as a bogus size-limit error; zero-byte chunks are skipped and only real over-limit growth fails. A non-UTF-8 Connection request header is parsed lossily instead of dropping every token it named, so the headers it marks hop-by-hop stay stripped. Routing failures now attribute the pinned backend with the affinity that pinned it (continuation, connection or session) instead of a hardcoded session label. And --metrics-port warns in HTTP Fetch mode, where the Prometheus endpoint is not served, instead of going dark.
The provider relayed upstream connection, transfer-encoding, content-length and keep-alive headers into the signed Adaptor.Http.Head event, so the evidence could carry a content-length contradicting the body its events actually hold. The signed head now carries end-to-end headers only; the body bytes are the single length truth. HTTPS.md gains the matching note, plus two operator clarifications from review: credential path scopes do not cover query strings, and successful courtesy http requests settle zero billable units, so courtesy spend quotas never accumulate on http routes (paid channels, which charge the fixed price, are unaffected).
Reconstruction now refuses an interim 1xx as the single signed head, since only a final status completes a paid response, and the reserved request-header list covers the remaining RFC 9110 hop-by-hop names. Admission rejects an Open Fetch response cap the channel can never deliver: base64 body events carry at most three raw bytes per four payload bytes, so a max_response_bytes beyond three quarters of the policy's output budget would run the provider's upstream fetch and refuse the terminal at payment. The Open Fetch host allowlist is normalised to the lowercase punycode form admission compares against, so a case- or space-spelled entry can no longer brick a mounted policy. The close descriptor's version 2 gains a named constant, and the code generator's open-method default handler gains the comment explaining why a name match is the rule.
The stream replay path commits a PlaintextReleased record per emitted frame, and every one after the first is redundant: nothing is written, but commit still cloned the whole channel state, transcript included, before apply could say so. At the documented 32 MiB spool that is gigabytes of memcpy per delivered response. A cheap exact redundancy check, mirroring apply_plaintext's own arms and falling through to the full apply whenever in doubt, now answers on the current state first. The same change documents why commit still validates bodies in full before they are stripped, why record_result reads the last fresh readiness rather than gating on observation, and why an input-less job answers Indeterminate. suspend no longer converts an observer-less endpoint to never-admitting, the README stops claiming client Fetch journals may retain payloads (mount is metadata-only on both endpoints), and the paid gateway doc notes the observation-age versus block-interval coupling.
The paid fetch admission path had no tests at all: none of prepare_paid_fetch's refusal branches, not the capacity accounting, not the progress wiring, sitting directly on the paid execution seam. Ten tests now drive the production path through the actor, from the tampered transcript and contract mismatches (retention, assurance, environment, route, capability) to the fail-fast capacity refusal and a happy path whose signed output verifies end to end. The seam also gets three fixes its review exposed: the PaidFetch completion runs the deferred quota retries like fetch completion does, so courtesy deferrals cannot linger until restart; the upstream/projection failure keeps its position and cause in the operator log while the peer still receives the sanitized message; and start_paid_fetch documents that a capacity refusal is terminal for the journaled job, not queued, so operators size the limit for bursts. PreparedFetchInput gains the public constructor the tests (and any out-of-crate backend) need. (PreparedFetchInput::new lives in hellas-work and is committed with the work changes.)
The CLI kept speaking names that no longer exist in the SDK (OpenPaidChannel, PaidOutput, check_policy_input) through import aliases, and carried two mid-file use statements where deleted code used to be. Call sites now use the SDK names directly. The --http-fetch-config read takes the bounded, path-labelled loader the pool file already used, and an empty provider candidate list is a startup error instead of a panic.
Contributor
Author
Review takeover (Kimi Code, requested by maintainer)Eight slice reviews over the full diff (gateway, providers, rpc, sdk, work, cli, executor+wire, cross-cutting payment-safety), plus local verification of everything CI can check. Verdict: not merge-safe at Blocker found and fixed (
|
georgewhewell
marked this pull request as draft
September 27, 2026 19:12
Separate paid client/provider features and type provider and provisioning errors. Give HTTP gateways dedicated paid options, reuse authenticated Work connections and verified HTTP output, and exercise credential injection over local TLS. Share bounded configuration reads and add the HTTP/storage and SDK feature CI gates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extend Fetch to signed HTTPS exchanges and let native API clients use standard HTTP endpoints backed by funded work channels. The gateway selects an account from the requested model, preserves the upstream response, and completes payment through the existing channel protocol.
Behavior
HttpGatewayOptionswith a required paid backend; tokenizer, inference-cache and Evaluate settings are absent. Route standard API paths by model across configured accounts. Pin existing sessions to their account, share capacity and cooldown for credential aliases, and direct new sessions toward available backends.Validation
At
d9407a2d, all local gates invoked bynix run .#checkhave passed across the aggregate runs and resumed checks: workspace and feature-specific strict Clippy, formatting and dependency checks, kernel/model, validator, RPC/work/client, SDK, HTTP/provider/storage, and WebAssembly builds. The SDK suite has 33 tests, the gateway 66, and the provider 70; the SDK client-only, provider-only and paid-gateway feature sets are also linted independently.The aggregate runs were not uninterrupted successes: one existing gateway local-control request timed out, and two existing filesystem tests exceeded their two-second limits. Their complete suites passed unchanged on rerun. The final checks also caught and fixed provider dependency ordering. A stale read-only zstd header in the local WebAssembly build cache needed its write permission restored; this required no source change.
Gate's integration passed
nix develop --command make check(bindings, TypeScript, frontend build, formatting, strict Clippy and 19 Rust tests); the final form edits also passedmake ui-build. See hellas-ai/gate#586. These checks ran on Linux; production Apple enrollment was not exercised.420f5e7efixes CLI tests that pinned superseded file-error wording and addscheck-clito the aggregate local gate.nix run .#check-clipasses all 179 tests, including the five that failed in the earlier hosted builds; formatting and Nix lint checks also pass.Hosted CI for the updated commit is queued: https://github.com/hellas-ai/hellas/actions/runs/36457595760.
Regression coverage includes slow and idle paid streams, restart between verified delivery and payment, duplicate payment recovery, payload-free journals, paid-only protocol negotiation, genesis mismatch, bounded queues under saturation, observer stalls, and close/payment ordering. New coverage exercises the public HTTP entry point over a real socket, credential injection over local TLS, authenticated connection reuse and reconnect checks, and startup recovery without accidentally funding from counter files alone. CI explicitly runs the SDK feature combinations and HTTP/provider/storage suites.
Earlier live devnet passes through Kimi Code, Codex and Claude Code completed their tool loops. Each pass produced six HTTP 200 exchanges and six units of acknowledged payment. Exported token counters matched response usage; all twelve established request traces contained zero validator RPCs. These were smoke tests, not latency benchmarks or final on-chain settlement tests.
Review boundaries
Paid Fetch changes Work/WorkSetup wire descriptors and the StreamResult schema. Gateways and providers must upgrade together; the on-chain payment certificate format is unchanged.
Jobs within one funded channel finish in order. Generic HTTP streams remain open through EOF. WebSocket upgrades and streaming uploads are unsupported. ZDR controls application payload persistence; it does not attest OS swap/dump handling or upstream retention. Explicitly configured Courtesy HTTP routes still have no token-based spend accounting; paid channels charge their agreed fixed job price.
This targets
masterindependently of #29 (Runpod worker provisioning). No stacking is required.Configuration: HTTP gateway, paid gateway, HTTPS provider.