Add owner-bound cloud workers and remote Fetch - #29
Draft
georgewhewell wants to merge 16 commits into
Draft
georgewhewell wants to merge 16 commits into
georgewhewell wants to merge 16 commits into
Conversation
georgewhewell
changed the title
Hydra: passedHead All 39 builds passed. |
georgewhewell
added a commit
that referenced
this pull request
Sep 28, 2026
Extend Fetch to signed HTTPS exchanges and let native API clients use standard HTTP endpoints backed by funded work channels. The gateway selects an account from the requested model, preserves the upstream response, and completes payment through the existing channel protocol. ### Behavior - Add a shared Fetch work profile alongside Evaluate, with SDK sessions, provider admission, authenticated streaming and restart recovery. The reusable SDK paid pool serves both the CLI and Gate; client and provider features are separate, with typed configuration and errors. The provider invokes upstream only after accepting the signed job against its funded channel, credit and route policy. The HTTP gateway requires a paid pool; SDK Fetch providers configured for paid work advertise only Work and WorkSetup. - Carry HTTP methods, ordered queries, headers, encoded bodies, SSE and non-2xx responses through Fetch. Restrict provider-owned credentials by origin, path and method. Reuse upstream HTTPS and authenticated Work/WorkSetup connections, preserve streaming backpressure, and keep paid RPCs open through slow responses and idle connections. Accepted jobs queue under provider capacity pressure instead of becoming permanent failures. - Start transparent HTTP through dedicated `HttpGatewayOptions` with a required paid backend; tokenizer, inference-cache and Evaluate settings are absent. Route standard API paths by model across configured accounts. Pin existing sessions to their account, share capacity and cooldown for credential aliases, and direct new sessions toward available backends. - Archive gateway exchanges by default, with per-request or gateway-wide ZDR opt-out. Archive failures report telemetry while serving continues. Fetch payment journals retain accounting, hashes and signatures without payloads. Recovery pays verified deliveries from retained evidence, retransmits certificates idempotently, and skips lost payloads without re-executing them. Compressed requests receive the same ZDR retention checks as uncompressed requests. - Keep validator reads in independent channel observers. Provider connections verify the configured genesis before advancing payment state. Short journal locks order closes and new signatures; stale observations stop new exposure, while retained certificates remain retransmittable. Established-channel requests perform no consensus RPCs. ### Validation At `d9407a2d`, all local gates invoked by `nix run .#check` have passed across the aggregate runs and resumed checks: workspace and feature-specific strict Clippy, formatting and dependency checks, kernel/model, validator, RPC/work/client, SDK, HTTP/provider/storage, and WebAssembly builds. The SDK suite has 33 tests, the gateway 66, and the provider 70; the SDK client-only, provider-only and paid-gateway feature sets are also linted independently. The aggregate runs were not uninterrupted successes: one existing gateway local-control request timed out, and two existing filesystem tests exceeded their two-second limits. Their complete suites passed unchanged on rerun. The final checks also caught and fixed provider dependency ordering. A stale read-only zstd header in the local WebAssembly build cache needed its write permission restored; this required no source change. Gate's integration passed `nix develop --command make check` (bindings, TypeScript, frontend build, formatting, strict Clippy and 19 Rust tests); the final form edits also passed `make ui-build`. See hellas-ai/gate#586. These checks ran on Linux; production Apple enrollment was not exercised. `420f5e7e` fixes CLI tests that pinned superseded file-error wording and adds `check-cli` to the aggregate local gate. `nix run .#check-cli` passes all 179 tests, including the five that failed in the earlier hosted builds; formatting and Nix lint checks also pass. Hosted CI for the updated commit is queued: https://github.com/hellas-ai/hellas/actions/runs/36457595760. Regression coverage includes slow and idle paid streams, restart between verified delivery and payment, duplicate payment recovery, payload-free journals, paid-only protocol negotiation, genesis mismatch, bounded queues under saturation, observer stalls, and close/payment ordering. New coverage exercises the public HTTP entry point over a real socket, credential injection over local TLS, authenticated connection reuse and reconnect checks, and startup recovery without accidentally funding from counter files alone. CI explicitly runs the SDK feature combinations and HTTP/provider/storage suites. Earlier live devnet passes through Kimi Code, Codex and Claude Code completed their tool loops. Each pass produced six HTTP 200 exchanges and six units of acknowledged payment. Exported token counters matched response usage; all twelve established request traces contained zero validator RPCs. These were smoke tests, not latency benchmarks or final on-chain settlement tests. ### Review boundaries Paid Fetch changes Work/WorkSetup wire descriptors and the StreamResult schema. Gateways and providers must upgrade together; the on-chain payment certificate format is unchanged. Jobs within one funded channel finish in order. Generic HTTP streams remain open through EOF. WebSocket upgrades and streaming uploads are unsupported. ZDR controls application payload persistence; it does not attest OS swap/dump handling or upstream retention. Explicitly configured Courtesy HTTP routes still have no token-based spend accounting; paid channels charge their agreed fixed job price. This targets `master` independently of #29 (Runpod worker provisioning). No stacking is required. Configuration: [HTTP gateway](https://github.com/hellas-ai/hellas/blob/codex/secure-fetch/docs/http-gateway.md), [paid gateway](https://github.com/hellas-ai/hellas/blob/codex/secure-fetch/docs/paid-gateway.md), [HTTPS provider](https://github.com/hellas-ai/hellas/blob/codex/secure-fetch/crates/providers/HTTPS.md).
Resolve cloud and bare-metal inventory from Fetch, LLM and gateway. Validate and install route configuration and credentials over owner-authenticated iroh and internal RPC, preserving refreshed credentials across restarts. Publish static and WASM-target build archives. Accommodate observed Codex SSE headers and explicitly discard new service metadata.
Allow the initial empty argument string and discard delta obfuscation metadata. Keep final JSON validation, declared-tool authorization and delta consistency checks. Cover the observed SSE lifecycle and malformed or inconsistent terminal arguments.
Retain generic machine selection and private configuration delivery. Keep legacy Codex compatibility experiments outside the cloud PR; acceptance will compose this branch with the generic HTTPS Fetch implementation.
georgewhewell
force-pushed
the
codex/cloud-workers
branch
from
September 28, 2026 22:11
d8203a2 to
1a04508
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deployed workers need a durable association with the Hellas identity that launched them. This adds Runpod provisioning, an inventory of owned cloud and bare-metal machines, and owner-authenticated administration and remote machine selection to the main CLI.
hellas cloud runpod --account NAME list|info|create|destroysupports environment and credential-command profiles. Creation resolves a digest-pinned template, preserves creator attribution, and verifies the allocated template and image.--interruptiblerequests spot capacity; on-demand is the default. Dry runs are offline. Missing account setup displays the Foundation referral link in CLI and internal-RPC errors.hellas fetch|llm|gateway --machine NAMEresolves an enrolled cloud worker or prepared bare-metal machine. Packaged Unix lean and Linux static CLIs include cloud management. The HTTP gateway's paid-pool mode requires--paid-work-configand rejects--machine; inventory selection does not create payment channels.hellas-agentbinds to one owner and refuses reassignment. Administration uses authenticated iroh;serve --ownerrestricts incoming Hellas RPC.machines configuredelivers routes and credentials over that channel, validates before restarting, and preserves refreshed credentials across ordinary restarts. GUI/automation clients use the same operations through private Unix JSON-RPC. Inventory and receipts exclude upstream secrets.Managed images keep identity/content on the persistent volume and configuration/credentials on the container disk at
/var/lib/hellas-private. Tested Runpod volumes ignored Unix directory permissions. After a provider stop/reset clears container storage, the owner must reinstall credentials; the persistent identity remains.Rebased onto master
ced53481after #30 merged generic HTTPS. The PR uses that upstream implementation and retains both its CLI checks and the cloud integration checks. Local validation passes: 22 cloud tests, 185 CLI tests, the HTTP/machine argument check with and withoutnode, Clippy for cloud and the relevant CLI feature sets, Rust/Nix formatting, and the explicit two-identity iroh integration covering ownership, private configuration, Fetch, gateway, and cache administration. Fresh PR CI is running.Historical evidence: pre-rebase CI passed all checks and published six images. The older HTTPS acceptance revision and its CI produced the matched CLI/image used for nine live L4 checks, including template attribution, enrollment, private configuration, verified Fetch, gateway SSE, tool continuation, upstream errors, rejected WebSocket upgrade, and confirmed deletion. Those tests exercised the older direct HTTP gateway, not the newly merged paid gateway. Paid-pool cloud acceptance has not been run; the historical harness now rejects incompatible CLIs before allocation. The Foundation template still pins that tested historical image. A prior spot request returned HTTP 500 without allocation, so live spot capacity remains unconfirmed.
Ancillary publication, acceptance tooling, and documentation remain in the extras repository. Current limits: Unix management, software trust rather than measured boot, one owner without delegation/rotation, buffered HTTP request bodies, and no WebSocket upgrades. The management socket is separate from cache-control RPC.