Add E2E Portals GitHub Workflow #1478
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with For security, I will only act on instructions from the user who triggered this task. New to Jules? Learn more at jules.google/docs. |
hulto
pushed a commit
that referenced
this pull request
Jan 3, 2026
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: KCarretto <Kcarretto@gmail.com>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jan 3, 2026
* initial stubbing and renaming * feat: implement tavern/portals/stream package (#1462) This commit introduces a new package `stream` in `tavern/portals` which provides utilities for handling ordered streams of `portalpb.Mote` messages. Key features: - `payloadSequencer`: Handles atomic sequence ID generation and mote creation. - `OrderedWriter`: Wraps a sender function (like a gRPC stream Send) to automatically sequence and write messages. - `OrderedReader`: Wraps a receiver function (like a gRPC stream Recv) to reorder incoming messages, handling out-of-order delivery with configurable buffering and stale stream detection. This package is designed to support both client and server sides of the portal stream. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> * refactor reader to use functional API * Implement PubSub Multiplexer (Mux) (#1466) * Implement PubSub Multiplexer (Mux) for Portals - Created `Mux` package in `tavern/internal/portals/mux`. - Implemented dual-mode operation: In-Memory (for dev) and GCP PubSub (for prod). - Implemented `CreatePortal` and `OpenPortal` lifecycle methods with resource provisioning. - Implemented `Publish` and `Subscribe` logic with local broadcasting (fast path) and global PubSub (slow path). - Added `HistoryBuffer` for message replay. - Added intelligent topic caching to handle `mempubsub` quirks and improve performance. - Added Prometheus metrics for observability. - Verified with comprehensive unit tests using `enttest` and `mempubsub`. * Implement PubSub Multiplexer (Mux) for Portals - **Mux Package:** Created `tavern/internal/portals/mux` to route messages between local streams and global PubSub. - **Dual-Mode Operation:** Implemented support for In-Memory (dev) and GCP PubSub (prod) drivers via `gocloud.dev/pubsub`. - **Portal Lifecycle:** Added `CreatePortal` and `OpenPortal` methods managing resource provisioning, database records, and subscription lifecycles. - **Message Routing:** Implemented `Publish` (fast-path local dispatch, slow-path global send) and `Subscribe` (local channel registration). - **History Management:** Added `HistoryBuffer` in `history.go` for message replay to new subscribers. - **Concurrency & Robustness:** - Handled race conditions in shared subscription creation (`OpenPortal`). - Managed `receiveLoop` lifecycle using stored `cancelFuncs`. - Added `AlreadyExists` error handling for resource creation. - Used `sync.RWMutex` for thread safety (`activeSubs`, `subscribers`, `histMu`). - **Observability:** Added Prometheus metrics for message counts. - **Testing:** Comprehensive unit tests covering In-Memory flow, Portal creation, and Portal opening. * Refactor Mux and Address PR Feedback - **Structure:** Refactored `Mux` to use composed structs (`SubscriptionManager`, `SubscriberRegistry`, `TopicManager`, `HistoryManager`) for clearer locking semantics. - **Safety:** - Handled race conditions in `CreatePortal` and `OpenPortal` by re-checking state after lock acquisition. - Updated `teardown` logic to use `client.Portal.UpdateOneID` for reliability. - Standardized on `context.Background()` for shutdown operations to prevent context leaks. - **Features:** - Added `WithSubscriberBufferSize` to configure channel buffers. - Added `WithHistoryReplay` option to `Subscribe` for optional history. - Added `mux_messages_dropped_total` metric. - **Concurrency:** Moved global lock handling into granular manager structs to reduce contention. - **Correctness:** Fixed `CreatePortal` to use task-based lookup for dependencies and removed invalid `portalID` parameter usage in logic (though kept signature for now as per instructions). Tests passed. * Implement PubSub Multiplexer (Mux) for Portals - **Mux Package:** Created `tavern/internal/portals/mux` to route messages between local streams and global PubSub. - **Dual-Mode Operation:** Implemented support for In-Memory (dev) and GCP PubSub (prod) drivers via `gocloud.dev/pubsub`. - **Portal Lifecycle:** Added `CreatePortal` and `OpenPortal` methods managing resource provisioning, database records, and subscription lifecycles. - **Message Routing:** Implemented `Publish` (fast-path local dispatch, slow-path global send) and `Subscribe` (local channel registration). - **History Management:** Added `HistoryBuffer` in `history.go` for message replay to new subscribers. - **Concurrency & Robustness:** - Handled race conditions in shared subscription creation (`OpenPortal`). - Managed `receiveLoop` lifecycle using stored `cancelFuncs`. - Added `AlreadyExists` error handling for resource creation. - Used composed structs (`SubscriptionManager`, `SubscriberRegistry`) for granular locking. - **Observability:** Added Prometheus metrics for message counts and dropped messages. - **Testing:** Comprehensive unit tests covering In-Memory flow, Portal creation, Portal opening, and Benchmarks. --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> * both grpc endpoints * stub out portal-stream package * Implement portal-stream crate (#1470) * Implement portal-stream crate with sequencer, reader, and writer logic - Added `implants/lib/portals/portal-stream` crate. - Implemented `PayloadSequencer` for atomic sequence ID generation. - Implemented `OrderedReader` for reordering incoming messages with timeout and buffer handling. - Implemented `OrderedWriter` for sequencing outgoing messages. - Added comprehensive unit tests for all components. - Added crate to `implants` workspace. * Switch to anyhow for error handling in portal-stream - Replaced `thiserror` with `anyhow` in `portal-stream`. - Updated `Cargo.toml` to use `anyhow` from workspace. - Updated `reader.rs` and tests to use `anyhow::Result` and `anyhow!`. --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> * Implement SOCKS5 Proxy using tavern/portals/stream (#1467) * Implement SOCKS5 proxy with gRPC tunneling - Added bin/socks5/proxy.go implementing a SOCKS5 proxy server. - Implemented tunneling over stream.OrderedWriter/Reader. - Supported TCP CONNECT and UDP ASSOCIATE commands. - Implemented robust lifecycle management and cleanup. - Added benchmarks in bin/socks5/proxy_test.go demonstrating high throughput. * Address PR comments: Refactor writes, defaults, and shutdown tracking - Refactored raw `conn.Write` calls into named helper functions. - Changed default upstream port to 8000. - Added `sync.WaitGroup` to track connection lifecycle. - Added logging for dropped motes in dispatcher. - Defined `maxStreamBufferedMessages` constant. --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> * Implement Portal Infrastructure in ImixV2 (#1471) * Implement portal infrastructure in imixv2 - Added `portal-stream` dependency to `imixv2`. - Updated `Transport` trait to include `create_portal` (async). - Implemented `create_portal` in `grpc` transport. - Updated `Agent` trait to include `create_portal`. - Created `imixv2/src/portal/` module with TCP, UDP, and Bytes support using `portal-stream`. - Implemented `create_portal` in `ImixAgent`. - Exposed `create_portal` via `eldritch-libpivot`. - Updated `portal-stream` to support async writers. * Implement portal infrastructure in imixv2 - Added `portal-stream` dependency to `imixv2`. - Updated `Transport` trait to include `create_portal` (async). - Implemented `create_portal` in `grpc` transport. - Updated `Agent` trait to include `create_portal`. - Created `imixv2/src/portal/` module with TCP, UDP, and Bytes support using `portal-stream`. - Implemented `create_portal` in `ImixAgent`. - Exposed `create_portal` via `eldritch-libpivot`. - Updated `portal-stream` to support async writers. - Updated `run_create_portal` to send initial registration message. --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> * some changes * Add integration test * Added TRACE Bytes kind * added trace protos * Implement Trace Motes for Portals (#1473) * feat: implement end-to-end trace motes Implements application-level tracing for Portals infrastructure using the new `tracepb` definitions. * **CLI (`bin/socks5`)**: * Added `trace` subcommand to generate trace motes, send them to the server, and print a latency report. * Refactored `proxy.go` to support subcommands. * Added `addTraceEvent` helper for modifying trace motes. * **Server (`tavern`)**: * Instrumented `api_open_portal.go` and `api_create_portal.go` to inject trace events at key checkpoints (Recv, Pub, Sub, Send). * Created `trace_helper.go` to share event injection logic. * **Agent (`imixv2`)**: * Updated `run.rs` to intercept `BYTES_PAYLOAD_KIND_TRACE` motes. * Implemented logic to add `AGENT_RECV` and `AGENT_SEND` events and immediately echo the mote back. * added retry to trace --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: KCarretto <Kcarretto@gmail.com> * update flag to --portal * update buf size * Update portal API to send keepalive motes (#1472) - Added keepalive ticker to sendPortalInput loop in api_create_portal.go - Sends a BYTES_PAYLOAD_KIND_KEEPALIVE mote at regular intervals - Prevents connection timeouts similar to the reverse shell implementation Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: KCarretto <Kcarretto@gmail.com> * add tokio-console support for imixv2 * Fix SOCKS5 proxy cold start hang by avoiding BiLock on TcpStream (#1476) Replaced `tokio::io::split(stream)` with `stream.into_split()` in `implants/imixv2/src/portal/tcp.rs`. The former uses a `BiLock` which can cause deadlocks when the read and write halves are accessed concurrently in separate tasks, specifically causing the "cold start" hang where the initial payload might be blocked. `into_split()` returns owned halves that operate independently. Added a regression test `implants/imixv2/src/tests/repro_issue.rs` to verify the fix. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: KCarretto <Kcarretto@gmail.com> * socks proxy sends registration message now * socks5 trace must send registration message * Add E2E Portals Workflow and Playwright Test (#1478) Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: KCarretto <Kcarretto@gmail.com> * cargo fmt * remove spammy log line * fix some tests * add create_portal fake impl * revert some changes to grpc.rs * remove grpc explicit sizes * update socks5 proxy to support auth and portals to support gcp pubsub * oops * minor cleanup * fix tests * Fix e2e workflow * use env var for auth * wait for socks to start before continuing * Add benchmark tests for cryptocodec (#1485) Added a new test file `tavern/internal/cryptocodec/cryptocodec_bench_test.go` to measure the throughput of `Encrypt` and `Decrypt` methods in `CryptoSvc`. Includes benchmarks for both encryption and decryption operations using standard payload sizes. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> * Fix race condition in TestPortalIntegration (#1487) The `TestPortalIntegration` test was flaky and often hung because the Agent would publish messages to the portal before the User had successfully subscribed to the output topic. This resulted in messages being dropped (as evidenced by "message sent to topic with no subscribers" warnings from the in-memory pubsub) and the User reader blocking indefinitely. This commit replaces the arbitrary `time.Sleep` with a deterministic synchronization mechanism. The User now sends a "ping" message to the Agent immediately after opening the portal. The Agent waits to receive this ping before proceeding. This ensures that the User's portal connection (and thus the underlying pubsub subscription) is fully established before the Agent attempts to send any data. This reduces test execution time and eliminates the race condition. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> * fix proxy upstream address parsing * minor fixes & cleanup * fix collapsible if * cargo fmt * fix portal workflow * Add Tavern User Guide for Authentication and API Token (#1501) * Add documentation for TAVERN_API_TOKEN and portal auth flow This commit adds a new user guide page for Tavern (`docs/_docs/user-guide/tavern.md`) detailing the purpose of `TAVERN_API_TOKEN`. It clarifies the distinction between this token and the web OAuth token and explains the "portal auth flow" for users SSH'd into remote environments (e.g., Kali VMs) where standard auth port forwarding is not feasible. * Update TAVERN_API_TOKEN docs: remove non-existent Portal Auth Flow Per code review feedback, the "Portal Auth Flow" feature does not exist yet. This commit removes that section from the documentation, leaving the explanation of what the token is and when to use it (SSH scenarios). --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> * cleanup * Merge origin/main, resolve conflicts, and fix go generate (#1515) - Merged changes from origin/main, resolving conflicts in app.go, server.go, and mock transport. - Updated `golang.org/x/tools` to fix `ent` generation failure ("context without types"). - Re-ran `go generate ./...` to update generated protobuf and ent code. - Fixed compilation errors in tests due to renamed protobuf enum constants (ActiveTransport_TRANSPORT_HTTP1). Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> * Add offline filters (#1505) * feat(online-offline): Online Offline in progress * fix(online-offline): Change text * fix(query): Simplify query * fix(build): Build UI * fix(create-quest): Hide filter from create quest * Add Link entity and update CDN to use link-based file access (#1444) * Add Link entity and update CDN to use link-based file access This commit implements a new Link entity system for the CDN: - Add Link entity schema with: - path (unique, default UUIDv4) - active_before (timestamp, default epoch 0) - active_clicks (int, default 0) - edge to File entity (one-to-many relationship) - Update CDN upload handler to: - Create a new Link entity for each uploaded file - Return both file ID and link path in response - Add new CDN link download handler to: - Serve files using Link entity path instead of file name - Check active_clicks and active_before before serving - Decrement active_clicks when file is served - Return 404 if link is not active - Replace CDN route to use link-based downloads: - Changes /cdn/ endpoint from file name access to link path access - Removes ability to serve tome assets via direct file name - Maintains FetchAsset gRPC API for agent communication - Update dependencies: - Upgrade entgo.io/ent from v0.14.1 to v0.14.5 - Update related dependencies (atlas, sqlite3, tools) * go generate * Add mutations * Clarify docs * typeable random short string * Don't create link on upload * Resolve feedback --------- Co-authored-by: Claude <noreply@anthropic.com> --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: Squidli <32434695+cmp5987@users.noreply.github.com> Co-authored-by: Hulto <7121375+hulto@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a new GitHub Actions workflow
.github/workflows/e2e-portals.ymland a corresponding Playwright testtests/e2e/tests/portal.spec.tsto perform end-to-end testing of the portal infrastructure (SOCKS5 relay).The workflow performs the following steps:
tavern(server),imixv2(agent), andbin/socks5(proxy).tavernandimix).Changes included:
.github/workflows/e2e-portals.ymltests/e2e/tests/portal.spec.tsPR created automatically by Jules for task 319589481741948535 started by @KCarretto