Skip to content

feat(replay): deliver optional execution with real routing consumers - #4537

Open
thymikee wants to merge 1 commit into
t3/replay-execution-pluginfrom
t3/replay-plugin-execution-routing
Open

thymikee wants to merge 1 commit into
t3/replay-execution-pluginfrom
t3/replay-plugin-execution-routing

Conversation

@thymikee

@thymikee thymikee commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Summary

Route optional replay/test execution through the plugin for #3871, preserving typed admission and lightweight installation disclosure.

Unpublished candidate: 0f8af82402c5888d519d2fd0055f6bd1c2451de8. 74 files / 1,884 gross lines above REG; 360 / 10,741 against default-main merge-base.

Validation

Independent Codex recheck found no actionable findings within the R1 repair and preservation interactions. The two-file repair survives all four rebases; eight range-diff entries match.

At 0f8af82402c5888d519d2fd0055f6bd1c2451de8, pinned frozen install/build and full-repository format passed. Exact affected: 42 passed / 25 skipped, exactCommit=true, checkoutChanged=false; receipt b9a5914cf6f0ad62b03b8d625d73c7e7ca63244135c2a4b75deab89033effc8f. Full: 41 passed / 25 skipped; receipt d97914929e672223a48821374649a841571a99e69e62ad6b33e9c3749e186ab4.

All seven new-head canonical invocations passed once, serially, under one frozen ENV. Counts overlap; skips retain their CI/manual owners. Existing refs stayed unchanged; source/logical index matched before and after each gate. Physical-index captures remain separate, with within-head byte-change causes unproven. Initial/final snapshots cannot exclude transient edits.

The candidate remains unpublished; published head ce66416f is unchanged and new-head hosted checks are unverified. Node22.13 packaging and earlier ENV/ref owner dispositions still block source publication. Both plugins remain private pending maintainer placeholders, trusted publishers and usable publication. Native ADR, live/current-client, performance and security remain open. No merge clearance.

@thymikee
thymikee added this pull request to stack #4529 November 14, 2026 00:00
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.11 MB 6.09 MB -29.1 kB
Package (unpacked) 6.11 MB 6.08 MB -29.1 kB
Package (download) 1.85 MB 1.85 MB -8.7 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 20.0 ms 20.6 ms +0.6 ms
CLI --help 42.3 ms 43.2 ms +1.0 ms

@thymikee
thymikee force-pushed the t3/replay-plugin-execution-routing branch from 33d14c8 to cf03d03 Compare November 14, 2026 00:06
@thymikee
thymikee force-pushed the t3/replay-plugin-execution-routing branch from cf03d03 to 7a799d9 Compare November 14, 2026 00:45
@thymikee thymikee changed the title feat(replay): extract native replay and test execution from core feat(replay): deliver optional execution with real routing consumers Nov 14, 2026
@thymikee
thymikee force-pushed the t3/replay-plugin-execution-routing branch from 7a799d9 to 08ce68f Compare November 14, 2026 01:05
@thymikee

Copy link
Copy Markdown
Member Author

I reviewed 08ce68f. The code has two problems to fix before merge. CI is green at this head: the 12 non-passing jobs in the first run were cancelled, and the same jobs pass in the re-runs. Smoke Tests and the Mutants matrix were skipped, so they give no device signal. There are no conflicts.

The Maestro help still tells agents to install only @agent-device/maestro and then run test <path> --maestro (replay-backend-declarations.ts#L56). Every test request now resolves native execution first (replay-plugin-admission.ts#L12-L13), so that run needs @agent-device/replay on the daemon host too. An agent that follows help maestro gets plugin-not-installed for a package the help never named, and has to install and restart the daemon again. The README states the two-plugin requirement, but the help does not. Please make the Maestro help say that test suites also need @agent-device/replay. Better, build that text from the ad declaration's installationHint, so one declaration owns the suite requirement for both the help and the error.

Native .ad step dispatch and suite scheduling now run engine code that the registry loads from the plugin (native-command.ts#L91). The only installed-plugin run is packed-runtime.mjs on an empty script (replayed 0) with no device, and the PR body lists the live Android, iOS, desktop and remote runs as pending. So nothing has shown a device-facing route (dispatch, session retention, repair, divergence, suite artifacts) working through the plugin engine. The move could break the identity of host-owned callbacks or cancellation without any check noticing. Please install @agent-device/replay with plugins add, restart the daemon, and run a real .ad script that taps or fills and then closes, plus a two-test suite with --retries, on Android and iOS. The replay result should show replayed above 0, the suite JSON should show the passed counts and artifactsDir, and plugins list should show the package active. Please also run the same replay with the plugin absent and show plugin-not-installed with no session created. Desktop and remote runs are needed only if the PR claims them.

Not blocking, take or leave: packages/replay/src/plugin.ts:10 hard-codes the exact version 0.21.38-dev, so the next root bump (for example 0.21.39-dev, which plugin.test.ts:67 itself rejects) makes setup throw incompatible_plugin in the workspace, so accept the root package's own version or derive the range from one shared constant; cli-help-overview.ts:52 says 'replay/test plugins', which reads as two plugins when one serves both commands, and the facet summaries at command-surface/src/replay/index.ts:317,331 write by hand the note that replayPluginInstallHelp already generates from the declaration; and request-router.ts:278 and cli-help.ts:881 each special-case the command names replay and test, so declaring the plugin requirement on the command facets and having both enumerate it would remove the duplicate checks.

This review is from reading the code. I did not run tests or the packed smoke, and I did not reproduce the earlier-commit claims (tree identity with 7a799d9, 1,714 focused tests, seven packed checks, 31 repaired hosted cases). I did not trace the composite-resource-admission fixture, the CI gate action changes or the install-workspace-replay-plugins script in depth. Whether ADR 0034's proposed native amendment is accepted is a maintainer decision outside this review.

Before merge, the Maestro help must state the @agent-device/replay requirement, and the live Android and iOS runs through the installed plugin must be attached.

@thymikee
thymikee force-pushed the t3/replay-plugin-execution-routing branch from 08ce68f to 67c0e73 Compare November 14, 2026 01:51
@thymikee

Copy link
Copy Markdown
Member Author

Thanks for the update. I re-read the changes at 67c0e73 against the earlier review (#4537 (comment)). Two findings from that review remain, and the other earlier findings are now addressed.

CI is green. The 12 jobs that first showed as not passing were cancelled runs, and all 12 passed in their re-runs. Smoke Tests and the Mutants matrix were skipped, so CI gives no device signal for the plugin execution route. I reviewed by reading the code. I did not run the new daemon-composition-replay-selection tests or the packed smoke. I reasoned about the four custom replace/remove cases from the earlier code path and did not run them against the old code. No conflicts.

The Maestro help still tells agents to install only @agent-device/maestro and then run test <path> --maestro (replay-backend-declarations.ts#L56). Every test request resolves native execution through @agent-device/replay first, so an agent that follows this help gets plugin-not-installed for a package the help never named. It then has to install it and restart the daemon a second time. The rule: every help text that tells agents to run test must name every package the test route admits. Please build the Maestro install line from REPLAY_BACKEND_DECLARATIONS.ad (package plus installationHint), so the declaration that drives the plugin-not-installed error also drives the help.

No run yet shows a device-facing route working through the plugin-provided engine (native-command.ts#L91). The PR body still lists live Android and iOS verification as pending. Please install @agent-device/replay with plugins add, restart the daemon, and show plugins list with it active. Then, on Android and on iOS, run a real .ad script that taps or fills and then closes, and show replayed > 0. Also run a two-test suite with --retries and show the suite JSON with passed counts and artifactsDir. With the plugin absent, run the same replay and show plugin-not-installed with no session created.

Before merge, the Maestro help must name @agent-device/replay for test suites, and the live Android and iOS runs plus the plugin-absent refusal must be attached.

@thymikee
thymikee force-pushed the t3/replay-plugin-execution-routing branch from 67c0e73 to 5fe9441 Compare November 14, 2026 05:57
@thymikee

Copy link
Copy Markdown
Member Author

Thanks for the update. At 651a68b, two findings from the earlier review (#4537 (comment)) are still open: the help maestro text and the missing live runs. CI is green with 19 checks reported and none failing, but the smoke and device jobs give no signal for the plugin execution route. There are no conflicts.

The agent-device help maestro text still tells users to install only @agent-device/maestro and then run test <path> --maestro (replay-backend-declarations.ts#L46, served at cli-help.ts:77). requireReplayPlugins calls getReplayExecution() for every test request before it checks Maestro. A user who follows the help therefore gets plugin-not-installed for @agent-device/replay, which the help never names. It then has to install a second plugin and restart the daemon again. Could every help text that tells users to run test name each package that requireReplayPlugins admits for that route? In the Maestro branch, build the suite line from getReplayPluginInstallation('ad'), for example "test <path> --maestro also requires agent-device plugins add ${ad.package} on the daemon host", and make that a value import. One assertion in the existing help test that help maestro contains @agent-device/replay would cover it.

Native replay and test suites now route through the plugin execution (native-command.ts#L91, test-command.ts:199). The PR body claims no device run, and Smoke Tests were skipped. Nothing yet shows that a device replay works through the plugin engine, or that the plugin-absent refusal happens before a session exists. Please attach these runs: (1) agent-device plugins add @agent-device/replay, a daemon restart, and plugins list showing it active; (2) on Android and on iOS, a real .ad script that taps or fills and then closes, showing replayed > 0; (3) a two-test suite with --retries, showing the suite JSON with passed counts and artifactsDir; (4) the same replay with the plugin removed, showing plugin-not-installed with details.package @agent-device/replay and an empty session list; (5) after the help fix, test <flow.yaml> --maestro with only @agent-device/maestro installed, showing the refusal that names @agent-device/replay.

Not blocking, take or leave: getReplayBackendDeclaration and replayPluginInstallHelp now take ReturnType<typeof getReplayPluginInstallation>, and every caller passes getReplayPluginInstallation(<literal id>), so you could keep the (id: ReplayFormat) signature and do the lookup inside both functions.

I read the code and did not run any tests, including command-tools-parity, cli-startup-import-closure, the installed-package-metro consumer, or the packed smokes. I did not re-trace the client-side Maestro collection route or the "core clients can send .ad sources" sentence, since they moved verbatim and the earlier review covered them. I compared the help text before and after by reading it, not by a run.

@thymikee

Copy link
Copy Markdown
Member Author

Fixed the Maestro prerequisite finding in e567c5b. help maestro now names both packages, source-collection/daemon scopes, and one restart after installing prerequisites. The native suite line receives the actual getReplayPluginInstallation('ad') data from the existing help owner. I retained the explicit-data construction: moving the full catalog lookup back into the heavy declaration entry previously grew its eager closure, so the optional id-based simplification would reintroduce a verified failure.

The existing help test failed on the missing native install command before the fix. Committed focused results: 1,019 passed across eight files, including all eager-closure roots and MCP text policy. Format, lint/typecheck, build and layering passed. Seven actual packed checks passed. The new Maestro-only packed embedded request uses the real factory and current client: replay.test({paths:[entry], backend:'maestro', platform:'ios'}) rejects with:

{"code":"INVALID_ARGS","details":{"reason":"plugin-not-installed","package":"@agent-device/replay","installCommand":"agent-device plugins add @agent-device/replay"}}

Sessions are empty before and after. This is a device-free packed embedded admission check, not an installed-store live device run. Client source collection may create Maestro's backend before daemon admission; I removed an incorrect test assumption that no backend could be created. No execution or session success is inferred from collection.

For the outstanding live requests, here are actual retained outputs at 67c0e73, not a new execution at e567:

{"backend":"ad","source":"store","package":"@agent-device/replay","version":"0.0.0","state":"active"}
{"nativeResume":{"success":true,"replayed":14,"healed":0,"sessionActive":false}}
{"nativeSuite":{"passed":0,"failed":1,"attempts":1}}
{"maestroReplay":{"success":true,"replayed":5,"healed":0,"sessionActive":true}}

The native package was explicitly installed from its real tarball (SHA256 10673f78cbf249e3c35b1efd293190c562818d587bd2d63983e2722e903617a6), removed/reinstalled at the same version and activated after operator restart. Native-absent replay/test returned plugin-not-installed, package @agent-device/replay, before journal actions; session inventory was empty. The real iOS full navigation fixture failed step3 Catalog selection; one unchanged --from 3 continuation passed 14 steps, while the full suite failed the same step. That fixture path stopped. The successful Maestro parent/include flow launched, tapped Catalog, asserted it and wrote its authored screenshot. Raw suite success:true was only the reporting envelope, not a passing test.

Android native replay and a passing two-test retry suite remain unproven. Recording's HTTP restriction exists in current code as well as the old bound client. I have not bypassed T3 with an unbound device CLI, retried stopped device paths, changed assertions or called these partial results full clearance. Evidence: /private/tmp/c4-live-67-20261114/C4-report.txt and raw native-replay/resume/suite, maestro-replay and admission-journal records. Current-head independent review and affected/full gates are next; C3, ADR0034 acceptance and npm prerequisites remain separate blockers. This stays draft.

@thymikee
thymikee marked this pull request as ready for review November 14, 2026 08:08

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 69 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread website/docs/docs/plugins.md Resolved
Comment thread .github/workflows/replays-nightly.yml Resolved
Comment thread packages/replay/test/package-smoke.mjs Resolved
Comment thread packages/plugins/src/store.ts Resolved
@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Apex Sentinel · ✅ Approve

Since the last review: 1 new · 0 still open · 0 no longer raised

1 finding(s): 1 minor.

🟡 Minor (1)

Where Finding
packages/replay/src/plugin.ts:10 🆕 Plugin accepts core 0.22.0-alpha.1 although the declared range starts at 0.22.0-0
Not in the reviewers' prompt (1 file)
  • pnpm-lock.yaml: generated or a lockfile, so its lines were left out

Reviewed with this repository's own Sentinel configuration (d78be6c88715).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

Earlier reviews

Review of 2e40c35e · 14.11.2026 20:57 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 1 no longer raised (1 marked fixed)

No open findings.

Not in the reviewers' prompt (1 file)
  • pnpm-lock.yaml: generated or a lockfile, so its lines were left out

Reviewed with this repository's own Sentinel configuration (d78be6c88715).

Review of 3c32e14e · 14.11.2026 10:19 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 1 still open · 0 no longer raised

1 finding(s): 1 minor.

🟡 Minor (1)

Where Finding
packages/replay-port/src/daemon-port/replay-plugin-admission.ts:17 Missing replay plugin still hides the explicit --maestro routing hint for .yaml test inputs
Not in the reviewers' prompt (1 file)
  • pnpm-lock.yaml: generated or a lockfile, so its lines were left out

Reviewed with this repository's own Sentinel configuration (d38e35e61193).

Review of 153a1279 · 14.11.2026 10:07 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 1 still open · 0 no longer raised

1 finding(s): 1 minor.

🟡 Minor (1)

Where Finding
packages/replay-port/src/daemon-port/replay-plugin-admission.ts:20 Missing replay plugin still hides the "requires explicit --maestro routing" hint for .yaml input
Not in the reviewers' prompt (1 file)
  • pnpm-lock.yaml: generated or a lockfile, so its lines were left out

2 findings below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d38e35e61193).

Review of e567c5ba · 14.11.2026 08:22 UTC

Apex Sentinel · ✅ Approve

1 finding(s): 1 minor.

🟡 Minor (1)

Where Finding
packages/replay-port/src/daemon-port/replay-plugin-admission.ts:20 Missing replay plugin hides the "requires explicit --maestro routing" hint for .yaml input
Not in the reviewers' prompt (1 file)
  • pnpm-lock.yaml: generated or a lockfile, so its lines were left out

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d38e35e61193).

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding on e567c5b. The verdict and the full list are in the summary comment.

@thymikee

Copy link
Copy Markdown
Member Author

Both new device-job failures are now traced to missing plugin installation, not assumed infrastructure faults:

  • Android job: the booted emulator reached smoke:maestro-clickable-first; its valid replay ... --maestro refused with INVALID_ARGS / plugin-not-installed / package @agent-device/maestro before Maestro execution.
  • iOS job: the native test .../01-settings.ad --retries 2 refused with the same typed reason for @agent-device/replay. The selected Apple phase correctly stayed failed; I am not treating its prerequisite refusal as passing suite execution.

The C3 Android workflow fix is locally committed: select an isolated AGENT_DEVICE_HOME and invoke its existing built-workspace installer after build and before daemon startup. When restacked here, that call uses the existing combined replay-plugin installer. I also prepared Apple workflow setup after successful shared build, exporting the same isolated plugin home to subsequent iOS/macOS phases. Linux already routes execution through run-gate, whose declared execution-gate setup installs the real plugins; no Linux change is proposed. No scenario/assertion/timeout/selector fallback or product auto-install is added.

Separately, the exact e567 local affected and full gates both passed once: 42/25 and 41/25, exactCommit=true, checkoutChanged=false, clean fingerprints. I independently verified both raw receipts and 113 original/copy SHA-256 pairs in the immutable report. These do not clear either hosted device failure. Corrections will be published with the coordinated stack refresh after the integration candidate is validated and the active source review is preserved. The current failed jobs will not be blindly rerun. This PR stays ready for review.

@thymikee

Copy link
Copy Markdown
Member Author

I reviewed e567c5b against the earlier findings (#4537 (comment)). The smoke jobs still fail because of this stack, and the native device evidence is still missing.

The smoke jobs start a daemon with an empty plugin store, but replay and test now need the installed plugins (requireReplayPlugins at request-router.ts:282). The workspace installer runs only from run-gate's gate ids (run-gate/action.yml:19-25), conformance-differential.yml:72 and replays-nightly.yml:61. Smoke Tests (Android) (https://github.com/callstack/agent-device/blob/e567c5b/.github/workflows/android.yml#L28) and the Apple smoke phase (.github/actions/smoke-ios-run, via apple.yml:88) skip run-gate. On Android, replay ... --maestro is refused with plugin-not-installed for @agent-device/maestro at smoke:maestro-clickable-first. The iOS refusal for @agent-device/replay comes from the author's report. The Apple jobs show only the aggregate failure. Three required checks are red because of this, and the lanes that would prove the plugin route on a device run nothing. The rule: every CI job that starts a daemon and runs replay or test installs both first-party plugins through one shared installer before the daemon starts. Please make that one owned step, for example a composite action that run-gate, smoke-android, smoke-ios-run/smoke-macos, conformance-differential and replays-nightly all call, instead of more inline copies. Then push it and get Smoke Tests (Android) and Smoke Tests (ios) green at the new head.

The second finding from the earlier review still stands. Native replay and test suites now run through the @agent-device/replay plugin (https://github.com/callstack/agent-device/blob/e567c5b/packages/replay-port/src/daemon-port/native-command.ts#L91). The only live runs I could see are iOS-only partial results at 67c0e73: the full replay failed at Catalog selection, the suite passed 0 and failed 1, and only a --from 3 continuation passed (replayed 14). That evidence is from an older head and I did not verify it. No Android native run exists, and no passing two-test --retries suite exists on any platform. At the current head, on Android and on iOS, please show: (1) plugins add @agent-device/replay, a daemon restart, and plugins list showing it active; (2) a full .ad script that taps or fills and then closes, succeeding with replayed > 0, not a --from continuation; (3) a two-test suite with --retries showing passed counts and artifactsDir; (4) the plugin-absent refusal naming @agent-device/replay, with an empty session list; (5) test <flow.yaml> --maestro with only @agent-device/maestro installed, refused naming @agent-device/replay. If the iOS full replay still fails at Catalog, please show the same fixture passing on main so we can tell the fixture from the plugin route. Green smoke lanes can cover (1)-(3) on both platforms if their output shows these fields.

Not blocking: the help path at cli-help.ts:887 passes the same ad installation twice to getReplayBackendDeclaration, and suiteInstallation is read only in the Maestro branch (replay-backend-declarations.ts:49), so you could make it optional there or leave it. I did not run any test, gate or packed smoke; this is from reading the code.

Open threads still apply on #4537 (comment) (private package, docs install) and #4537 (comment) (ADR 0016 scope). Eight lower-priority threads also still apply. One thread does not apply, so please resolve it: #4537 (comment) (the empty backend is consistently treated as native).

CI looks related to this change, since the failing jobs run replay and test on a booted device. No conflicts. Before merge, the shared plugin install needs to land with the smoke jobs green, and the live Android and iOS runs need to be attached.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 20 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread .github/workflows/apple.yml Resolved
@thymikee
thymikee force-pushed the t3/replay-plugin-execution-routing branch from 153a127 to 3c32e14 Compare November 14, 2026 10:11

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@thymikee

Copy link
Copy Markdown
Member Author

Thanks for the update. At 3c32e14 the code looks right, but I still can't see live proof for the new route. I read the code and did not run any test, gate, installer or packed smoke.

Since the earlier review (#4537 (comment)), the docs now say both plugin packages are private and unpublished, the ADR scope and the Maestro requirement are fixed, the empty backend is refused, and the nightly and prepack install steps are fixed. Two points remain: live validation and the CI installer step.

Native replay and every test suite now run only through the installed @agent-device/replay plugin (native-command.ts#L91, admission.ts lines 17-31). No live Android or iOS run of that route at this head is attached. The hosted smoke jobs at 3c32e14 were all cancelled, so they prove nothing either. The only earlier live evidence was iOS-only and partial, at 67c0e73. This is a device-facing dispatch change, so please attach, on Android and on iOS at this head: (1) plugins add @agent-device/replay, a daemon restart, and plugins list showing it active; (2) a full .ad script that taps or fills and then closes, succeeding with replayed > 0 (not --from); (3) a two-test suite with --retries showing passed counts and artifactsDir; (4) the plugin-absent refusal naming @agent-device/replay with an empty session list; (5) test <flow.yaml> --maestro with only @agent-device/maestro installed, refused naming @agent-device/replay. Completed green Smoke Tests (Android) and (ios) whose logs show these fields can cover (1)-(3).

The previous review asked for one owned installer step, but the delta added inline installer calls at android.yml#L96 and apple.yml#L88. This copy-per-site approach already misses a site. The manual-android job in replays-manual.yml runs android-manual.sh, which calls replay (line 13) and pnpm gate replay-android (line 23) directly, outside run-gate, with no AGENT_DEVICE_HOME and no plugin installation. The install lives in the run-gate composite (run-gate/action.yml lines 19-26), so a shell call to the gate skips it. The rule: every daemon that runs replay or test in CI or through pnpm gate gets both workspace plugins from one owner before it starts. Four installer sites remain with different build and home handling. Please enumerate the call sites from the workflows and scripts that start a replay or test daemon, and make them all use that one owner.

Not blocking, and you can take or leave these: the unrouted-YAML refusal now exists in three places with the same predicate and message (admission.ts#L27, native-command.ts:81-82, source-discovery.ts:87-88), and backend !== undefined at admission.ts:15 is another routing guard around the same concept, so one ad-script routing function could own the decision; and in replay-e2e.md#L47 the "private and not yet published" paragraph sits between the lead-in sentence ending with a colon and the code block it introduces, so it reads better above the lead-in, as plugins.md does.

Could both collapses be done together? (a) One ad-script function returns the route or throws the typed routing error (unknown or empty backend, YAML without --maestro), and admission, native-command and source discovery call it, deleting the three isMaestroYamlPath guards. (b) One composite action, or a gate-declaration trait that pnpm gate consumes, builds both plugins, sets AGENT_DEVICE_HOME and installs them. Then run-gate, smoke-android, apple smoke, conformance-differential and replays-manual all call it, and macOS stops reinstalling three times.

All 13 non-passing checks at 3c32e14 are cancelled, not failed, so there is no failure to attribute. The delta changes the route that Smoke Tests exercise, so those jobs need to rerun to completion before the smoke lanes count as fixed. I did not check whether the cancellations came from a newer push or a manual cancel. I also did not trace every gate id to confirm the run-gate install list covers all gates that start a daemon; I only grepped workflow and script call sites.

There are no conflicts. Before merge, please attach the live Android and iOS plugin-route evidence at this head with Smoke Tests (Android) and (ios) green, and fold the per-workflow installer copies into one owned step that also covers manual-android.

On the open inline threads: the thread on redundant macOS installs still holds (#4537 (comment)), and it folds into the single installer step above. No other blocking thread still applies. These threads do not apply and can be resolved: admission order for suites, because every suite needs @agent-device/replay and all surfaces refuse bare YAML before the daemon (#4537 (comment)); plugin-not-installed advice for raw requests, which is correct since test always needs the plugin (#4537 (comment)); and the threads fixed at this head: private-package docs (#4537 (comment)), ADR 0016 scope (#4537 (comment)), nightly install step (#4537 (comment)), prepack builds (#4537 (comment)), installation hint (#4537 (comment)), ADR 0012 Maestro requirement (#4537 (comment)), smoke script command runner (#4537 (comment)), empty backend (#4537 (comment)), selection stamp (#4537 (comment)), installer rollback (#4537 (comment)), and startup import closure test (#4537 (comment)).

@thymikee
thymikee force-pushed the t3/replay-plugin-execution-routing branch from 3c32e14 to 2e40c35 Compare November 14, 2026 20:34
@thymikee
thymikee removed this pull request from stack #4529 November 14, 2026 20:35
@thymikee
thymikee added this pull request to stack #4556 November 14, 2026 20:35
@thymikee

Copy link
Copy Markdown
Member Author

At 2e40c35 the routing fixes from the earlier review (#4537 (comment)) are in: test suite.yaml without --maestro now gets the explicit routing INVALID_ARGS before getReplayExecution(), and the router test covers the yaml/yml rows. What remains is live proof of the plugin route.

Native .ad replay and every test suite still run only through the installed @agent-device/replay plugin (native-command.ts#L91). At this commit there is no live Android or iOS run of that route, and the PR body says platform and live validation remain blocked. This is a device-facing dispatch change, so it would merge without proof that a plugin-installed daemon replays scripts and runs suites. Please attach these five runs on Android and on iOS at this head: (1) plugins add @agent-device/replay, a daemon restart, and plugins list showing it active; (2) a full .ad replay that taps or fills and then closes, with replayed > 0 (not --from); (3) a two-test suite with --retries showing passed counts and artifactsDir; (4) the plugin-absent refusal naming @agent-device/replay with an empty session list; (5) test <flow.yaml> --maestro with only Maestro installed, refused naming @agent-device/replay. Completed green Smoke Tests (Android) whose log reaches the 01-settings.ad test step, plus Smoke Tests (ios), can cover (1)-(3).

Hosted Android Smoke (run 43638214672) fails in smoke:maestro-clickable-first, where replay ... --maestro diverges at step 2 (assertVisible 'Maestro selection: clickable'). The base PR's Android run 43638215393 fails the same way, and main's Android passes at d78be6c, so this looks like it comes from the stack below and not from this delta. It does hide this PR's native .ad step (android.yml#L99), so that failure needs a fix first. The Apple jobs were cancelled after 1s, and the other non-passing checks are cancelled or queued. This delta moves AGENT_DEVICE_HOME to github.workspace/.tmp and makes the Apple install iOS-only, so both smoke lanes must rerun to completion.

Not blocking, and you can take or leave these: four install sites still remain (run-gate/action.yml:18-26, android.yml:96, apple.yml:88, conformance-differential.yml:67-73) and replays-manual.yml manual-android still runs android-manual.sh#L13 without a plugin or AGENT_DEVICE_HOME, and macOS installs twice; the rule would be that every daemon that runs replay or test in CI or through pnpm gate gets both workspace plugins from one owner before it starts; admission now carries a second unrouted-YAML guard (replay-plugin-admission.ts#L27), so the isMaestroYamlPath plus maestroBackendRequiredMessage pair appears at four sites (admission x2, native-command.ts:81-82, source-discovery.ts:87-88); and the PR body says no new package, export or gate exception is added, while the diff adds the private @agent-device/replay package with its .fallowrc.json entry allowances.

Could one @agent-device/ad-script function resolve the route for an entry list and backend, or throw the typed routing error, so admission, native-command and source discovery call it and the four YAML guards go away? Could one composite action, or a gate-declaration trait that pnpm gate consumes, build both plugins, set AGENT_DEVICE_HOME and install, so run-gate, smoke-android, apple smoke, conformance-differential and manual-android all use it and the inline copies go away?

The open inline thread on the macOS double install (r4572890870) still applies and is covered by the install note above. These two are fixed at this head and can be resolved: r4572985095 (admission now refuses YAML without maestro before getReplayExecution()) and r4572549692 (test suite.yaml without --maestro now gets the routing INVALID_ARGS).

Before merge, this needs live Android and iOS evidence for the plugin route at this head, or green Smoke Tests (Android, reaching the native 01-settings.ad step) and (ios). I ran no tests, gates, installer or packed smoke, so regression validity is reasoned from the pre-change admission code. I did not open the #4505 evidence comment, did not check whether the Maestro clickable-first failure is a flake or a real base-stack regression, and grepped only workflow and script call sites for the install list.

Land the native plugin with its routing consumer, installation, workflows and package guards. Retain C3 rejected-load retry and inventory ownership with typed repair guidance; preserve lower-layer slot identity and refuse raw YAML suites before plugin or device admission.
@thymikee
thymikee force-pushed the t3/replay-plugin-execution-routing branch from 2e40c35 to ce66416 Compare November 15, 2026 01:14

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding on ce66416. The verdict and the full list are in the summary comment.

Comment thread packages/replay/src/plugin.ts Resolved
@thymikee

Copy link
Copy Markdown
Member Author

This PR is ready at ce66416, and the findings from the earlier review (#4537 (comment)) are now fixed. I read the code and the hosted logs. I did not run any test, gate or packed smoke myself. The checks show green on this head: Android 43662183826, Apple 43662183827 (iOS and macOS), Linux 43662183807, CI 43662183783 and mutants 43662183833 all pass. The failed Apple run and cancelled mutants run in the list belong to a duplicate run set that concurrency superseded on the same head. The smoke lanes reach 01-settings.ad natively on iOS and Android, so the routes this diff touches are covered. The plugin-absent refusals are pre-device daemon refusals, and request-router-replay-plugin.test.ts covers them through the router handler. Not blocking, and you can take or leave it: the PR body says the Apple and mutation jobs were cancelled with no replacement result, so please update the Validation section to link the green Android 43662183826 and Apple 43662183827 runs. There are no conflicts. The one open inline thread, #4537 (comment), does not apply: the check in packages/replay/src/plugin.ts accepts any 0.22.* version, and the declared range '>=0.22.0-0 <0.23.0' admits 0.22.0-alpha.1 under semver, so the check and the range agree, and you can resolve it. Before merge, publish @agent-device/replay and @agent-device/maestro (npm placeholders and trusted publishers) and lift private.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant