Skip to content

feat(plugins): validate native execution in the managed replay registry - #4535

Open
thymikee wants to merge 1 commit into
t3/replay-data-contractsfrom
t3/replay-execution-plugin
Open

thymikee wants to merge 1 commit into
t3/replay-data-contractsfrom
t3/replay-execution-plugin

Conversation

@thymikee

@thymikee thymikee commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Summary

Introduce optional replay execution registry contracts for #3871, preserving backend identity, transient recovery and disposal fencing.

Unpublished candidate: 8f2ac64f935d4fbf740c5de02d3c5a8b779a5a6e. 12 files / 385 gross lines above DATA; 333 / 9,141 against default-main merge-base.

Validation

Independent Codex recheck found no actionable findings within the R1 repair and preservation interactions. The two-file repair survives all four rebases; eight range-diff entries match.

At 8f2ac64f935d4fbf740c5de02d3c5a8b779a5a6e, pinned frozen install/build and full-repository format passed. Exact affected: 42 passed / 25 skipped, exactCommit=true, checkoutChanged=false; receipt c43ad7c22d01eeea3b699e4c88499cf6311aa0c8f6c5d054845884880284093e.

All seven new-head canonical invocations passed once, serially, under one frozen ENV. Counts overlap; skips retain their CI/manual owners. Existing refs stayed unchanged; source/logical index matched before and after each gate. Physical-index captures remain separate, with within-head byte-change causes unproven. Initial/final snapshots cannot exclude transient edits.

The candidate remains unpublished; published head 61ae2772 is unchanged and new-head hosted checks are unverified. Node22.13 packaging and earlier ENV/ref owner dispositions still block source publication. Both plugins remain private pending maintainer placeholders, trusted publishers and usable publication. Native ADR, live/current-client, performance and security remain open. No merge clearance.

@thymikee
thymikee added this pull request to stack #4529 November 13, 2026 23:31
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.08 MB 6.11 MB +34.9 kB
Package (unpacked) 6.08 MB 6.11 MB +34.9 kB
Package (download) 1.84 MB 1.85 MB +10.1 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 28.9 ms 32.2 ms +3.3 ms
CLI --help 61.9 ms 57.1 ms -4.8 ms

@thymikee
thymikee force-pushed the t3/replay-execution-plugin branch from aedf9f5 to 0315a02 Compare November 13, 2026 23:35
@thymikee

Copy link
Copy Markdown
Member Author

At 0315a02, one defect failed CI, and the design needs a decision before merge. The newer head 8180363 changes only packages/replay/src/plugin.ts to declare the factory export explicitly, which looks like the fix for the first point; the next pass will confirm once Repo Guards reruns. The Repo Guards & Compatibility check fails at the import-direction DAG step, and this diff causes the failure. There are no conflicts.

packages/replay/src/plugin.ts:4 uses export default function replayPlugin(). This file is a manifest-declared façade, and scripts/layering/facade-surface.test.ts:21 cannot enumerate that export form. The PR cannot merge while this gate is red. Please copy the Maestro spelling: declare function replayPlugin(): ReplayEnginePlugin and end the file with export { replayPlugin as default };. The rule is that every façade a package manifest declares uses only named export forms, and facadeEntryFiles lists those façades. Later steps in that job (fallow, native-dupes, gate-of-gates) may also fail once this is fixed, because I only inspected the first failing step.

Not blocking, and you can take or leave these: (1) resolveReplayExecution, the execution union, validateReplayExecution and @agent-device/replay have no production consumer at this head, while replay-port still imports @agent-device/ad-replay and the root package still depends on it and on @agent-device/replay-test, so the new package re-bundles engine code that core also ships. (2) validateReplayEngineSetup at validate.ts:162 accepts replayBackend or execution without knowing the backend id, so a mismatch passes setup, shows as 'active' in inventory, and only throws at resolve time without recording slot.failure. It would be better to require execution exactly when the backend is 'ad', with one constant shared with registry.ts:210. (3) replay-execution.ts:5 spells the public contract as typeof the private engine functions, which adds @agent-device/ad-replay and @agent-device/replay-test as runtime dependencies of the plugins package; declaring these port types in @agent-device/contracts next to replay-backend would avoid that. (4) The core-version gate in plugin.ts:10 pins '0.21.38-dev' and will reject the next dev bump, and Maestro has no such gate, so one owner such as discovery could hold the supported range for both. (5) plugin.test.ts:22 feeds { ... } as never to the suite and asserts replay-test's own message, and the 'shares backend claims' test registers only one candidate, so a case with two 'ad' candidates is missing. (6) @agent-device/replay is wired into neither the root build script nor a packed-entry smoke like Maestro's, so nothing proves the packed dist/agent-device.mjs loads through discovery.

Could this land as one layer instead of a standalone package and an unconsumed resolver? That layer would declare the native execution port in @agent-device/contracts, add the factory keyed by the 'ad' backend in one validated constant, switch replay-port's replay/test path to registry.resolveReplayExecution, and drop ad-replay and replay-test from the root dependencies. The ~30-line packages/replay factory would then land with a packed smoke. If the stack must stay split, could this PR be cut to the registry seam with contracts-owned types and no new dependencies, with packages/replay following alongside the routing layer? Either way, the execution port types must move to contracts first, and an ADR 0034 amendment should decide whether native execution is a factory kind on 'replay-backend' or its own plugin kind.

I did not run tests, typecheck, or build and pack @agent-device/replay, so the bundled output and the unresolved-import guard are unverified. I judged the new registry tests from the diff only. Before merge, the façade export must be fixed so Repo Guards turns green, and then the contracts-owned port and its routing consumer need a decision.

@thymikee
thymikee force-pushed the t3/replay-execution-plugin branch from 8180363 to 9f1fa69 Compare November 14, 2026 00:45
@thymikee thymikee changed the title feat(plugins): add optional native replay execution package feat(plugins): validate native execution in the managed replay registry Nov 14, 2026
@thymikee
thymikee force-pushed the t3/replay-execution-plugin branch from 9f1fa69 to 25e621d Compare November 14, 2026 01:05
@thymikee
thymikee force-pushed the t3/replay-execution-plugin branch from 25e621d to 499d4bc Compare November 14, 2026 01:51
@thymikee

Copy link
Copy Markdown
Member Author

The code at 499d4bc is ready; I found no problems that need a change. Not blocking, and fine to take or leave: resolveReplayExecution in registry.ts near-duplicates resolveRegisteredReplayBackend, so one slot.loading promise and a shared createOnce helper could serve both; the SDK re-export of the ReplayExecution types in plugins.ts could wait until #4537 lands or the ADR is accepted; and the references reflow in tsconfig.json could be reverted, with NATIVE_REPLAY_BACKEND in replay-execution.ts moved below the imports. I did not run tests, typecheck or gates locally, and no production route reaches resolveReplayExecution at this head, since routing lands in #4537. The 11 jobs that did not pass were cancelled, not failed, and several of the same jobs pass on a later run set for this head, so CI looks unrelated to the change. Before merge, the maintainer needs to accept the ADR 0034 native amendment, and hosted CI needs to be green on the newest run set.

@thymikee
thymikee force-pushed the t3/replay-execution-plugin branch from 499d4bc to 3e5030d Compare November 14, 2026 05:57
@thymikee

Copy link
Copy Markdown
Member Author

Current registry head is 3e5030d2dc690bddaac1c5b713b9a19b97c2aab2. The contracts-owned execution port, explicit native capability validation, first-claim/failure inventory, named factory export and routing/packed consumers requested in the earlier review are present. The custom winning-package freshness finding was fixed at the registry/composition boundary and independently rechecked; embedded runtimes retain store-free behavior.

I am retaining the small existing registry implementation rather than adding a shared create-once abstraction solely to remove similar lines. The independent smaller-design assessment accepted the three execution entrypoints and accounted for the +34.7 kB package growth (mostly declarations); it did not establish a performance win. The native ADR0034 amendment remains Proposed, pending maintainer acceptance.

Exact-head affected admission passed: 42 passed / 25 skipped, exactCommit=true, checkoutChanged=false. All 19 hosted checks passed. Later source/help fixes are owned by the top routing layer, so this head has not been churned. No registry/source readiness statement here clears C3, live/platform, ADR or npm placeholder/trusted-publisher prerequisites.

@thymikee
thymikee marked this pull request as ready for review November 14, 2026 08:08

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 11 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@blacksmith-sh

This comment has been minimized.

@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 1 no longer raised

1 finding(s): 1 minor.

🟡 Minor (1)

Where Finding
website/docs/docs/plugins.md:172 Public plugin docs still describe a setup shape the new validator now rejects

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d78be6c88715).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

Earlier reviews

Review of f3901320 · 14.11.2026 20:59 UTC

Apex Sentinel · ✅ Approve

Since the last review: 1 new · 0 still open · 1 no longer raised

1 finding(s): 1 minor.

🟡 Minor (1)

Where Finding
packages/plugins/src/registry.ts:219 🆕 A one-off native execution create() failure permanently disables the slot, unlike backend loads

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d78be6c88715).

Review of 89400750 · 14.11.2026 10:19 UTC

Apex Sentinel · ✅ Approve

Since the last review: 1 new · 0 still open · 0 no longer raised

1 finding(s): 1 minor.

🟡 Minor (1)

Where Finding
packages/plugins/src/registry.ts:209 🆕 resolveReplayExecution can pick another backend's slot and report invalid_plugin, not plugin-not-installed

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d38e35e61193).

Review of 2c5f4d13 · 14.11.2026 10:04 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 0 no longer raised

No open findings.

Reviewed with this repository's own Sentinel configuration (d38e35e61193).

Review of 3e5030d2 · 14.11.2026 08:21 UTC

Apex Sentinel · ✅ Approve

No open findings.

3 findings below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d38e35e61193).

@thymikee

Copy link
Copy Markdown
Member Author

Confirmed at 3e5030d2: smoke:deep-link-navigation fails when the stored-URL Maestro YAML is replayed without @agent-device/maestro installed (INVALID_ARGS, details.reason: plugin-not-installed). Blacksmith reports that same failure, and the selected Apple aggregate correctly fails with iOS.

The lower Maestro-layer Apple setup correction is already queued in the coordinated stack refresh, so it will reach this PR too. It installs the built plugin after the shared build under a persistent job-scoped plugin home. No additional engine change or CI retry is needed for this diagnostic. Fresh hosted iOS success remains required after publication.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding on 8940075. The verdict and the full list are in the summary comment.

Comment thread packages/plugins/src/registry.ts Resolved
@thymikee

Copy link
Copy Markdown
Member Author

I reviewed 8940075 and found no code problems. The earlier review of 3e5030d (#4535 (comment)) found the code clean, and the code commits are unchanged since then. The new commit 8940075 only reverts the ADR changes, and ADR 0034 now matches base.

Hosted CI is unclear. Coverage failed and ten jobs were cancelled, which fits a superseding push. I could not read the Coverage failure, so I cannot say whether it relates to this PR. I also did not rerun tests locally. Please rerun CI on 8940075 and check the Coverage result.

On the open threads: the minor thread about the package-name fallback in the registry still holds (#4535 (comment)). It is not urgent, because resolveReplayExecution has no production caller yet. When a caller lands, take the package fallback only for slots whose backend is undefined or equals the id. The two ADR threads no longer apply, so please resolve them. The first is moot because 8940075 deletes the section with the cancellation-port sentence (#4535 (comment)). The second is fixed because the proposal is gone from the accepted ADR (#4535 (comment)).

Before merge, CI must pass on 8940075 and the two moot threads must be resolved. The maintainer's native-contract acceptance, which the PR names as a merge condition, is still needed. I could not confirm the preserved proposal text in #3871 beyond your reply.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 14, 2026
@thymikee
thymikee force-pushed the t3/replay-execution-plugin branch from 8940075 to f390132 Compare November 14, 2026 20:34
@thymikee
thymikee removed this pull request from stack #4529 November 14, 2026 20:35
@thymikee
thymikee added this pull request to stack #4556 November 14, 2026 20:35
@thymikee

Copy link
Copy Markdown
Member Author

This PR is ready at f390132. The earlier review of 8940075 found no problems, and the fallback by package name is now limited to slots with no backend and a recorded failure. The test 'a package name cannot select an active foreign backend for native execution' covers it. Tests were not run locally. I checked this by reading the code.

Not blocking: a rejected native execution.create() sets a permanent slot.failure and keeps the rejected promise cached at https://github.com/callstack/agent-device/blob/f390132/packages/plugins/src/registry.ts#L220, while failed format loads now retry. You can take or leave this. The rule would be that setup failures stay permanent and every lazy factory load, format or native, is retryable and reports a transient failure. One lazy-load helper on the slot, plus the native case in the existing retry test, would cover it.

The open inline thread on registry.ts:322-343 is fixed at this head, so you can resolve it: #4535 (comment). Two threads no longer apply because the head does not touch the ADR 0034 docs, so you can resolve them too: #4535 (comment) (outdated amendment) and #4535 (comment) (proposal removed, it lives in #3871).

All checks pass at this head except Android Smoke Tests run 43638215393. It failed at step 2, assertVisible 'Maestro selection: clickable', in 'resolve duplicate target through no-index Maestro tapOn'. The diff only touches backend selection, and that selection and step 1 succeeded, so this looks unrelated. I did not compare it with a main run. Before merge, please rerun that job or show it fails the same way on main.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding on f390132. The verdict and the full list are in the summary comment.

Comment thread packages/plugins/src/registry.ts Resolved
@thymikee
thymikee force-pushed the t3/replay-execution-plugin branch from f390132 to 61ae277 Compare November 15, 2026 01:14
@thymikee

Copy link
Copy Markdown
Member Author

This PR is ready at 61ae277. Since the earlier review (f390132), the failed-create recovery and the package-name fallback concerns are fixed in the code.

Not blocking: the native lazy-load block in registry.ts is a line-for-line copy of the format lazy-load in resolveRegisteredReplayBackend. One helper could own the rule "a resolved factory is cached; a rejection clears the in-flight promise, records replayFailure and refreshes inventory", and both resolvers could call it with their own loading key, factory and validator. You can take this or leave it.

About the open review threads: two of the threads no longer apply, because the PR no longer changes ADR 0034. Please resolve #4535 (comment) and #4535 (comment). The other two threads are fixed at this head (failed create() recovery and the package-name fallback). Please resolve #4535 (comment) and #4535 (comment).

I did not run the tests, so the affected and full check counts in the description are unverified. CI on this head was cancelled for 13 jobs, which looks like a superseded stacked push. One Smoke Tests job failed in run 43662183448. I did not open its log, but it looks unrelated: resolveReplayExecution has no production caller at head, and the daemon uses only resolveReplayBackend. There are no conflicts. Before merge, please re-run CI to completion on 61ae277, so the cancelled lint, typecheck, coverage, integration and mutation gates finish.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant