Skip to content

Automatic desktop recovery of pending tool operations - #39

Merged
midhatn merged 3 commits into
mainfrom
feat/automatic-desktop-tool-recovery
Nov 9, 2026
Merged

midhatn merged 3 commits into
mainfrom
feat/automatic-desktop-tool-recovery

Conversation

@midhatn

@midhatn midhatn commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Built on #38. Validated development integration, 9 November 2026; the existing v0.4.0a1 release assets retain their original scope.

What this adds

  • DesktopComposition automatically recovers a pending compiler/video operation after worker loss. On a non-zero worker exit with one pending effect whose exact terminal result the live tool host still holds, it:
    1. joins the worker;
    2. exports and saves CompilerHost.terminal_receipt;
    3. retires the old binding and runs the ordinary run_recovery reconcile command (no provider call);
    4. requires a clean ready checkpoint;
    5. launches one replacement worker through a narrow recovered-state lifecycle entry.
  • The replacement keeps the task, native session, original deadline, counters, reopen_context and the durable execution budget.
  • Recovery is refused after a clean or Stop-requested exit, on a closing or faulted host, near the deadline, at the attempt limit, or for a repeat recovery of the same worker and operation.
  • Only three runtime modules change: composition.py, desktop_controls.py and task_handover.py. Source lineage is in provenance/automatic-tool-recovery/source-lineage.json.

Validation (local Windows, Python 3.11.9)

  • Core suite: the first full run had 2 errors in 1,595 tests. Two existing launch tests used a bare object() as the tool host, which has no lifecycle(). I fixed the test doubles (tests only, runtime unchanged) and kept the failed run. The rerun passed 1,595/1,595.
  • Workflow: all 16 remaining workflow commands passed, including 90/90 provenance tests. I reran them on the final tree with docs included, and they passed again.
  • Package: 1,421 source files and 77 wheel modules verified. The wheel members are byte-identical to the build used in the installed trials.
  • Installed SDK trial (scripted provider): worker exit 86 → automatic reconcile → worker-2 completed the same task. Independent closed review: 21/21 checks, 48 case files unchanged.
  • Physical IQ3_S trial, with quality settings unchanged:
    • Explorer F5 during request 1 parked inference.
    • The host ran the NVENC clip.
    • Worker exit 86 → automatic recovery → a second native owner served request 2, which used the recovered video feedback.
    • The task was accepted and stopped normally.
    • Independent closed audit: 49/49 checks, all original files unchanged.
    • Two earlier failed attempts and the separate observer shutdown failure are kept and disclosed.
  • Publication review: 52/52 checks (hashes, links, sanitization, runtime identity).

Evidence: provenance/automatic-tool-recovery/{local-validation,installed-worker-loss,physical-foreground-recovery}-2026-11-08.json. Docs: HANDOFF.md, PLAN.md, docs/evidence.md, docs/video-tool-recipe.md.

Hosted validation

The original attempt-1 push workflow and PR workflow passed at 08053a00d92c065b88bda3ce14f80a7cfe078f0f: all 14 jobs, 12 core-test steps and 12 distribution checks. The final condition-polling test correction does not alter the installed, physically tested runtime.

Remaining qualification

Not qualified: interruption during an active tool, host loss, repeated reliability, application restart and reboot.

…er loss

Adds automatic worker-loss detection in DesktopComposition, host terminal receipt export, bounded reconcile and replacement worker that keeps reopen context and the durable execution budget. Includes source-lineage records and focused tests.

Not yet validated for publication: the local full core run (1595 tests) reported 2 errors in test_compiler_composition and test_video_composition. compiler_launch_args now calls compiler_host.lifecycle(), which the existing test doubles do not provide. No public evidence records or doc updates are included.
Fix two launch tests whose inert tool-host doubles lacked lifecycle(); tests only, all 77 runtime modules unchanged. Both modules are added to the automatic-tool-recovery source lineage.

Add sanitized evidence: local validation (1,595 core tests, 16 workflow commands with 90 provenance tests, package and isolated install; the failed first core run is retained), the installed SDK automatic recovery trial (21 closed-review checks) and the physical IQ3_S foreground recovery trial (49 closed-audit checks; two earlier failed attempts and the observer shutdown failure retained). Update HANDOFF, PLAN, evidence and video recipe docs, and cite Erlang/OTP supervision as a design reference.
@midhatn midhatn changed the title WIP: Automatic desktop recovery of pending tool operations Automatic desktop recovery of pending tool operations Nov 8, 2026
On virtualized Windows CI runners, a fixed 50ms sleep was too short under load, causing test_it_m2_01_automatic_recovery_compiler_harness to fail before interact() completed worker replacement. Replace with a bounded polling loop (up to 2.0s, polling every 10ms) matching adjacent tests.

Updates source lineage and the three sanitized evidence records with the new lineage SHA. All 77 runtime files remain identical.
Base automatically changed from feat/pending-tool-recovery to main November 9, 2026 03:07
@midhatn
midhatn marked this pull request as ready for review November 9, 2026 03:07
@midhatn
midhatn merged commit f362ed0 into main Nov 9, 2026
14 checks passed
@midhatn
midhatn deleted the feat/automatic-desktop-tool-recovery branch November 9, 2026 03:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant