Automatic desktop recovery of pending tool operations

HANDOFF.md

@@ -40,7 +40,23 @@ citations and unsuccessful evidence. The full milestone sequence is in
 
 ## Current continuation state
 
-**Active isolated M4 increment:** recover a pending compiler/video operation when
+**Active isolated M4 increment:** automatic desktop recovery of a pending
+compiler/video operation after worker loss. `DesktopComposition` now detects
+a non-zero worker exit with one pending effect whose exact terminal result is
+held by the live tool host, exports `CompilerHost.terminal_receipt`, runs the
+ordinary `reconcile` command and launches one replacement worker that keeps the
+task, session, original deadline, counters, reopen context and durable budget.
+It passed the complete [1,595-test local core suite](provenance/automatic-tool-recovery/local-validation-2026-11-08.json)
+and all 16 remaining workflow commands after a tests-only correction (the
+failed first run is retained). The [installed SDK trial](provenance/automatic-tool-recovery/installed-worker-loss-2026-11-08.json)
+passed 21 closed-review checks, and the [physical IQ3_S foreground trial](provenance/automatic-tool-recovery/physical-foreground-recovery-2026-11-08.json)
+passed 49 closed-audit checks: Explorer F5 parked inference, NVENC ran, the
+first worker exited with code 86, and the host recovered automatically. Draft
+[PR #39](https://github.com/midhatn/adaptive-inference/pull/39) is stacked on
+[PR #38](https://github.com/midhatn/adaptive-inference/pull/38). Next: active-tool
+interruption, host loss, observer shutdown diagnosis and M5 restart/reboot.
+
+**Preceding increment (PR #38):** recover a pending compiler/video operation when
 the tool host has retained its exact terminal result but the worker disappeared
 before recording completion. Reuse the existing explicit reconciliation path,
 native lease and task journal; unknown outcomes remain unresolved. This new

PLAN.md

@@ -998,6 +998,19 @@ verified host export and reconciliation path, then qualify automatic SDK recover
 and unchanged-IQ3_S recovery. Interruption during active tool execution remains a
 subsequent gate.
 
+**Automatic desktop recovery result:** `DesktopComposition` now runs that verified
+export, reconcile and replacement sequence itself, through a narrow recovered-state
+lifecycle entry that keeps the task, session, original deadline, counters, reopen
+context and durable budget. The first complete local suite found two test-double
+errors (bare `object()` hosts without `lifecycle()`); tests were corrected, runtime
+unchanged, and the failed run retained. The [local validation record](provenance/automatic-tool-recovery/local-validation-2026-11-08.json)
+then passed 1,595 core tests and all 16 remaining workflow commands. The
+[installed SDK trial](provenance/automatic-tool-recovery/installed-worker-loss-2026-11-08.json)
+(21 closed-review checks) and the [physical IQ3_S foreground trial](provenance/automatic-tool-recovery/physical-foreground-recovery-2026-11-08.json)
+(49 closed-audit checks, two earlier failed attempts retained) both completed with
+automatic recovery after worker exit 86. Next: interruption during an active tool,
+host loss, observer shutdown diagnosis and the M5 restart/reboot gates.
+
 **Validation adjustment:** the [original video CI](provenance/video-tool/hosted-ci-timeout-2026-11-02.json)
 closed with all 12 core-test steps passing, but two final distribution checks
 were cancelled at the whole-job 15-minute limit. Preserve those failures. Give

REFERENCES.bib

@@ -1124,3 +1124,10 @@ @misc{ffprobeFixedVideoOracle
   url = {https://www.ffmpeg.org/ffprobe.html},
   note = {Inspected 2026-11-02 with https://www.ffmpeg.org/ffmpeg-all.html and https://github.com/FFmpeg/FFmpeg/blob/master/doc/ffprobe.texi. Reuses count_frames, select_streams, show_entries and JSON output for the existing exact H264 1280x720 90-frame oracle, followed by full decode. FFmpeg and FFprobe remain external pinned executables. Build-dependent licensing is recorded at https://ffmpeg.org/legal.html; no FFmpeg code or binary is bundled}
 }
+
+@misc{erlangOtpSupervision,
+  author = {{Ericsson AB and Erlang/OTP contributors}},
+  title = {Supervisor Behaviour, Erlang/OTP System Documentation},
+  url = {https://www.erlang.org/doc/system/sup_princ.html},
+  note = {Inspected 2026-11-08 for automatic desktop tool recovery. Design reference only for a supervisor-owned, bounded replacement of a failed worker (restart intensity). Implemented through this repository's existing retained-child lifecycle, task journal and reconcile command; no Erlang/OTP code copied}
+}

docs/evidence.md

@@ -179,6 +179,56 @@ Provider replies and resource observations were scripted; this gate does not
 establish automatic desktop recovery, active-tool interruption, actual-IQ3_S
 recovery or computer reboot continuation.
 
+## Automatic desktop recovery of a pending tool operation
+
+`DesktopComposition` now detects the pending-effect worker-loss boundary itself
+and runs the verified export, reconcile and replacement sequence described in the
+[video recipe](video-tool-recipe.md#automatic-desktop-recovery). Only three runtime
+modules changed from the published pending-effect increment (`composition.py`,
+`desktop_controls.py`, `task_handover.py`); the
+[source-preservation record](../provenance/automatic-tool-recovery/source-lineage.json)
+binds them and three modified test modules to `9242434`.
+
+The first complete local suite run found **2 errors in 1,595 tests**: two existing
+launch tests used a bare `object()` as the tool host, and the new binding-reuse
+guard calls `compiler_host.lifecycle()`. Production hosts always provide it. The
+test doubles were corrected to report a bound host, tests only; that failed run is
+retained. The complete suite then passed **1,595 tests with no failures, errors or
+skips** in 867.39 seconds, and all 16 remaining workflow commands passed, including
+90 provenance tests ([local validation record](../provenance/automatic-tool-recovery/local-validation-2026-11-08.json)).
+
+### Installed SDK automatic recovery
+
+The [installed SDK trial](../provenance/automatic-tool-recovery/installed-worker-loss-2026-11-08.json)
+ran the installed composition with the real Hermes SDK and one real NVENC
+operation. The first worker exited with code 86 after the host finished the video
+and before the task journal recorded it. With no operator action, the composition
+ran `reconcile` (exit 0, no provider call) and `worker-2` (exit 0), which resumed
+the same task under the same deadline with a fresh owner and completed it after two
+cumulative provider requests. Independent closed review passed **21 checks** with
+**48 original case files unchanged**. Provider replies and resource observations
+were scripted.
+
+### Physical IQ3_S foreground recovery
+
+The [physical trial](../provenance/automatic-tool-recovery/physical-foreground-recovery-2026-11-08.json)
+repeated this with real IQ3_S inference on the default desktop and unchanged quality
+settings. An Explorer `This PC` refresh (F5) during the first request drained and
+parked inference. After return, the host ran the fixed NVENC clip, the first worker
+exited with code 86, and the desktop host reconciled and launched `worker-2`
+automatically. A second native owner served the second request, which consumed
+the reconciled video feedback; the task was accepted and stopped normally. The
+independent closed audit passed **49 checks** with all original files unchanged
+(166 foreground samples, one action, one video operation, empty job accounting).
+
+Two earlier attempts are retained as failures. In the first, the F5 came 1.9 s into
+the request, so the 60 s input-idle window expired 2 s before `parked` and the driver
+timed out. In the second, cold parking correctly refused to start because the engine
+active pointer still named the owner terminated by that timeout. The independent
+observer again failed at shutdown (`three_connection_errors_after_response`); that
+diagnosis remains a separate gate. This is one bounded workflow, not repeated
+reliability, performance, restart or reboot evidence.
+
 ## Original video CI timeout and correction
 
 The [original hosted CI record](../provenance/video-tool/hosted-ci-timeout-2026-11-02.json)

docs/video-tool-recipe.md

@@ -173,6 +173,35 @@ host-reporting pattern also draws on
 [Temporal's asynchronous activity completion](https://docs.temporal.io/develop/python/activities/asynchronous-activity);
 it is implemented through the existing local task journal and host ownership.
 
+### Automatic desktop recovery
+
+The desktop composition now performs that same sequence itself. When the
+retained worker exits non-zero while its checkpoint holds one pending effect
+and the live tool host has the exact joined terminal result, `DesktopComposition`
+joins the worker, exports and saves `CompilerHost.terminal_receipt`, retires the
+old binding and runs the ordinary `run_recovery reconcile` command as a retained
+child. It then requires a cleanly repaired `ready` checkpoint and launches one
+replacement worker through a narrow recovered-state lifecycle entry, which keeps
+the task, native session, original deadline, request counters, reopen context and
+durable execution budget. The replacement is committed into the current handover
+episode, so Stop and cleanup still join every child.
+
+Recovery is refused after a clean or Stop-requested exit, when the host is
+closing or faulted, without a pending effect or bound host, within 5 seconds of
+the original deadline, at the attempt limit, or for a second recovery of the same
+worker and operation. A failed reconcile or unclean checkpoint stops recovery
+rather than dispatching around it. The bounded-restart shape follows
+[Erlang/OTP supervision principles](https://www.erlang.org/doc/system/sup_princ.html).
+
+The [local validation record](../provenance/automatic-tool-recovery/local-validation-2026-11-08.json)
+retains 41 focused component tests, the complete 1,595-test core suite, all 16
+remaining workflow commands, package verification and an isolated installation.
+The [installed SDK trial](../provenance/automatic-tool-recovery/installed-worker-loss-2026-11-08.json)
+(scripted provider) and the [physical IQ3_S foreground trial](../provenance/automatic-tool-recovery/physical-foreground-recovery-2026-11-08.json)
+both completed with automatic recovery after worker exit 86 and passed independent
+closed reviews of 21 and 49 checks. Loss of the host, interruption during a running
+tool, application restart and reboot remain separate gates.
+
 ## Sources and licensing
 
 The commands and media oracle reuse this repository's `protected_video` work;

provenance/automatic-tool-recovery/README.md

@@ -0,0 +1,43 @@
+# Automatic tool recovery source preservation
+
+The source-lineage record binds automatic desktop tool recovery to the exact
+published pending-effect increment at `9242434991f2c2c21461061d61d898649299b847`.
+Eight complete predecessor Git blobs preserve the three changed runtime modules
+(`composition.py`, `desktop_controls.py`, `task_handover.py`), three modified
+test modules (desktop composition, plus the compiler and video launch tests
+whose inert host doubles now expose `lifecycle()`), and two source-lineage files. They remain data
+under this repository's Apache-2.0 license; tests do not import or execute the
+retained copies.
+
+The test-only view reuses [pending tool recovery's source-preservation mechanism](../../tests/pending_tool_lineage.py)
+and the [new layer](../../tests/automatic_tool_recovery_lineage.py). Complete-file
+SHA-256 and a portable Python syntax-tree fingerprint must both identify a
+declared successor before projection to its exact predecessor. Unknown complete
+bytes, including an added comment, do not earn that projection.
+
+The complete 77-module runtime is checked before the preceding pending-effect,
+recipe, video and compiler views run. Existing projected views remain idempotent.
+The 24 original pending-effect, recipe, video and compiler JSON records keep
+their original bytes, source identities and qualification limits.
+
+This preserves the meaning of historical evidence. It does not qualify the new
+automatic recovery behavior: component checks, the complete local suite,
+distribution, installed-agent recovery and physical IQ3_S tests remain separate
+gates.
+
+## Evidence records
+
+- [local-validation-2026-11-08.json](local-validation-2026-11-08.json): 41
+  focused component tests, the complete 1,595-test local core suite, all 16
+  remaining workflow commands (including 90 provenance tests), 77-module package
+  verification and a fresh isolated host/worker installation. The first full
+  core attempt (two test-double errors) is retained as failed.
+- [installed-worker-loss-2026-11-08.json](installed-worker-loss-2026-11-08.json):
+  installed Hermes SDK trial in which the desktop composition automatically
+  recovered a pending NVENC operation after worker exit 86; scripted provider.
+  Independent closed review passed 21 checks with 48 case files unchanged.
+- [physical-foreground-recovery-2026-11-08.json](physical-foreground-recovery-2026-11-08.json):
+  one real IQ3_S task with an Explorer foreground action, parked inference, one
+  NVENC operation, worker loss and automatic recovery. Independent closed audit
+  passed 49 checks with all original files unchanged. Two earlier unsuccessful
+  attempts and the separate observer shutdown failure are retained.