Fix task executor worker identity reconciliation races

[Andyz26]

Summary

• reserve the accepted worker identity before acknowledging task submission and report that identity while preparation is in progress
• classify assignment failures as not sent, possibly sent, or conflicting so ambiguous executors remain fenced
• reconcile heartbeat and status reports without discarding worker ownership, including across disconnect/reconnect
• dispose prepared tasks and their classloaders when cancellation or shutdown races with preparation
• acknowledge cancellation when accepted while keeping teardown asynchronous, settling stop failures, and bounding shutdown waits

State transition analysis

Agent task ownership

flowchart LR
    I[Idle] -->|submit A: reserve before Ack| P[Preparing A]
    P -->|initialization succeeds| R[Running A]
    P -->|cancel: mark and Ack| C[Cancelling A]
    R -->|cancel: mark and Ack| C
    P -->|shutdown| C
    R -->|shutdown or completion| C
    C -->|task and classloader disposed| I
    P -->|submit B| X[Reject with actual worker A]
    R -->|submit B| X

Loading

The reserved slot is the single source of worker identity for submission checks, heartbeats, cancellation, and cleanup. A second assignment cannot observe the executor as idle while the first task is preparing. Cancellation Ack confirms acceptance rather than teardown completion; Available is reported only after cleanup. Stop failures settle the internal completion future, and shutdown waits are bounded by the configured RPC timeout.

Control-plane reconciliation

flowchart TB
    P[Pending] -->|scheduler assigns A| A[Assigned A]
    A -->|Occupied A| R[Running A]
    R -->|authoritative Available status| P

    R -->|Available heartbeat; capable agent| V[Verifying A]
    V -->|second Available heartbeat| P
    V -->|Occupied worker| Q[Quarantined reported worker]

    A -->|Occupied B| Q
    R -->|Occupied B| Q
    Q -->|cancel reported worker| Q
    Q -->|authoritative Available status| P

    R -->|Available heartbeat; legacy agent| L[Legacy quarantine]
    L -->|authoritative Available status or capability upgrade| P

Loading

Only a registered Pending executor enters the scheduling index. Ambiguous ownership remains fenced. Reservation-capable agents can prove that repeated Available heartbeats are safe; legacy agents require an authoritative status change or upgrade.

Assignment failure direction

flowchart TB
    F[Assignment attempt] --> G{Did submit begin?}
    G -->|No| N[NotSent]
    N -->|retry budget remains| RF[Retry with fresh gateway]
    N -->|attempt expired| B[Block late gateway submission]
    G -->|Yes or unknown| M[MayHaveRun]
    M --> Q[Fence executor and cancel expected worker]
    F -->|AlreadyRunning actual worker B| C[Conflict]
    C --> QC[Quarantine and cancel B]
    F -->|failure from older assignment epoch| S[Ignore stale failure]

Loading

Tests

./gradlew :mantis-control-plane:mantis-control-plane-server:test :mantis-server:mantis-server-agent:test

The tests cover preparation-time ownership, prompt cancellation acknowledgement, stop failure cleanup, bounded shutdown, conflicting assignments, stale and mismatched reports, capable and legacy recovery, delayed assignment failures, late gateway completion, and disconnect/reconnect state preservation.

[github-actions]

Test Results

845 tests  +22   834 ✅ +22   10m 11s ⏱️ -7s
167 suites ± 0    11 💤 ± 0 
167 files   ± 0     0 ❌ ± 0 

Results for commit eadc133. ± Comparison against base commit 9ff3102.

♻️ This comment has been updated with latest results.