[fix][video-gateway] harden discovery recovery, task admission and worker retries

[DJSaa]

Overview:

Improve the resilience of etcd-backed Video Gateway deployments and detached vLLM-Omni workers under discovery stalls, Runtime termination, admission contention, and worker loss.

Details:

Changes

• Detect stale discovery watches and compare the Runtime worker view with etcd.
  Coordinate recovery across Gateway replicas using an etcd lease lock while
  preserving graceful drain behavior.
• Exit detached video workers after unexpected permanent Runtime termination,
  allowing the supervisor to restart them instead of leaving stale processes.
• Harden concurrent task admission and cancellation against etcd CAS races.
• Add bounded, persistent retry accounting and a retry waiting queue for one
  retry after confirmed worker loss or known execution-engine unavailability.
• Independently check worker liveness while waiting for results, avoiding
  indefinite waits when the result stream does not report worker loss.
• Keep parameter, media, model, and unknown execution errors non-retryable.
• Add targeted tests and recovery/retry telemetry.

Compatibility

• Discovery watchdog and worker retry are opt-in.
• The Runtime termination guard applies only to detached video workers.
• No changes to ordinary non-video worker startup paths.

Validation

Historical validation of the corresponding image/source tree:

• 227 unit and real-etcd contract tests passed.
• 14 retry strategy scenarios and 6 cancellation-log scenarios passed.
• 14 regression groups passed through per-case acceptance, including retests:
  API behavior, concurrent creation/cancellation, Gateway failover, etcd
  outages, and worker-loss retry.

These results combine the initial run and corrective retests; they do not
represent a single clean full run or production capacity/SLO certification.

Included commits

• 3ca3302 — recover stale discovery watches safely
• 6954663 — restart detached workers after Runtime termination
• ce0a802 — harden admission and retry lost workers

Related Issues

🚫 This PR is NOT linked to an issue:

• Confirmed — no related issue

[github-actions]

👋 Hi DJSaa! Thank you for contributing to githubgxll/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀