Write and rehearse delegated-management incident, recovery and exit runbooks

[ShalomMaman]

Parent epic: #41

Dependencies: #43, #44, #45, #46, #47, #50 and #51.

Goal

Make remote management operable and recoverable without relying on undocumented developer knowledge or weakening the device's local safety path.

Required runbooks

Document decision trees, required authority, evidence capture, customer communication and recovery verification for at least:

• management-service or identity-provider outage;
• unreachable device and stale verified state;
• stuck, repeatedly failing or superseded policy revision;
• compromised owner or delegated-administrator account;
• stolen session or invitation;
• compromised/revoked device identity;
• suspected remote-policy signing-key compromise;
• update-metadata or APK-signing incident and boundary with Freeze production identity and external signing operations #25;
• service/operator compromise or unauthorized insider access;
• mistaken lockout-prone change and local break-glass recovery;
• lost/replaced device, reprovisioning and ownership recovery;
• rollback by issuing a higher corrective revision, never by downgrade;
• breach response, audit preservation and required notifications;
• controlled feature shutdown;
• customer data export, deletion and service exit/migration to another management model.

Operational requirements

• Define who may invoke each procedure and where recent strong authentication or dual approval is required.
• Keep private keys, credentials, PINs, recovery codes and customer payloads out of tickets, chat, logs and ordinary artifacts.
• Distinguish server-side action from device-confirmed recovery.
• Preserve last-known-good device policy through service outages wherever the protocol permits.
• Ensure the local recovery path remains documented and tested until remote recovery has independent supported-device evidence.

Acceptance criteria

• Every incident class has detection signals, containment, recovery, verification and communication steps.
• Compromise procedures distinguish account, session, invitation, device key, policy key, update key and APK signer.
• Rollback is implemented as a higher signed corrective revision with clear supersession evidence.
• At least one tabletop exercise covers account compromise and one controlled drill covers stuck-policy recovery.
• Exit/migration procedures include customer export, device behavior during shutdown and deletion verification.
• A second operator can execute the documented procedures without ad hoc repository knowledge.
• Runbook evidence contains no secrets or customer content.

Non-goals

• Depending on remote recovery before local recovery is proven on supported hardware.
• Publishing security-sensitive secrets or exact emergency credentials in the public repository.

Hebrew summary

יש להכין ולתרגל נהלים לתקלת שרת, חשבון או מפתח שנפרץ, מדיניות שנתקעה, שחזור, החלפת מכשיר וסגירת השירות. כל תיקון נשלח כגרסה חתומה גבוהה יותר, והצלחה נקבעת רק לאחר אישור מהמכשיר.