Skip to content

Developer SSH listener loses projected key during concurrent boot reconciliation #641

Closed

Description

@boburning

Observed during the first controlled boot of signed Mini Plus candidate 0.1.0-dev-55e0664b.

Evidence

  • Wi-Fi associated and TCP/22 served the pinned Dropbear host key.
  • The server allowed public-key authentication only but rejected the retained automation Ed25519 key before signature.
  • Bloom Remote Keys reported the same durable key fingerprint ending b9002791.
  • The installed bloom-dev-ssh and bloom-dropbearmulti bytes are identical to the previously working 2933f70 release.
  • No second reboot, Wi-Fi mutation, key mutation, host-key bypass, password authentication, or alternate identity was attempted.

Root cause
Boot reconciliation has two SSH lifecycle owners. bloom-network calls the compatibility services path, which starts bloom-dev-ssh in the background, then immediately calls bloom-network-services reconcile, which starts the same helper again. Concurrent starts can race stop_dropbear and prepare_runtime_key. On firmware that needs Bloom's /home bind-mount fallback, the losing starter can unmount /home after the surviving Dropbear is listening. The process remains reachable but its authorized_keys path has disappeared.

Required correction

  • Make bloom-network-services the sole SSH/SFTP lifecycle owner.
  • Serialize primary bloom-dev-ssh start and stop operations with a bounded, stale-owner-recoverable lock.
  • Add a concurrent-reconcile regression test that models exclusive listener ownership.
  • Preserve key-only authentication, the sole durable automation key, strict host-key pinning, and the network-transaction shadow listener.
  • Revalidate with focused and exhaustive shell tests, affected ARM/package CI, and one controlled signed Mini Plus boot before confirmation.

Hardware status
The candidate remains unconfirmed. Its retained known-good release and pre-update snapshot are intact. Physical display, controls, shutdown behavior, gameplay, and save semantics are not established by this SSH failure evidence.

Activity

  1. boburning commented on Aug 31, 2026

    @boburning
    OwnerAuthor

    Corrective implementation is published in #642. The final head 0b3b5d9 removes the duplicate compatibility owner, serializes primary SSH lifecycle operations, and adds explicit operator rejection of an unconfirmed testing candidate so signed recovery does not require repeated failed boots. Latest PR shell CI is green in 2m24s; signed ARM hardware build and preserved-card inspection remain in progress.

  2. boburning commented on Aug 31, 2026

    @boburning
    OwnerAuthor

    Offline recovery evidence and exact repaired runtime hashes are recorded on PR #642. The card was repaired from a clean, hash-preserved backup, remained unconfirmed at testing boot attempt 1, and was safely ejected. Awaiting one guarded boot and strict-SSH validation; no merge yet.

  3. boburning commented on Aug 31, 2026

    @boburning
    OwnerAuthor

    The final signed PR #642 candidate passed the guarded Mini Plus gate and is confirmed known-good as 0.1.0-dev-0b3b5d96. Strict key-only SSH survived the first boot, listener audit and aggregate health are clean, exact installed hashes match the PR, rollback 2933f706 is retained, and temporary transfer/installer files are cleaned. Proceeding to final diff/check review and merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions