Skip to content

Stage QA login on the browser host was lost on 2026-11-01 and nothing told the owner; the session file is rewritten in place on a full disk #954

Open

Description

@alex-mextner

What

Observed 2026-11-02: every Stage QA run (for example https://github.com/alex-mextner/HyperCalendarBot/actions/runs/41392714990 for #941, and run 41394026341 for another pull request) ended with STAGE_SMOKE_UNAVAILABLE 'qa-cli is not logged in to Telegram', status error 'could not run: not provisioned', so gh ship refused. On the browser host 65.21.40.51 the qa-cli session state file for the qa-telegram session had shrunk from about 26 KB to 442 bytes (empty cookies, no auth keys) at 2026-11-01 21:24 UTC, and a QR login had been attempted on that host at 20:23 UTC. The same host's root disk was 100% full on 2026-11-02 (1.9 GB available, server-run suites failing with EDQUOT in 888 tests), so a write failure during the in-place rewrite is the likely cause; this is an inference, not proven (no earlier backup of the file exists to compare). A copy of the PC session did not restore the login, so Telegram had also revoked the key. Wanted: (1) the stage QA setup keeps a timestamped backup of the session file and checks it is valid JSON with auth keys before and after each run; (2) a run that finds qa-cli logged out raises a repair-flow alert and a Telegram notice to the owner instead of only a red status; (3) the runbook (https://github.com/alex-mextner/HyperCalendarBot/blob/main/docs/reference/deploy-runbook.md, section Stage QA) says how to log in again on the host and how to free its disk.

Why (motivation)

Stage QA is a required merge gate; a silent logout blocks every pull request until the owner happens to look.

User impact

People waiting for fixes to the bot cannot get them: no pull request can ship while the login is lost, and nobody is told why.

Cost of inaction

Every future disk-full or revoked-key event blocks all merges again with a status that only says not provisioned.

Acceptance criteria

  • Test: a session file that is empty of auth keys makes the browser half report logged-out before any case runs and raise an alert
  • Test: the session file is written through a temporary file and a backup, so a failed write leaves the previous valid copy
  • The runbook Stage QA section lists the re-login steps and the disk-space check on the browser host

Screenshots

(none)

Deploy

Deployed: no

Links

  • (none)

Skipped gates

Activity

  1. alex-mextner commented on Nov 2, 2026

    @alex-mextner
    OwnerAuthor

    Additional observation 2026-11-02: on the browser host, files copied with scp (the sftp write path) arrived as all-zero files of the right size and scp printed 'write remote: Failure', both into the tmpfs temp directory and into the home directory, while the same bytes streamed with 'ssh host cat > file' arrived intact (sha256 verified). So a copy of the session file by scp silently produces an unusable file on this host; any provisioning step (including the re-login or restore steps in the runbook) must stream through ssh and verify a checksum. The root disk is still 100% full (1.9 GB available).

  2. alex-mextner commented on Nov 2, 2026

    @alex-mextner
    OwnerAuthor

    Evidence from PR 940 (head b711de1): Stage QA runs 41368771131 and 41394026341 (attempts 1 and 2) ended 'could not run: not provisioned' with 'qa-cli is not logged in to Telegram'. After the session file on the browser host was replaced from the PC at 05:43 UTC on 2026-11-02, a status check under the browser lock still said logged-out (06:12, 07:03, 08:03, 09:04 UTC). Stage QA run 41392155405 (another head) passed at 05:07 UTC and the next two runs failed within minutes, so the restored session seems to die after about one run. A QR login attempt on the host at 20:23 UTC on 2026-11-01 preceded the first logout. Needs a fresh login on the host by the owner; PR 940 is blocked on this gate.

  3. alex-mextner commented on Nov 2, 2026

    @alex-mextner
    OwnerAuthor

    Cleanup owed after the owner re-logs in: delete the copy file named qa-telegram-qa-telegram.json.copy-20261102T0620Z and the .empty-20261102 backup in the agent-browser sessions directory of the Stage QA host (the copy is a live Telegram login, mode 0600).

  4. alex-mextner commented on Nov 2, 2026

    @alex-mextner
    OwnerAuthor

    Pin change on the Stage QA browser host (65.21.40.51), 2026-11-02 11:48 UTC: qa-cli was upgraded from 875b11a0f3965de54e2015d525720008d8926f2f (qa 0.6.4, the pin in scripts/install-stage-qa-browser.sh) to 3fdd684e39af843b570f31c01f7c14a9272ab4a3 (qa 0.6.6, adds re-entry of the search query when Web A clears it). Why: after the fresh login, Stage QA runs 41420979611, 41392714990 and 41424818802 failed with 'qa send failed: timed out after 10s waiting for search result for @HyperCalendarStageBot' and status not-provisioned although qa telegram status said logged-in. How: under the browser lock, uv tool install --force --python python3 with git+https://git.hyperide.ai/ultrabricks/qa-cli at the new SHA, with the user-level git on PATH. Right after, qa telegram status said logged-in. Roll back with the same command at the old SHA under the browser lock. The follow-up PR will move QA_CLI_REF and its pin test and the runbook text to the new SHA.

  5. alex-mextner commented on Nov 2, 2026

    @alex-mextner
    OwnerAuthor

    Result of the qa-cli upgrade (2026-11-02 12:29 UTC): it did NOT fix Stage QA. Re-run of Stage QA run 41394026341 (head b711de1, qa 0.6.6) again ended 'qa send failed: timed out after 10s waiting for search result for @HyperCalendarStageBot', status not-provisioned, cases=0; the stage bot health check was ok. So the cause is not (only) the missing search re-entry. Open: what the Web A search shows for the stage bot username in the fresh login session on the host (the old session found it from the chat list). The host now runs qa 0.6.6 (rollback: same uv tool install at 875b11a0f3965de54e2015d525720008d8926f2f under the browser lock).

  6. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    State on 2026-11-04 (from PR 940): Stage QA works again. Runs succeeded from 2026-11-02 20:44 UTC; the last one on PR 940's head passed 6 cases (run 41776088442). A read-only 'qa telegram chat open' for the stage bot on the host found the chat. The host's qa-cli is now 0.6.9 (I installed 0.6.6 on 2026-11-02; someone upgraded it later). Drift to fix: scripts/install-stage-qa-browser.sh still pins QA_CLI_REF at 875b11a0f3965de54e2015d525720008d8926f2f (qa 0.6.4) and re-pins on every run, so the next install would downgrade the host to the version that failed to find the stage bot after the fresh login. Next: move the pin, its test and the runbook text to the version that works on the host, and have the installer or driver say when the Telegram session is logged out (the original ticket).

  7. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    Diagnosis from the QA search debug (2026-11-04), kept for the next login loss: with a restored/copied session, qa telegram status said logged-in because it only sees the cached page shell, while Web A never got an authorized connection (chat list = endless spinner, search empty; DC2 socket renegotiated a temporary key about every 3.6 s). Network and NTP were fine. Likely cause: Telegram rejected the restored permanent auth key (unconfirmed). Fix path: owner checks Settings > Devices in the QA account; if the Web session is gone, fresh qa telegram login by QR on the QA host under the browser lock, never copy it back. Optional hardening: make qa telegram status fail when the chat list does not load within about 20 s. Note: Stage QA later passed for PR 940 (run 41776088442), so the login was evidently restored afterwards. Cleanup owed on the QA host: the debug scratch directory qa-search-debug under .local/state (no secrets; the removal guard blocks agents).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions