Skip to content

ci: keep only core CI, release and deploy; fix deploy activation on containerd hosts - #569

Merged
cunoe merged 5 commits into
mainfrom
fix/actions-reliability
Nov 13, 2026
Merged

cunoe merged 5 commits into
mainfrom
fix/actions-reliability

Conversation

@cunoe

@cunoe cunoe commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

  1. Deploys cannot activate. Every real activation since fix: verify Nightly sources, deployment upgrades and npm services #539 fails with imported image differs from the pinned image ID (Deploy / Product services, 2026-11-13, node and app). The pin is the configuration digest from the build runner's classic Docker store. The host runs Docker 29.7.2 on the containerd image store, where docker load reports the manifest digest instead: the host loaded sha256:768befb6…, whose manifest references config sha256:d6d074f6…, which is the pin. The self-test mocked docker load with the pinned ID, so the mismatch only showed up on the host. The last real activation (node, 2026-11-10) predates fix: verify Nightly sources, deployment upgrades and npm services #539, and the 2026-11-11 run skipped activation.
  2. Too many Actions, too much failure noise, slower iteration. Across 2026-11-07..13 most failing runs came from manual acceptance and diagnostic entries: CI / Mobile acceptance failed 29 of 46 runs, and the desktop and Windows platform jobs failed 23–76% of the time. We want iteration speed and only core CI/CD. iOS acceptance in particular is not useful.

Change

  • Deploy activation (scripts/deploy-activate.py): before docker load, read back manifest.json, index.json and their blobs and check every digest, proving the archive holds exactly the pinned image. After loading, accept only the pinned ID or the manifest digest proven to reference it, then activate under the ID the host's store reports. deploy-run already records that ID separately from prepared_image. The self-test now builds a real OCI-layout archive and covers both stores, plus tampered, multi-image and unrelated-ID archives. scripts/deploy-run.py runs the same archive proof right after docker save, so a format change fails on the runner, not on the host.

  • Workflows: Actions keeps ci.yml (CI / Core) with _ci-crypto.yml and _ci-web.yml, every release* workflow, hourly.yml, npm-publish.yml and every deploy* workflow. Removed:

    • ci-desktop-acceptance.yml and ci-desktop.yml
    • ci-windows.yml and ci-windows-native.yml
    • mobile.yml and ci-mobile-device.yml
    • ci-native-diagnostics.yml and native-installer-diagnostic.yml
    • delivery-small.yml and delivery-live.yml

    Two scripts that only those entries ran are also removed: scripts/native-desktop/installer.ps1 and diagnostic-inputs.py.

  • Self-tests: scripts/ci.py and scripts/check-delivery.py drop only the assertions about the deleted files. Every Core, reusable and release contract stays.

  • Docs: packages/docs/ci.md, ios.md, e2e.md, the delivery proof README and AGENTS.md (the "installer or platform changes must trigger extended CI" rule) no longer point to the removed entries. Local e2e, desktop window tests and the delivery host commands still run on demand.

Withdrawn and not in this PR: a local Windows cross-lint preflight check (added, then reverted in this branch) and flaky-test fixes.

Left as is, possible follow-up cleanup:

  • ci.yml keeps its now-unused workflow_call / package_policy=linux-native path.
  • check-delivery.py --prove-small relation and accept items require an Actions source and are unavailable until an entry is restored. Local prepare and run-delivery-live.py still work.

Validation

  • Local checks. These pass:
    • Every quick-check command from ci.yml except actionlint, shellcheck and atlas, which are not installed here. This includes scripts/ci.py --self-test and check-delivery.py --self-test.
    • Deploy, runtime, static-config and check-release-workflow self-tests.
    • offline-docs.py --self-test.
    • vibeface-www tests (5/5) and portal typecheck/build/check:static (22 pages, anchors).
    • check-delivery.py --check-fixtures.
  • Deploy host (read-only checks). Python 3.11.2, Docker 29.7.2 with io.containerd.snapshotter.v1, and the manifest loaded on 2026-11-13 references the pinned config.
  • Remote. CI / Core run 43445879915 on 9d68dc53: Workflow and source checks (actionlint, shellcheck, atlas, ci.py/deploy/check-delivery self-tests), Static portal and Mobile types passed; the Rust and package jobs are still running and this PR changes no Rust code. The Deploy / Node dry run (apply=false) waits for the first complete Hourly release, because Nightly tags are no longer accepted after feat(release): replace the four-hour Nightly with an hourly product release #566.
  • Not run. The Docker sandbox delivery preflight (no local Docker daemon) and the full preflight skill.

Status: ready for review. Merging and the real deploy need explicit authorization.

Every real activation since #539 failed with "imported image differs from
the pinned image ID". The pin is the configuration digest from the build
runner's classic Docker store, but the deployment host runs Docker 29 on
the containerd image store, where `docker load` reports the manifest
digest instead (sha256:768befb6... for pin sha256:d6d074f6... on
2026-11-13). The self-test mocked `docker load` with the pinned ID, so
the mismatch only surfaced on the host.

Before loading, the host now proves the archive holds exactly the pinned
image by reading manifest.json, index.json and their blobs back and
checking each digest. After loading it accepts only the pinned ID or the
manifest digest proven to reference it, and activates under the ID the
host's store reports, which deploy-run already records separately from
the prepared ID. The build step runs the same archive proof right after
`docker save`, so a format change fails on the runner, not on the host.
Windows-only Clippy and compile errors were the largest group of code
failures in CI (37 failed runs in 2026-11-07..13): Windows scopes failed
on lints in winsec.rs, platform/discovery.rs and the CLI service while a
Linux checkout never compiles that code, and main pushes no longer run
Windows Clippy at all.

`ci.py check --windows-cross` runs the same Clippy invocation for
x86_64-pc-windows-gnu. No crate distinguishes GNU from MSVC, so it covers
the cfg(windows) code the native job lints; ring, libsqlite3-sys and
blake3 still need MinGW-w64, and a missing compiler fails with install
guidance instead of skipping. A new preflight domain check runs it for
the native Windows job's packages and is deferred when that head already
has a passing "Windows native gate".

On commit 63fb451, which failed CI's Windows bridge lint, this
reproduces the same three errors at discovery.rs:302, :318 and :351;
current main passes.
Iteration speed matters more than the manual acceptance and diagnostic
entries, which also produced most of the failing runs (CI / Mobile
acceptance failed 29 of 46 runs in 2026-11-07..13, the desktop and
Windows platform jobs 23-76%). Actions now keeps CI / Core with its
reusable crypto and web workflows, every release workflow and every
deploy workflow.

Removed: CI / Desktop and its reusable desktop acceptance, CI / Windows
native and its reusable checks, CI / Mobile builds, CI / Mobile device
builds, CI / Native diagnostics with the Windows installer diagnostic,
and the Delivery small and live proof experiments. The installer script
and diagnostic input checker that only those entries ran are removed too.

ci.py and check-delivery.py self-tests drop the assertions about the
deleted files and keep every Core, reusable and release contract. The CI
guide, iOS and e2e docs, the proof README and AGENTS.md no longer point
to the removed entries; local e2e, desktop window tests and the delivery
host commands still run on demand.
@cunoe cunoe changed the title fix(ci): stop recurring Actions failures in deploys, Windows lint and flaky tests ci: keep only core CI, release and deploy; fix deploy activation on containerd hosts Nov 13, 2026
@cunoe
cunoe marked this pull request as ready for review November 13, 2026 14:49
@cunoe
cunoe merged commit f9b587c into main Nov 13, 2026
8 of 16 checks passed
@cunoe
cunoe deleted the fix/actions-reliability branch November 13, 2026 14:51
cunoe added a commit that referenced this pull request Nov 13, 2026
…erable (#563)

* ci: save caches only from main

The Actions cache held 11.2-11.6 GB, above the 10 GB limit, mostly in
branch and tag entries no other ref can restore, so main's caches kept
being evicted. All 12 rust-cache steps in Core, the crypto reusable and
the release producers now save only on refs/heads/main; other refs,
including release tags and manual branch runs, still restore main's
caches by key prefix. Updated onto #568 and #569: the changes to the
deleted workflows and the Chromium cache for the removed e2e step are
dropped.

* ci(release): keep the iOS archive as long as every other producer artifact

Drop the 14-day retention override so a failed product release keeps
every original producer artifact for publisher recovery.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant