diff --git a/.agents/skills/release/SKILL.md b/.agents/skills/release/SKILL.md index 5c671cab4..31f2f3ce9 100644 --- a/.agents/skills/release/SKILL.md +++ b/.agents/skills/release/SKILL.md @@ -32,18 +32,22 @@ description: 使用统一产品 Tag 或独立官网 Tag 发布 Vibeface,构建 ## 每 4 小时自动 Nightly -`ci.yml` 的 main schedule 使用 `23 */4 * * *`,每日 UTC 00:23、04:23、08:23、12:23、16:23、20:23 发起扩展检查,在 `ci-gate` 成功后运行 `scripts/daily-nightly.py`,手动 CI 和普通 main push 不运行它。每四小时机械版本 PR 的自动 squash 已授权并默认开启,需配置单仓库 fine-grained `NIGHTLY_RELEASE_TOKEN`,授予 Contents、Pull requests、Actions 所需权限。缺 secret 时明确失败,不挪用站点部署 token,不改保护,不 fallback。 +`ci.yml` 保留 `23 */4 * * *`,UTC 槽位为 00:23、04:23、08:23、12:23、16:23、20:23。GitHub schedule 是 best effort,可能延迟或跳过,不保证每槽运行。仅 main schedule 的 `ci-gate` 成功后运行 `scripts/daily-nightly.py`;手动 CI 和普通 main push 不自动发布 Nightly。 -协调器将原 run 的 `created_at` 转为 UTC,向下取整到 00、04、08、12、16、20 时的批次,用 `bot/daily-nightly/YYYYMMDDHH` 恢复该批次唯一 PR、版本和真实 squash SHA。发布版本日期仍取原 run 的 `YYYYMMDD`,不同批次独立创建 PR 并递增同日序号、Android versionCode 与 iOS buildNumber。即使业务源码未变化,每个批次仍递增产品组和官网 Nightly;各产品组件使用同一版本,保留 marketing version 和依赖解析结果,Android versionCode 与 iOS buildNumber 各递增一次。协调器保持全局串行。版本 PR 只允许机械版本字段、两个 Cargo.lock 本仓包版本,以及这两个原生构建号。iOS 递增 CFBundleVersion 的末段数字,超出 Apple 分段上限时停止,不重置或自动进位。PAT 只用于 PR 创建与普通立即 squash;PAT 没有路径级权限,限制由受信脚本的完整 tree、仓库、作者、base、head 校验执行。普通 GITHUB_TOKEN 处理分支、只创建的 tag 和精确 dispatch。 +协调器只用 `GITHUB_TOKEN`,不需要 PAT,不创建版本分支或 PR,不合入 main。以原 schedule run 的精确 main SHA 作唯一 parent,生成普通机械版本提交,只改七个版本文件中的发行版本、Cargo.lock 两个本仓包版本及 Android、iOS 构建号,保留 marketing version、依赖和其他内容。共享验证器对比完整 tree,并运行 `cargo metadata --locked --no-deps`,不以路径白名单放行。 -合并后仅实际 SHA 的 main push CI success 放行。产品先创建标明构建中的公开 Release;各平台独立完成校验后追加自己的附件,产品与官网运行可并行。`collect` 在后台继续完整审计,最终 `publish-product` 核对全套已公开字节并最后追加 `release.json`,之后才放行自动消费。产品 Nightly 自动发布 npm,Nightly 不发站,也不上传 TestFlight 或 App Store;仍构建 iOS 设备归档,签名配置完整时导出 IPA。失败或响应未知先对账,只从原 run 完整 artifact 重跑失败 publisher,不能重建后覆盖。npm 失败不能当作成功;原产品与 npm artifact 完整时,协调器可重跑原 release run 的失败 job,复用原 npm artifact id。artifact 丢失、过期或存在字节冲突时停止。延迟启动或跨日重跑原 schedule run 仍恢复原批次任务,不另建版本 PR,不重新分配版本。旧批次若尚无分支与 PR,而 main 的产品或官网 Nightly 日期已更晚,则停止初始化,避免版本倒退。不要在人工发版会话中无授权调用此自动协调入口。 +原 run 的 UTC `created_at` 向下取整到四小时批次,版本日期仍取该 run 的 `YYYYMMDD`。提交记录原 schedule run ID、批次和上一可信产品 Nightly 来源。移动构建号从 main 与上一可信 Nightly 取数值上界后加一;iOS 按数字分段比较,末段溢出时停止,不回绕。协调器全局串行。同批次、延迟或跨日恢复保持原版本、构建号、提交与 artifact;上一来源完整发布且 npm 成功、main 无新提交时跳过。部分 Tag、Release 或 npm 必须先恢复原来源,不新建版本覆盖。 + +所有发布入口统一验证 main 来源或精确机械 Nightly。detached 提交仅允许 Nightly;Stable/RC 仍要求 main。机械 Nightly 的 CI 证明必须是原 schedule run 的精确仓库、workflow path、main、parent SHA 和成功 `ci-gate`,不等待包含协调 job 的整个 run success。正常 main Tag 仍要求同一 SHA 的 main push CI success。Tag、制品与 provenance 的 source 始终是实际版本提交,不替换为 parent。 + +产品与官网并行发布,产品最终完整清单和 npm 分别核对。Tag 只创建,冲突立即停止;未知响应先读远端对账,不重复写入。仅从原运行的完整不可变 artifact 重跑失败 publisher;原产品和 npm artifact 完整时可恢复原 npm 发布。缺失、过期或字节冲突时停止。自动 Nightly 不部署、不上传 TestFlight 或 App Store;仍构建设备归档,签名配置完整时导出 IPA。不要在人工会话中无授权调用自动协调入口。 ## 准备并验证版本 1. 在任务 worktree 中同步各产品组件版本及 `Cargo.lock` 的本仓包版本,或只修改独立官网版本。Mobile marketing version 和原生 build number 按目标分发规则处理。准备好的精确版本可直接发布,不为打 tag 再自动递增一次。 2. Web、macOS、Windows、Linux x64、Linux ARM64 原生程序、Android 和 iOS 在同一运行并行构建。桌面 App 只等待本平台完整运行包。每个平台组包只等自己的原生输入和 Web,以不可变 artifact ID 验证同一源码、版本与目标。不得等待或拉取独立公开组件 Release。每组校验通过即可追加并供手动下载;完整产品集通过才追加全局 `release.json`,标记整套发布完成。 3. 在同一 PR 完成相关检查和独立审查,再按已有授权合并。不得直接在 main 上修改、commit 或 push 版本。 -4. 固定包含版本修改的完整 `SOURCE_SHA`。在指向该提交的干净 worktree 中验证;保留主 checkout 的无关修改。提交必须已进入远端 main,版本必须与 tag 完全一致。 +4. 固定包含版本修改的完整 `SOURCE_SHA`。在指向该提交的干净 worktree 中验证;保留主 checkout 的无关修改。人工准备的提交必须已进入远端 main,版本必须与 tag 完全一致;自动机械 Nightly 使用前述共享来源门。 5. 核对该 SHA 的必要检查与 main CI 均已通过。`pending`、缺少检查或只有其他 SHA 的成功记录均不能放行。失败先修复,不用打 tag 探测构建是否能过。 打 tag 前使用现有只读校验入口,`APP` 取 `product` 或 `www`,`VERSION` 取已确认的版本: @@ -71,7 +75,7 @@ git tag -a "$TAG" "$SOURCE_SHA" -m "$TAG" git push origin "refs/tags/${TAG}:refs/tags/${TAG}" ``` -GitHub 的不可变发布 Tag 规则保护产品 `v*`、官网与历史应用 Tag,禁止更新和删除,没有绕过者。创建时的完整格式由预检和 CI 验证,CI 还核对版本文件与 main 归属。规则保护不代替打 tag 前的检查。 +GitHub 的不可变发布 Tag 规则保护产品 `v*`、官网与历史应用 Tag,禁止更新和删除,没有绕过者。创建时的完整格式由预检和 CI 验证,CI 还核对版本文件与共享来源规则。规则保护不代替打 tag 前的检查。 ## 处理现存 tag 与失败 @@ -90,7 +94,7 @@ GitHub 的不可变发布 Tag 规则保护产品 `v*`、官网与历史应用 Ta 官网发布同样不自动部署。部署官方站点时,在本仓库执行 `deploy www <精确官网 Tag>`,与产品服务使用相同的独立部署流水线。核对实际运行版本、镜像和线上页面,不能把 workflow 启动当作上线。 -npm 使用 [npm-publish.yml](../../../.github/workflows/npm-publish.yml),当前仅支持 CLI Nightly。`release.yml` 在产品 `publish-product` 成功后自动调用它,以 `collect` 产生的原不可变 npm artifact ID 验证源码与包,再通过 OIDC 公开发布。npm trust 须分别允许调用方 `release.yml` 与手动入口 `npm-publish.yml`,均绑定 `npm-release` environment。手动入口默认 dry run,实际发布须显式 `publish=true`,不得配置长期 npm token 或索取聊天凭证。配置命令和 registry 对账规则见 [发布说明](../../../packages/docs/releasing.md#npm-nightly-制品与发布);仓库改动不代表远端 trust 已生效或真实 OIDC 发布已验收。 +npm 使用 [npm-publish.yml](../../../.github/workflows/npm-publish.yml),当前仅支持 CLI Nightly。`release.yml` 在产品 `publish-product` 成功后自动调用它,以 `collect` 产生的原不可变 npm artifact ID 验证源码与包,再通过 OIDC 公开发布。npm trust 须分别允许调用方 `release.yml` 与手动入口 `npm-publish.yml`,均绑定 `npm-release` environment。手动入口允许同 Tag、同 SHA 的唯一已完成 release run,只要 `collect` 和 `publish-product` 成功,即使总 run 因 npm 标红。手动入口默认 dry run,实际发布须显式 `publish=true`,不得配置长期 npm token 或索取聊天凭证。配置命令和 registry 对账规则见 [发布说明](../../../packages/docs/releasing.md#npm-nightly-制品与发布);仓库改动不代表远端 trust 已生效或真实 OIDC 发布已验收。 iOS 默认使用 `app-store-connect` 分发方式,先通过 TestFlight 内测,再准备 App Store 发布。签名 IPA 的 `testflight_eligible=true` 只代表导出方式和描述文件通过校验,不代表 Apple 已接收或内测已可安装。TestFlight 上传、Apple 处理、内部测试组分配、App Store 提交和真机验收另行完成;不能把未签名设备归档或 CI 成功报告成可安装的 iOS App。 @@ -100,6 +104,6 @@ iOS 默认使用 `app-store-connect` 分发方式,先通过 TestFlight 内测 手动 `release.yml` 默认 `publish=false`,可从开发分支执行产品或官网完整构建,不要求真实 Tag。该结果只证明构建与验收,不代表公开发布、部署、npm registry 写入或升级已发生。使用 `gh workflow run release.yml --ref -f app=product -F publish=false` 验证实际并行与最终完整 artifact。 -真正发布必须使用现存精确 Tag 作为 workflow ref,显式 `publish=true`;身份检查还要求该源码的 main push CI success。`release-start` 只负责创建或确认一个公开 Release,各平台发布任务仅写自己独占的附件名称,最终 `publish-product` 最后写入全局 `release.json` 和完成状态;生产者与 `collect` 仍只上传 Actions 制品。 +真正发布必须使用现存精确 Tag 作为 workflow ref,显式 `publish=true`;身份检查对 main 来源要求同一 SHA 的 main push CI success,对机械 Nightly 要求原 schedule parent 的成功 `ci-gate`。`release-start` 只负责创建或确认一个公开 Release,各平台发布任务仅写自己独占的附件名称,最终 `publish-product` 最后写入全局 `release.json` 和完成状态;生产者与 `collect` 仍只上传 Actions 制品。 旧 CLI 只识别旧 `cli-v` 格式,首次切换按安装说明手动安装新版完整包,之后由新版跟踪产品 `v`。不创建兼容 Tag、不自动覆盖已有安装或迁移数据。 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d765a8261..bd103e92f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -104,6 +104,8 @@ jobs: if: needs.plan.outputs.rust == 'true' run: atlas migrate validate --dir file://apps/core/migrations - name: Check workflow, selectors and shell scripts + env: + GOTOOLCHAIN: go1.26.3 run: | go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.12 python3 scripts/ci.py --self-test @@ -117,7 +119,7 @@ jobs: python3 scripts/checksum.py --self-test python3 scripts/ios_device.py --self-test python3 scripts/verify-node.py --binary /usr/bin/true --artifacts .artifacts/native-clipboard-selfcheck --native-clipboard-selfcheck - python3 -c 'import ast, pathlib; [ast.parse(pathlib.Path(path).read_text(), filename=path) for path in ("scripts/ci.py", "scripts/verify-node.py", "scripts/check-delivery.py", "scripts/install-delivery-toolchain.py", "scripts/run-delivery-live.py", "scripts/release.py", "scripts/package-node-windows.py", "scripts/verify-windows-imports.py", "scripts/check-release-workflow.py", "scripts/daily-nightly.py")]' + python3 -c 'import ast, pathlib; [ast.parse(pathlib.Path(path).read_text(), filename=path) for path in ("scripts/ci.py", "scripts/verify-node.py", "scripts/check-delivery.py", "scripts/install-delivery-toolchain.py", "scripts/run-delivery-live.py", "scripts/release.py", "scripts/package-node-windows.py", "scripts/verify-windows-imports.py", "scripts/check-release-workflow.py", "scripts/daily-nightly.py", "scripts/nightly_source.py")]' bash -n install.sh bash -n scripts/package-node.sh scripts/install-node-service.sh scripts/build-web-crypto.sh scripts/build-web.sh scripts/import-mac-identity.sh shellcheck install.sh scripts/package-node.sh scripts/install-node-service.sh scripts/build-web-crypto.sh scripts/build-web.sh scripts/import-mac-identity.sh @@ -918,9 +920,12 @@ jobs: needs: [plan, package] if: ${{ (needs.plan.outputs.npm == 'true') && !(github.event_name == 'workflow_dispatch' && (inputs.desktop_only || inputs.windows_only || inputs.mobile_apps != 'none')) }} runs-on: ubuntu-24.04 + outputs: + baseline_artifact: ${{ steps.baseline.outputs.artifact-id }} steps: - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 with: + fetch-depth: 0 persist-credentials: false - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 with: @@ -940,6 +945,21 @@ jobs: path: .artifacts/npm/leaperone-vibeface-*.tgz* if-no-files-found: error retention-days: 14 + - name: Pin one complete public npm baseline for every service runner + env: + GH_TOKEN: ${{ github.token }} + VIBEFACE_RELEASE_SCRATCH: ${{ github.workspace }}/.artifacts/npm-baseline-source + run: python3 scripts/verify-node.py --npm-baseline-pin --artifacts .artifacts/npm-service-baseline + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + id: baseline + with: + name: npm-service-baseline-${{ github.sha }} + path: | + .artifacts/npm-service-baseline/baseline.tgz + .artifacts/npm-service-baseline/pin.json + .artifacts/npm-service-baseline/release.json + if-no-files-found: error + retention-days: 14 npm-smoke: needs: [plan, npm-package] @@ -965,6 +985,12 @@ jobs: with: name: npm-package-${{ github.sha }} path: .artifacts/npm + - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 + if: matrix.target != 'x86_64-pc-windows-msvc' + with: + artifact-ids: ${{ needs.npm-package.outputs.baseline_artifact }} + path: .artifacts/npm-service-baseline + merge-multiple: true - name: Install npm artifact without lifecycle scripts id: install env: @@ -994,6 +1020,30 @@ jobs: run: | prefix="$PWD/.artifacts/npm-installed" python3 scripts/verify-node.py --binary "$prefix/bin/vibeface" --core-binary "$VENDOR/vibeface-core" --crypto-module "$VENDOR/web/vibeface_secure.js" --artifacts .artifacts/ci-npm --web-only + - name: Verify native npm service and package replacement + if: matrix.target != 'x86_64-pc-windows-msvc' + env: + VENDOR: ${{ steps.install.outputs.vendor }} + PACKAGE_VERSION: ${{ needs.plan.outputs.version }} + NPM_BASELINE_ARTIFACT_ID: ${{ needs.npm-package.outputs.baseline_artifact }} + run: | + [[ "$GITHUB_ACTIONS" == true && "$RUNNER_ENVIRONMENT" == github-hosted ]] + if [[ "$RUNNER_OS" == Linux ]]; then + if compgen -G "$HOME/.vibeface*" >/dev/null; then exit 1; fi + [[ "${XDG_CONFIG_HOME:-$HOME/.config}" == "$HOME/.config" ]] + for directory in "$HOME/.config/systemd/user" /etc/systemd/user /usr/lib/systemd/user /usr/local/lib/systemd/user; do + if compgen -G "$directory/vibeface*" >/dev/null || compgen -G "$directory/*/vibeface*" >/dev/null; then exit 1; fi + done + service_uid="$(id -u)" + sudo loginctl enable-linger "$(id -un)" + sudo systemctl start "user@$service_uid.service" + export XDG_RUNTIME_DIR="/run/user/$service_uid" + export DBUS_SESSION_BUS_ADDRESS="unix:path=$XDG_RUNTIME_DIR/bus" + [[ -S "$XDG_RUNTIME_DIR/bus" ]] + /usr/bin/systemctl --user show-environment >/dev/null + fi + prefix="$PWD/.artifacts/npm-installed" + python3 scripts/verify-node.py --binary "$prefix/bin/vibeface" --core-binary "$VENDOR/vibeface-core" --crypto-module "$VENDOR/web/vibeface_secure.js" --artifacts .artifacts/ci-npm-service --npm-service --npm-package ".artifacts/npm/leaperone-vibeface-$PACKAGE_VERSION.tgz" --npm-baseline .artifacts/npm-service-baseline windows-native: needs: [plan, package] @@ -1186,22 +1236,16 @@ jobs: permissions: contents: write actions: write - pull-requests: read steps: - - name: Require the Nightly release token - env: - NIGHTLY_RELEASE_TOKEN: ${{ secrets.NIGHTLY_RELEASE_TOKEN }} - run: test -n "$NIGHTLY_RELEASE_TOKEN" || { echo '::error::NIGHTLY_RELEASE_TOKEN is missing'; exit 1; } - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 with: ref: ${{ github.sha }} fetch-depth: 0 persist-credentials: false - uses: dtolnay/rust-toolchain@b725eda525362e7cf34af9a7243cac1e3f01a7ec - - name: Reconcile four-hour version PR and immutable Nightly archives + - name: Reconcile tag-only Nightly sources and immutable archives env: GH_TOKEN: ${{ github.token }} - NIGHTLY_RELEASE_TOKEN: ${{ secrets.NIGHTLY_RELEASE_TOKEN }} run: | gh auth setup-git python3 scripts/daily-nightly.py --run-id "$GITHUB_RUN_ID" diff --git a/.github/workflows/deploy-services.yml b/.github/workflows/deploy-services.yml index da45128c9..5fb229bee 100644 --- a/.github/workflows/deploy-services.yml +++ b/.github/workflows/deploy-services.yml @@ -70,7 +70,9 @@ jobs: run: python3 -B scripts/deploy-run.py build --service "$SERVICE" --tag "$RELEASE_TAG" --output .artifacts/deploy - name: Rehearse the Core activation on this runner if: matrix.service == 'node' - run: sudo python3 -B scripts/deploy-run.py rehearse --output .artifacts/deploy + env: + GH_TOKEN: ${{ github.token }} + run: sudo --preserve-env=GH_TOKEN python3 -B scripts/deploy-run.py rehearse --output .artifacts/deploy - name: Activate the pinned container on the configured host if: inputs.apply env: diff --git a/.github/workflows/npm-publish.yml b/.github/workflows/npm-publish.yml index 78dfdf127..b637d3e21 100644 --- a/.github/workflows/npm-publish.yml +++ b/.github/workflows/npm-publish.yml @@ -66,7 +66,7 @@ jobs: NPM_ARTIFACT_ID: ${{ inputs.artifact_id }} run: | [[ "$(git rev-parse HEAD)" == "$GITHUB_SHA" ]] - git merge-base --is-ancestor "$GITHUB_SHA" origin/main + python3 -B scripts/release.py plan --app product --tag "$RELEASE_TAG" --require-main --publish node scripts/publish-npm.mjs artifact - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 with: diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 31f51904b..200ffd817 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -48,13 +48,17 @@ jobs: GH_TOKEN: ${{ github.token }} run: | python3 -B - <<'PY' - import json, os, subprocess + import json, os, subprocess, sys + sys.path.insert(0, 'scripts') + import release scope, tag = os.environ['RELEASE_APP'], os.environ['RELEASE_TAG'] publish = os.environ['PUBLISH'] == 'true' command = ['python3', '-B', 'scripts/release.py', 'plan', '--app', scope] if tag: command += ['--tag', tag] if publish: + if os.environ['GITHUB_REPOSITORY'] != release.GITHUB_REPO: + raise SystemExit('Publication requires the official release repository') if not tag or os.environ['GITHUB_REF'] != 'refs/tags/' + tag: raise SystemExit('Publishing requires the exact existing tag as the workflow ref') command += ['--require-main', '--publish'] @@ -66,15 +70,7 @@ jobs: if result['source_commit'] != os.environ['GITHUB_SHA']: raise SystemExit('Build source differs from the workflow commit') if publish: - runs = json.loads(subprocess.check_output([ - 'gh', 'run', 'list', '--repo', os.environ['GITHUB_REPOSITORY'], - '--workflow', 'ci.yml', '--commit', result['source_commit'], - '--json', 'event,headBranch,headSha,status,conclusion', - ], text=True)) - if not any(run['event'] == 'push' and run['headBranch'] == 'main' and - run['headSha'] == result['source_commit'] and run['status'] == 'completed' and - run['conclusion'] == 'success' for run in runs): - raise SystemExit('The exact source must have a successful main push CI before publication') + release.require_ci(result) with open(os.environ['GITHUB_OUTPUT'], 'a') as output: for key in ('app', 'version', 'tag', 'source_commit', 'channel'): value = result[key] diff --git a/.preflight.toml b/.preflight.toml index 93d7d8af2..5e2bbfeac 100644 --- a/.preflight.toml +++ b/.preflight.toml @@ -4,7 +4,7 @@ base_branch = "main" prefix = "" name = "CI policy and Python syntax" check = "python3 scripts/ci.py --self-test && python3 scripts/check-delivery.py --self-test && python3 scripts/install-delivery-toolchain.py --self-test && python3 scripts/run-delivery-live.py --self-test && python3 scripts/daily-nightly.py --self-test && python3 -B scripts/test_progressive_publication.py && actionlint" -build = "python3 -c 'import ast,pathlib; [ast.parse(pathlib.Path(p).read_text()) for p in (\"scripts/ci.py\", \"scripts/verify-node.py\", \"scripts/build-inputs.py\", \"scripts/check-delivery.py\", \"scripts/install-delivery-toolchain.py\", \"scripts/run-delivery-live.py\", \"scripts/release.py\", \"scripts/daily-nightly.py\")]'" +build = "python3 -c 'import ast,pathlib; [ast.parse(pathlib.Path(p).read_text()) for p in (\"scripts/ci.py\", \"scripts/verify-node.py\", \"scripts/build-inputs.py\", \"scripts/check-delivery.py\", \"scripts/install-delivery-toolchain.py\", \"scripts/run-delivery-live.py\", \"scripts/release.py\", \"scripts/daily-nightly.py\", \"scripts/nightly_source.py\")]'" [[domain_check]] name = "Rust workspace" @@ -48,7 +48,7 @@ command = "node --check scripts/crypto-client.mjs && node --check apps/cli/npm/b [[domain_check]] name = "Delivery scripts and workflow" -trigger_paths = ["install.sh", "scripts/*.sh", ".github/workflows/**"] +trigger_paths = ["install.sh", "scripts/*.sh", "scripts/release.py", "scripts/daily-nightly.py", "scripts/nightly_source.py", ".github/workflows/**"] command = "bash -n install.sh && bash -n scripts/package-node.sh scripts/install-node-service.sh scripts/build-web-crypto.sh scripts/build-web.sh && shellcheck install.sh scripts/package-node.sh scripts/install-node-service.sh scripts/build-web-crypto.sh scripts/build-web.sh && bash install.sh --self-test && python3 scripts/release.py self-test && bash -n scripts/sign-release.sh && actionlint" [[domain_check]] diff --git a/apps/cli/src/service.rs b/apps/cli/src/service.rs index 3e74031c2..af529dd10 100644 --- a/apps/cli/src/service.rs +++ b/apps/cli/src/service.rs @@ -444,25 +444,10 @@ mod tests { let binary = package.join(PROGRAMS[0]); let data = root.join("data with spaces"); std::fs::create_dir(&data).unwrap(); - let install = command(&root, &data, &binary, Action::Install).unwrap(); - let mut expected = expected_script_call(&package.join(SERVICE_SCRIPT)); - expected.extend([ - "install".into(), - "--data-dir".into(), - script_os_argument(&data).into_os_string(), - "--binary".into(), - script_os_argument(&binary).into_os_string(), - ]); - assert_eq!(whole_call(&install), expected); - assert!(command(&root, &data, &binary, Action::InstallUpdater).is_err()); - assert!(command(&root, &data, &binary, Action::UninstallUpdater).is_ok()); - assert!(command(&root.join("another-install"), &data, &binary, Action::Stop).is_err()); let running = update::lock_data_dir(&data).unwrap(); assert!(command(&root, &data, &binary, Action::Install).is_err()); assert!(command(&root, &data, &binary, Action::Stop).is_ok()); drop(running); - let pending = data.join("update-pending.json"); - std::fs::write(&pending, b"{}").unwrap(); // A child another test thread is spawning may still hold the released lock, as in // `Runtime::reopen`. let install = || { @@ -481,6 +466,21 @@ mod tests { } } }; + let installation = install().unwrap(); + let mut expected = expected_script_call(&package.join(SERVICE_SCRIPT)); + expected.extend([ + "install".into(), + "--data-dir".into(), + script_os_argument(&data).into_os_string(), + "--binary".into(), + script_os_argument(&binary).into_os_string(), + ]); + assert_eq!(whole_call(&installation), expected); + assert!(command(&root, &data, &binary, Action::InstallUpdater).is_err()); + assert!(command(&root, &data, &binary, Action::UninstallUpdater).is_ok()); + assert!(command(&root.join("another-install"), &data, &binary, Action::Stop).is_err()); + let pending = data.join("update-pending.json"); + std::fs::write(&pending, b"{}").unwrap(); let error = install().unwrap_err(); assert!(error.to_string().contains("recover the pending update")); assert!(command(&root, &data, &binary, Action::Status).is_ok()); diff --git a/apps/core/src/client.rs b/apps/core/src/client.rs index 24570768f..951f10cf0 100644 --- a/apps/core/src/client.rs +++ b/apps/core/src/client.rs @@ -37,6 +37,7 @@ use std::{ use tokio::{sync::mpsc, time::Instant}; #[cfg(test)] use tokio_tungstenite::tungstenite::Message as NativeMessage; +use vibeface_protocol::valid_business_id; use vibeface_secure::{ Channel, join::{InviteRecord, Ticket, invite_id, join_commit, short_code}, @@ -2189,15 +2190,7 @@ fn from_field(value: &Value, key: &str) -> Resul .with_context(|| format!("invalid {key}")) } fn validate_request(request: &Request) -> Result<()> { - ensure!( - !request.id.is_empty() - && request.id.len() <= 128 - && request - .id - .bytes() - .all(|b| b.is_ascii_alphanumeric() || b"-_.:".contains(&b)), - "invalid request ID" - ); + ensure!(valid_business_id(&request.id), "invalid request ID"); ensure!( !request.method.is_empty() && request.method.len() <= 64 && request.params.is_object(), "invalid client method or params" @@ -4814,15 +4807,22 @@ mod tests { #[test] fn channel_requests_require_bounded_ids_and_object_params() { let mut request = Request { - id: "valid-request".into(), + id: "valid:request-._1".into(), method: "info".into(), params: json!({}), }; assert!(validate_request(&request).is_ok()); + request.id = "a".repeat(128); + assert!(validate_request(&request).is_ok()); request.params = json!("untyped payload"); assert!(validate_request(&request).is_err()); request.params = json!({}); - request.id = "\n".into(); - assert!(validate_request(&request).is_err()); + for id in ["", "\n", "bad/id", "bad id", "é", &"a".repeat(129)] { + request.id = id.into(); + assert_eq!( + validate_request(&request).unwrap_err().to_string(), + "invalid request ID" + ); + } } } diff --git a/apps/core/src/manager.rs b/apps/core/src/manager.rs index 592c6c444..e21a272d4 100644 --- a/apps/core/src/manager.rs +++ b/apps/core/src/manager.rs @@ -11,6 +11,7 @@ use std::{ }; use tokio::sync::watch; pub use vibeface_bridge::agent::{AgentFailure as ManagerFailure, StartEvidence}; +pub(crate) use vibeface_bridge::agent::{redact_progress as redact, redact_text as redact_string}; use vibeface_bridge::{ agent::{self, AgentResult, CursorAgent, CursorTurn, CustomToolConfig}, codex::DirectSession, @@ -1218,31 +1219,6 @@ fn codex_metadata_text(value: &toml::Value) -> Result<&str> { ); Ok(text) } -pub(crate) fn redact_string(mut text: String, secrets: &[String]) -> String { - for secret in secrets.iter().filter(|value| !value.is_empty()) { - text = text.replace(secret, "[redacted]"); - } - text -} -pub(crate) fn redact(value: Value, secrets: &[String]) -> Value { - match value { - Value::String(text) => Value::String(redact_string(text, secrets)), - Value::Array(items) => Value::Array( - items - .into_iter() - .map(|item| redact(item, secrets)) - .collect(), - ), - Value::Object(items) => Value::Object( - items - .into_iter() - .map(|(key, value)| (key, redact(value, secrets))) - .collect(), - ), - value => value, - } -} - #[cfg(test)] mod tests { use super::*; @@ -2188,14 +2164,15 @@ mod tests { let config = ManagerConfig { tool_env: BTreeMap::from([("TOKEN".into(), "other-secret".into())]), tool_token: Some("turn-token-secret".into()), + provider_env: BTreeMap::from([("API_KEY".into(), "other-secret-provider".into())]), ..codex(".".into(), "app-server.sock".into()) }; assert_eq!( redact( - json!({"nested":["other-secret","turn-token-secret"]}), + json!({"other-secret":["other-secret","turn-token-secret",{"provider":"other-secret-provider"}],"number":1}), &secret_values(&config) ), - json!({"nested":["[redacted]","[redacted]"]}) + json!({"other-secret":["[redacted]","[redacted]",{"provider":"[redacted]"}],"number":1}) ); } } diff --git a/apps/core/src/runtime.rs b/apps/core/src/runtime.rs index aafffa0f2..2d324be24 100644 --- a/apps/core/src/runtime.rs +++ b/apps/core/src/runtime.rs @@ -18,6 +18,7 @@ use std::{ time::{Duration, Instant, SystemTime, UNIX_EPOCH}, }; use tokio::sync::{Mutex, broadcast, mpsc}; +use vibeface_protocol::valid_business_id; pub(crate) mod agent_tools; mod bots; @@ -673,12 +674,7 @@ fn signed(v: &Value, key: &str) -> Result { } fn id(v: &Value, key: &str) -> Result { let s = field(v, key)?; - ensure!( - s.len() <= 128 - && s.chars() - .all(|c| c.is_ascii_alphanumeric() || "-_.:".contains(c)), - "invalid {key}" - ); + ensure!(valid_business_id(s), "invalid {key}"); Ok(s.into()) } impl Runtime { diff --git a/apps/core/src/store/bots.rs b/apps/core/src/store/bots.rs index 9def11b30..01210abd0 100644 --- a/apps/core/src/store/bots.rs +++ b/apps/core/src/store/bots.rs @@ -5,6 +5,7 @@ use rusqlite::{Connection, OptionalExtension, TransactionBehavior, params}; use serde::Serialize; use std::collections::BTreeMap; use std::path::{Component, Path}; +use vibeface_protocol::valid_business_id; const PREVIEW_CHARS: usize = 120; @@ -26,14 +27,6 @@ pub struct SentBotMessage { pub created_at_ms: i64, } -fn valid_id(value: &str) -> bool { - !value.is_empty() - && value.len() <= 128 - && value - .chars() - .all(|character| character.is_ascii_alphanumeric() || "-_.:".contains(character)) -} - fn validate_definition( bot_id: &str, conversation_id: &str, @@ -41,8 +34,11 @@ fn validate_definition( instructions: &str, project: Option<&BotProject>, ) -> Result<()> { - ensure!(valid_id(bot_id), "invalid bot_id"); - ensure!(valid_id(conversation_id), "invalid conversation_id"); + ensure!(valid_business_id(bot_id), "invalid bot_id"); + ensure!( + valid_business_id(conversation_id), + "invalid conversation_id" + ); validate_bot_name(name)?; validate_instructions(instructions)?; ensure!( @@ -51,7 +47,7 @@ fn validate_definition( ); if let Some(project) = project { ensure!( - valid_id(&project.target_node), + valid_business_id(&project.target_node), "invalid project target_node" ); let cwd = Path::new(&project.cwd); diff --git a/crates/bridge/src/agent.rs b/crates/bridge/src/agent.rs index f290fd2f2..580f303ff 100644 --- a/crates/bridge/src/agent.rs +++ b/crates/bridge/src/agent.rs @@ -1544,14 +1544,38 @@ fn url_with_password(word: &str) -> bool { }) } -fn redact_text(mut text: String, secrets: &[&str]) -> String { - for secret in secrets.iter().copied().filter(|secret| !secret.is_empty()) { - text = text.replace(secret, "[redacted]"); +pub fn redact_text(text: String, secrets: &[impl AsRef]) -> String { + redact_prefix(&text, secrets, text.len(), 0).0 +} + +fn redact_prefix( + text: &str, + secrets: &[impl AsRef], + split: usize, + mut redacted_until: usize, +) -> (String, usize) { + let mut clean = String::with_capacity(split); + for (index, ch) in text[..split].char_indices() { + let end = secrets + .iter() + .map(AsRef::as_ref) + .filter(|secret| !secret.is_empty() && text[index..].starts_with(secret)) + .map(|secret| index + secret.len()) + .max() + .unwrap_or(index); + if end > index { + if index >= redacted_until { + clean.push_str("[redacted]"); + } + redacted_until = redacted_until.max(end); + } else if index >= redacted_until { + clean.push(ch); + } } - text + (clean, redacted_until.saturating_sub(split)) } -fn redact_progress(value: Value, secrets: &[&str]) -> Value { +pub fn redact_progress(value: Value, secrets: &[impl AsRef]) -> Value { match value { Value::String(text) => Value::String(redact_text(text, secrets)), Value::Array(items) => Value::Array( @@ -1570,10 +1594,16 @@ fn redact_progress(value: Value, secrets: &[&str]) -> Value { } } +#[derive(Default)] +struct PendingRedaction { + text: String, + redacted_until: usize, +} + fn redact_stream_progress( mut value: Value, secrets: &[&str], - pending: &mut BTreeMap, + pending: &mut BTreeMap, ) -> Value { if let Some(item) = value .get("item_id") @@ -1587,27 +1617,31 @@ fn redact_stream_progress( && let Some(text) = value.get("text").and_then(Value::as_str) { let buffered = pending.entry(item).or_default(); - buffered.push_str(text); - let Value::String(clean) = redact_progress(json!(buffered), secrets) else { - unreachable!() - }; + buffered.text.push_str(text); let max_prefix = secrets .iter() .map(|secret| secret.len().saturating_sub(1)) .max() .unwrap_or(0); - let split = clean + let split = buffered + .text .char_indices() .map(|(index, _)| index) - .filter(|index| clean.len() - index <= max_prefix) + .filter(|index| buffered.text.len() - index <= max_prefix) .find(|index| { secrets.iter().any(|secret| { - secret.len() > clean.len() - index && secret.starts_with(&clean[*index..]) + secret.len() > buffered.text.len() - index + && secret.starts_with(&buffered.text[*index..]) }) }) - .unwrap_or(clean.len()); - *buffered = clean[split..].to_owned(); - value["text"] = json!(&clean[..split]); + .unwrap_or(buffered.text.len()); + let (text, redacted_until) = + redact_prefix(&buffered.text, secrets, split, buffered.redacted_until); + buffered.text = buffered.text[split..].to_owned(); + buffered.redacted_until = redacted_until; + value = redact_progress(value, secrets); + value["text"] = json!(text); + return value; } } redact_progress(value, secrets) @@ -1631,7 +1665,130 @@ pub(crate) async fn bounded_line( #[cfg(test)] mod credential_tests { - use super::credential_value; + use super::{credential_value, redact_progress, redact_stream_progress}; + use serde_json::json; + use std::collections::BTreeMap; + + #[test] + fn redaction_preserves_keys_and_masks_overlapping_secrets() { + let secrets = [ + "synthetic-sensitive", + "synthetic-sensitive-token", + "token-suffix", + "", + "redacted", + ]; + assert_eq!( + redact_progress( + json!({ + "synthetic-sensitive-token": [ + "prefix synthetic-sensitive-token-suffix ✓", + {"nested": "synthetic-sensitive"}, + ], + "literal": true, + "number": 7, + "null": null, + }), + &secrets, + ), + json!({ + "synthetic-sensitive-token": [ + "prefix [redacted] ✓", + {"nested": "[redacted]"}, + ], + "literal": true, + "number": 7, + "null": null, + }), + ); + } + + #[test] + fn redaction_buffers_nested_and_overlapping_secrets_across_frames() { + let secrets = [ + "synthetic-sensitive", + "synthetic-sensitive-token", + "aba", + "秘密🙂", + ]; + for text in [ + "before synthetic-sensitive-token after", + "before synthetic-sensitive after", + "before ababa after", + "before 秘密🙂 after", + ] { + let mut pending = BTreeMap::new(); + let mut streamed = String::new(); + for ch in text.chars() { + let update = redact_stream_progress( + json!({"type": "text_delta", "item_id": "reply", "text": ch.to_string()}), + &secrets, + &mut pending, + ); + streamed.push_str(update["text"].as_str().unwrap()); + } + assert_eq!(streamed, "before [redacted] after"); + let snapshot = redact_stream_progress( + json!({"type": "text_snapshot", "item_id": "reply", "text": text}), + &secrets, + &mut pending, + ); + assert_eq!(snapshot["text"], "before [redacted] after"); + assert!(pending.is_empty()); + } + } + + #[test] + fn redaction_keeps_periodic_overlap_streams_bounded() { + let secrets = ["aba"]; + let mut pending = BTreeMap::new(); + let mut streamed = String::new(); + for ch in format!("{}a end", "ab".repeat(4096)).chars() { + let update = redact_stream_progress( + json!({"type": "text_delta", "item_id": "reply", "text": ch.to_string()}), + &secrets, + &mut pending, + ); + let buffered = pending["reply"].text.len(); + assert!(buffered <= 2, "pending grew to {buffered} bytes"); + streamed.push_str(update["text"].as_str().unwrap()); + } + assert_eq!(streamed, "[redacted] end"); + } + + #[test] + fn redaction_keeps_items_separate_and_does_not_reprocess_stream_markers() { + let secrets = ["aba", "秘密🙂", "redacted"]; + let mut pending = BTreeMap::new(); + let mut streamed = BTreeMap::<&str, String>::new(); + for (item, chunk) in [ + ("left", "ab"), + ("right", "秘"), + ("left", "aba"), + ("right", "密🙂 "), + ("left", " "), + ] { + let update = redact_stream_progress( + json!({"type": "text_delta", "item_id": item, "text": chunk}), + &secrets, + &mut pending, + ); + streamed + .entry(item) + .or_default() + .push_str(update["text"].as_str().unwrap()); + } + assert_eq!(streamed["left"], "[redacted] "); + assert_eq!(streamed["right"], "[redacted] "); + let snapshot = redact_stream_progress( + json!({"type": "text_snapshot", "item_id": "left", "text": "ababa"}), + &secrets, + &mut pending, + ); + assert_eq!(snapshot["text"], "[redacted]"); + assert!(!pending.contains_key("left")); + assert!(pending.contains_key("right")); + } #[test] fn credential_values_are_selected_by_name_or_shape_only() { diff --git a/crates/bridge/tests/resident_cursor_bridge.rs b/crates/bridge/tests/resident_cursor_bridge.rs index 29d213ef4..9abaa2861 100644 --- a/crates/bridge/tests/resident_cursor_bridge.rs +++ b/crates/bridge/tests/resident_cursor_bridge.rs @@ -373,6 +373,8 @@ async fn cursor_stream_and_journal_redact_fragmented_credentials() -> Result<()> let mut turn = turn(&root, "redaction"); turn.env .insert("FIXTURE_SECRET".into(), "synthetic-sensitive-token".into()); + turn.env + .insert("FIXTURE_SHORT_SECRET".into(), "synthetic-sensitive".into()); turn.env.insert("FAKE_ACP_CHUNK_SIZE".into(), "1".into()); let updates = Arc::new(Mutex::new(Vec::new())); let seen = updates.clone(); @@ -397,6 +399,7 @@ async fn cursor_stream_and_journal_redact_fragmented_credentials() -> Result<()> .filter_map(|r| r["text"].as_str()) .collect::(); assert!(!joined.contains("synthetic-sensitive-token")); + assert!(!joined.contains("-token")); { let updates = updates.lock().unwrap(); let snapshot = updates.last().context("final reply snapshot")?; diff --git a/crates/protocol/src/lib.rs b/crates/protocol/src/lib.rs index 83c75a44b..0f49e79b6 100644 --- a/crates/protocol/src/lib.rs +++ b/crates/protocol/src/lib.rs @@ -8,6 +8,14 @@ pub enum RpcId { String(String), } +pub fn valid_business_id(id: &str) -> bool { + !id.is_empty() + && id.len() <= 128 + && id + .bytes() + .all(|c| c.is_ascii_alphanumeric() || b"-_.:".contains(&c)) +} + pub fn validate_id(id: &str) -> Result<(), String> { if id.is_empty() || id.len() > 128 @@ -34,6 +42,17 @@ mod tests { use super::*; use serde_json::json; + #[test] + fn business_ids_allow_colons_without_widening_protocol_ids() { + for id in ["Az09-_.:", ".", "..", ":", &"a".repeat(128)] { + assert!(valid_business_id(id), "{id:?}"); + } + for id in ["", "bad/id", "bad id", "bad\nid", "é", &"a".repeat(129)] { + assert!(!valid_business_id(id), "{id:?}"); + } + assert!(validate_id("scope:item").is_err()); + } + #[test] fn ids_reject_path_components_and_rpc_ids_preserve_their_wire_type() { assert!(validate_id("thread-1.turn_2").is_ok()); diff --git a/deploy/README.md b/deploy/README.md index 73af4627c..0022cefdf 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -103,7 +103,7 @@ Core 数据目录必须已经初始化,权限为 0700,UID/GID 与配置一 激活日志和 Compose 文件留在私有 `deploy_dir`。旧 Nginx 文件和离线备份只供激活中回滚;确认新容器后删除,同时删除其他旧 release 目录和上一版镜像,主机只保留当前版本。需要留档时在激活前把数据拷到本地。镜像健康需同时满足容器 running/healthy 与 Core inspect 的 running、版本和 device ID。失败时先停新容器,再恢复旧容器,并恢复 Nginx;全新安装没有旧服务,只停新容器;不会自动覆盖现有数据。中断后的非终态日志会先回滚并退出,检查旧服务后可重跑原命令。新容器无法确认停止时拒绝启动旧进程。确认后的同一请求会再次核对实际容器健康。 -`deploy-services` 部署 node 时,无论是否 `apply`,先在 runner 上按上文初始化一个 fixture 数据目录,用只改了 `public-bootstrap.json` 的同一镜像依次演练全新安装、交接和重复激活;演练失败则不触碰主机。 +`deploy-services` 部署 node 时,无论是否 `apply`,先选择候选发布时间之前最近的完整产品 Release,复用制品与摘要校验后构建旧 Core 镜像。旧 Core 初始化并启动 fixture,再在它运行时原子写入候选 Core 生成的完整配置,随后激活候选镜像并重复激活,核对原身份、bootstrap 文件、epoch 和数据库中的初始化记录保留。两个镜像只替换 fixture 的 `public-bootstrap.json`;数据格式和配置均按各自发布版实际生成,不做兼容迁移。fixture 只用回环地址、独立 mesh 和生成的数据。缺少历史完整产品 Release 时输出 `first_release`,验证首次安装和重复激活;其他查询、校验或升级错误一律停止。演练失败则不触碰主机。 本地验证入口均不访问生产服务。 @@ -113,4 +113,4 @@ python3 deploy/static-config.py --self-test python3 scripts/deploy-activate.py --self-test ``` -镜像跨 Docker 存储后端传输时,接收端的不可变 ID 可能与构建端不同。导入流程应先校验传输归档的 SHA-256,再使用 Docker 返回的实际导入 ID 执行激活检查。 +导入流程先校验传输归档的 SHA-256,再核对 Docker 返回的唯一不可变 ID 与调用方指定的 `--image` 严格一致。ID 不一致时在停止服务或修改部署数据前拒绝激活,不改用导入结果。 diff --git a/install.sh b/install.sh index ff15ac650..e117a3f7b 100755 --- a/install.sh +++ b/install.sh @@ -355,6 +355,138 @@ install_release() ( case ":$PATH:" in *":$bin_dir:"*) ;; *) printf 'Add your bin directory to PATH, or use the absolute CLI path above:\n export PATH=%q:"$PATH"\n' "$bin_dir" ;; esac ) +# The sourced production action reads fixture variables and calls the manager stubs. +# shellcheck disable=SC2034,SC2317,SC2329 +service_self_test() ( + local service_script=$1 directory=$2 service data_dir domain=gui/fixture label=com.leaperone.vibeface.fixture action + mkdir -p "$directory/data" + service="$directory/service.plist" + data_dir="$directory/data" + printf 'identity and signed history fixture\n' > "$data_dir/history" + printf 'attachment\000\377\n' > "$data_dir/attachment" + cp "$data_dir/history" "$directory/history.expected" + cp "$data_dir/attachment" "$directory/attachment.expected" + sed -n '/^launchd_node_action() {$/,/^}$/p' "$service_script" > "$directory/action.sh" + [[ -s "$directory/action.sh" ]] || fail 'self-test: missing launchd service action' + # shellcheck source=/dev/null + source "$directory/action.sh" + + model_count() { awk -v command="$1" '$1 == command {n++} END {print n+0}' "$directory/calls"; } + reset_manager() { + : > "$service" + : > "$directory/calls" + rm -f "$directory/cleaned" "$directory/stuck" "$directory/query-error" "$directory/bootout-error" "$directory/bootstrap-error" "$directory/query-overrun" + printf '%s\n' "$1" > "$directory/state" + printf '101\n' > "$directory/next-pid" + printf '2\n' > "$directory/ticks" + if [[ $1 == running ]]; then + printf '100\n' > "$directory/pid" + printf '1\n' > "$directory/children" + else + printf '0\n' > "$directory/pid" + printf '0\n' > "$directory/children" + fi + } + launchctl() { + printf '%s\n' "$*" >> "$directory/calls" + case "$1" in + print) + [[ $# == 2 && $2 == "$domain/$label" ]] || return 97 + if (( $(model_count print) > 350 )); then touch "$directory/query-overrun"; return 98; fi + if [[ $(cat "$directory/state") == absent ]]; then + printf 'Could not find service fixture.\n' >&2 + return 113 + fi + if [[ $(cat "$directory/state") == removing && -f "$directory/query-error" ]]; then + printf 'Operation not permitted.\n' >&2 + return 1 + fi + ;; + bootout) + [[ $# == 2 && $2 == "$domain/$label" ]] || return 97 + [[ ! -f "$directory/bootout-error" ]] || return 1 + printf 'removing\n' > "$directory/state" + ;; + bootstrap) + [[ $# == 3 && $2 == "$domain" && $3 == "$service" ]] || return 97 + [[ ! -f "$directory/bootstrap-error" ]] || return 1 + [[ $(cat "$directory/state") == absent && $(cat "$directory/children") == 0 ]] || return 97 + cp "$directory/next-pid" "$directory/pid" + printf 'running\n' > "$directory/state" + printf '1\n' > "$directory/children" + ;; + kickstart) + [[ $# == 2 && $2 == "$domain/$label" ]] || return 97 + [[ $(cat "$directory/state") != absent ]] || return 1 + ;; + *) return 97 ;; + esac + } + sleep() { + [[ $# == 1 && $1 == 0.1 ]] || return 97 + [[ ! -f "$directory/stuck" ]] || return 0 + local ticks + ticks=$(cat "$directory/ticks") + ticks=$((ticks - 1)) + printf '%s\n' "$ticks" > "$directory/ticks" + if ((ticks == 0)); then + printf 'absent\n' > "$directory/state" + printf '0\n' > "$directory/pid" + printf '0\n' > "$directory/children" + touch "$directory/cleaned" + printf 'children-stopped\n' >> "$directory/calls" + fi + } + + reset_manager running + action=restart + launchd_node_action + [[ $(cat "$directory/state") == running && $(cat "$directory/pid") == 101 && -f "$directory/cleaned" ]] || fail 'self-test: restart did not replace the stopped process' + [[ $(model_count bootout) == 1 && $(model_count bootstrap) == 1 && $(model_count kickstart) == 1 ]] || fail 'self-test: restart manager sequence' + awk '/^children-stopped$/ {cleaned=1} /^bootstrap / && !cleaned {bad=1} END {exit bad}' "$directory/calls" || fail 'self-test: restarted before child cleanup' + reset_manager idle + launchd_node_action + [[ $(cat "$directory/state") == running && $(cat "$directory/pid") == 101 ]] || fail 'self-test: restart of an idle registered service' + reset_manager absent + launchd_node_action + [[ $(cat "$directory/state") == running && $(model_count bootout) == 0 ]] || fail 'self-test: restart of an unloaded service' + reset_manager running + action=start + launchd_node_action + [[ $(cat "$directory/pid") == 100 && $(model_count bootout) == 0 && $(model_count bootstrap) == 0 ]] || fail 'self-test: start disturbed a running service' + reset_manager absent + launchd_node_action + [[ $(cat "$directory/state") == running && $(model_count bootstrap) == 1 ]] || fail 'self-test: start did not bootstrap an unloaded service' + reset_manager running + action=stop + launchd_node_action + [[ $(cat "$directory/state") == removing && $(model_count bootstrap) == 0 && $(model_count kickstart) == 0 ]] || fail 'self-test: stop restarted the service' + reset_manager running + action=restart + touch "$directory/stuck" + if launchd_node_action > "$directory/output" 2>&1; then fail 'self-test: restart ignored incomplete removal'; fi + [[ $(model_count bootstrap) == 0 && $(model_count kickstart) == 0 && ! -f "$directory/query-overrun" ]] || fail 'self-test: restart did not bound removal observation' + reset_manager running + touch "$directory/query-error" + if launchd_node_action > "$directory/output" 2>&1; then fail 'self-test: query failure counted as removal'; fi + [[ $(model_count bootstrap) == 0 && $(model_count kickstart) == 0 ]] || fail 'self-test: query failure restarted the service' + reset_manager running + touch "$directory/bootout-error" + if launchd_node_action > "$directory/output" 2>&1; then fail 'self-test: bootout failure ignored'; fi + [[ $(model_count bootstrap) == 0 && $(model_count kickstart) == 0 ]] || fail 'self-test: failed stop restarted the service' + reset_manager absent + touch "$directory/bootstrap-error" + if launchd_node_action > "$directory/output" 2>&1; then fail 'self-test: bootstrap failure ignored'; fi + [[ $(model_count kickstart) == 0 ]] || fail 'self-test: kickstart followed a failed bootstrap' + reset_manager running + rm "$service" + if launchd_node_action > "$directory/output" 2>&1; then fail 'self-test: missing service was accepted'; fi + [[ ! -s "$directory/calls" ]] || fail 'self-test: missing service reached the manager' + cmp "$data_dir/history" "$directory/history.expected" || fail 'self-test: service action changed history' + cmp "$data_dir/attachment" "$directory/attachment.expected" || fail 'self-test: service action changed attachment bytes' + printf 'User service self-test passed: asynchronous restart, cleanup ordering, stop/start, failure protection and preserved data.\n' +) + self_test() ( local test_dir script fixture version target package case_dir script=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}") @@ -681,6 +813,7 @@ JSON mkdir "$case_dir" expect_failure nightly 'no published Nightly release' reset_progressive_markers + service_self_test "$(dirname "$script")/scripts/install-node-service.sh" "$test_dir/service" printf 'Public installer self-test passed: channels, numeric sorting, layout, SHA256, manifest, links, paths, progressive completion, and existing-install protection.\n' ) diff --git a/packages/client/src/identity/context.tsx b/packages/client/src/identity/context.tsx index d085b1dc5..9765430f4 100644 --- a/packages/client/src/identity/context.tsx +++ b/packages/client/src/identity/context.tsx @@ -19,6 +19,7 @@ import { candidates, defaultGateway, gatewaysOf, markReached, mergeEntrances, op import type { Entrance, Gateway, OpenEntrance } from '../lib/entrances.ts' import type { KeyValue, LanSighting, LocalLaunch, Platform } from '../lib/platform.ts' import { connectLogSink, reportLog } from '../lib/web-log.ts' +import { storageOf } from '../workspace/storage.ts' type Phase = 'initializing' | 'blocked' | 'key_lost' | 'disconnected' | 'connecting' | 'reconnecting' | 'unpaired' | 'removed' | 'restarting' | 'erasing' | 'erase_failed' | 'ready' type Review = { kind: 'bootstrap'; networkId: string } | { kind: 'recovery'; networkId: string; members: Member[]; newManager: Member; requiredNode: string } @@ -138,8 +139,9 @@ const hex = (bytes: Uint8Array) => [...bytes].map(byte => byte.toString(16).padS const randomHex = () => hex(crypto.getRandomValues(new Uint8Array(32))) /** Saved connections, pins and pending joins in this device's storage. */ function deviceStore(storage: KeyValue, self: string) { - function load(key: string): unknown { try { const value = storage.getItem(`vibeface.${key}`); return value ? JSON.parse(value) : null } catch { return null } } - function store(key: string, value: unknown) { try { storage.setItem(`vibeface.${key}`, JSON.stringify(value)) } catch { throw new Error(`${self}无法保存设备记录。允许保存本地数据后重试。`) } } + const { readStorage, writeStorage } = storageOf(storage) + const load = (key: string) => readStorage(key, null) + function store(key: string, value: unknown) { try { writeStorage(key, value, true) } catch { throw new Error(`${self}无法保存设备记录。允许保存本地数据后重试。`) } } function savedEntries(): ConnectionEntry[] { const value = load('connections') if (!Array.isArray(value)) return [] diff --git a/packages/client/src/lib/experiments.test.ts b/packages/client/src/lib/experiments.test.ts index 22ee30157..17ca13794 100644 --- a/packages/client/src/lib/experiments.test.ts +++ b/packages/client/src/lib/experiments.test.ts @@ -2,6 +2,7 @@ import assert from 'node:assert/strict' import test from 'node:test' import { experimentsOn, setExperiments } from './experiments.ts' import type { KeyValue } from './platform.ts' +import { storageOf } from '../workspace/storage.ts' const memory = (): KeyValue => { const values = new Map(); return { getItem: (key) => values.get(key) ?? null, setItem: (key, value) => { values.set(key, value) } } } @@ -14,3 +15,26 @@ test('实验功能 is off until turned on, and stays as set across restarts', () assert.equal(experimentsOn(storage), false) assert.equal(experimentsOn({ getItem: () => { throw new Error('blocked') }, setItem() {} }), false, 'unreadable storage keeps it off') }) + +test('shared JSON storage preserves device keys and optional read fallbacks', () => { + const storage = memory() + const { readStorage, writeStorage } = storageOf(storage) + writeStorage('connections', [{ device_id: 'node' }], true) + assert.equal(storage.getItem('vibeface.connections'), '[{"device_id":"node"}]') + assert.deepEqual(readStorage('connections', null), [{ device_id: 'node' }]) + storage.setItem('vibeface.broken', '{') + assert.equal(readStorage('broken', null), null) + assert.throws(() => readStorage('broken', null, true), { message: '读不了本地需要确认的记录,已暂停发送。先别清除这台设备上的应用数据。' }) + assert.equal(readStorage('missing', null, true), null) + for (const value of [false, 0, '']) { + writeStorage('value', value, true) + assert.equal(readStorage('value', null), value) + } +}) + +test('required JSON reads and writes stop when local storage is unavailable', () => { + const { readStorage, writeStorage } = storageOf({ getItem() { throw new Error('blocked') }, setItem() { throw new Error('blocked') } }) + assert.equal(readStorage('pending-join', null), null) + assert.throws(() => readStorage('sequence', 0, true), { message: '读不了本地需要确认的记录,已暂停发送。先别清除这台设备上的应用数据。' }) + assert.throws(() => writeStorage('sequence', 1, true), { message: '保存不了这条记录。允许保存本地数据后重试。' }) +}) diff --git a/packages/client/src/lib/node.ts b/packages/client/src/lib/node.ts index ce6bca416..2cc45eb01 100644 --- a/packages/client/src/lib/node.ts +++ b/packages/client/src/lib/node.ts @@ -110,15 +110,17 @@ export function record(value: unknown, label = '设备响应'): Record { return value !== null && typeof value === 'object' && !Array.isArray(value) } +export function isRecord(value: unknown): value is Record { return value !== null && typeof value === 'object' && !Array.isArray(value) } +export function isText(value: unknown): value is string { return typeof value === 'string' } +export function isSafeInteger(value: unknown): value is number { return typeof value === 'number' && Number.isSafeInteger(value) } export function text(value: unknown, label = '文本'): string { - if (typeof value !== 'string' || !value) throw new Error(`${label}缺失或格式无效。`) + if (!isText(value) || !value) throw new Error(`${label}缺失或格式无效。`) return value } -export function boolean(value: unknown): boolean { if (typeof value !== 'boolean') throw new Error('设备返回的状态格式无效。'); return value } -function integer(value: unknown): number { if (typeof value !== 'number' || !Number.isSafeInteger(value) || value < 1) throw new Error('设备返回的编号或版本无效。'); return value } -function strings(value: unknown): string[] { if (!Array.isArray(value)) throw new Error('设备返回的列表格式无效。'); return value.map(item => text(item)) } -export function array(value: unknown): unknown[] { if (!Array.isArray(value)) throw new Error('设备返回的列表格式无效。'); return value } +export function boolean(value: unknown, error = '设备返回的状态格式无效。'): boolean { if (typeof value !== 'boolean') throw new Error(error); return value } +function integer(value: unknown): number { if (!isSafeInteger(value) || value < 1) throw new Error('设备返回的编号或版本无效。'); return value } +function strings(value: unknown): string[] { return array(value).map(item => text(item)) } +export function array(value: unknown, error = '设备返回的列表格式无效。'): unknown[] { if (!Array.isArray(value)) throw new Error(error); return value } /** An entry as Nodes send it and this device stores it: `{address, device_id}`, plus `local: true` for a local page, or `{device_id, addrs}` on the LAN. */ export function parseEntry(value: unknown): ConnectionEntry { const v = record(value, '访问地址') diff --git a/packages/client/src/lib/progress.test.ts b/packages/client/src/lib/progress.test.ts index 75ba88681..7e545be58 100644 --- a/packages/client/src/lib/progress.test.ts +++ b/packages/client/src/lib/progress.test.ts @@ -15,6 +15,11 @@ test('resync is a snapshot request and does not alter live progress', () => { test('only the normalized progress contract crosses the changed boundary', () => { const event = manager(1, { type: 'text_snapshot', text: '', item_id: 'reply-a' }) assert.equal(event.type, 'manager_progress') + for (const sequence of [0, -1, 1.5, Number.MAX_SAFE_INTEGER + 1]) { + assert.deepEqual(manager(sequence, { type: 'text_snapshot', text: '' }), { type: 'invalidate' }) + } + assert.deepEqual(manager(1, { type: 'text_snapshot', text: '' }, ''), { type: 'invalidate' }) + assert.deepEqual(manager(1, { type: 'status', text: '' }), { type: 'invalidate' }) for (const value of [ null, [], { type: 'unknown' }, { type: 'manager_progress', conversation_id: 'chat-a', trigger: 'turn-a', progress: { provider: 'arbitrary-json' } }, diff --git a/packages/client/src/lib/progress.ts b/packages/client/src/lib/progress.ts index 8d0bd3cd0..66806c3b0 100644 --- a/packages/client/src/lib/progress.ts +++ b/packages/client/src/lib/progress.ts @@ -1,3 +1,5 @@ +import { isRecord, isSafeInteger, isText } from './node.ts' + type ProgressUpdate = | { type: 'text_snapshot' | 'text_delta'; item_id?: string; text: string } | { type: 'status' | 'tool'; item_id?: string; text?: string; status?: string; name?: string } @@ -23,11 +25,8 @@ export type LiveProgress = LiveProgressFields & ( | { kind: 'native'; work_id: string; target_node: string } ) -function isRecord(value: unknown): value is Record { - return value !== null && typeof value === 'object' && !Array.isArray(value) -} function requiredText(value: unknown): string { - if (typeof value !== 'string' || !value) throw new Error('无效的实时进度编号。') + if (!isText(value) || !value) throw new Error('无效的实时进度编号。') return value } function optionalText(value: unknown): string | undefined { @@ -37,7 +36,7 @@ function parseProgress(value: unknown): ProgressUpdate { if (!isRecord(value)) throw new Error('无效的实时进度。') const item_id = optionalText(value.item_id) if (value.type === 'text_snapshot' || value.type === 'text_delta') { - if (typeof value.text !== 'string') throw new Error('无效的实时文本。') + if (!isText(value.text)) throw new Error('无效的实时文本。') return { type: value.type, item_id, text: value.text } } if (value.type === 'status' || value.type === 'tool') { @@ -57,7 +56,7 @@ export function parseChanged(value: unknown): ChangedEvent { if (value.type !== 'manager_progress' && value.type !== 'native_progress') return { type: 'invalidate' } const conversation_id = requiredText(value.conversation_id), progress = parseProgress(value.progress) const sequence = value.sequence - if (typeof sequence !== 'number' || !Number.isSafeInteger(sequence) || sequence < 1) return { type: 'invalidate' } + if (!isSafeInteger(sequence) || sequence < 1) return { type: 'invalidate' } if (value.type === 'manager_progress') return { type: value.type, conversation_id, progress, sequence, trigger: requiredText(value.trigger), source_node: requiredText(value.source_node), invocation_id: optionalText(value.invocation_id) } return { type: value.type, conversation_id, progress, sequence, work_id: requiredText(value.work_id), target_node: requiredText(value.target_node) } } catch { return { type: 'invalidate' } } diff --git a/packages/client/src/workspace/context.tsx b/packages/client/src/workspace/context.tsx index 05ac29de6..97342fadd 100644 --- a/packages/client/src/workspace/context.tsx +++ b/packages/client/src/workspace/context.tsx @@ -29,7 +29,7 @@ import type { PendingMessage, WorkspaceView, BotRecord, CreateBotRequest, PendingBot, BotProject, PendingCheck, WriteState, WriteStatus, OpenWork, ScheduleAction, WorkspaceOverview, InputControlAction, InputOptions, InputReceipt, InputTarget, WorkDetailWork, WorkspaceCommand, WorkspaceCommandResult, WorkspaceCommandNotSaved, WorkspaceDomainState, WorkspaceFilter, } from "./types.ts" -import { isMainNotification, isSessionNotification } from "./types.ts" +import { isMainNotification, isReplyEnded, isSessionNotification, isTerminalJobState } from "./types.ts" const writeNotices: Record, string> = { aligning: "正在和你的其他设备同步,稍后就能发送。", @@ -268,7 +268,7 @@ function useWorkspaceState(platform: Platform, connection: ReadyConnection, open await outboxChanged() } // Streamed text of a reply that ended without a final message is kept locally and shown as interrupted. - const ended = (job: { state: string; reply_message_id?: string } | undefined) => job && ["completed", "failed", "cancelled", "interrupted", "rejected"].includes(job.state) && !(job.reply_message_id && next.messages.some((message) => message.message_id === job.reply_message_id)) + const ended = (job: { state: string; reply_message_id?: string } | undefined) => job && isReplyEnded(job.state) && !(job.reply_message_id && next.messages.some((message) => message.message_id === job.reply_message_id)) const kept = readStorage(scoped("interrupted"), []).filter((item) => item.conversation_id !== id || next.scheduler?.jobs.some((job) => job.trigger === item.trigger && ended(job))) for (const progress of liveRef.current) { if (progress.kind !== "manager" || !progress.text || progress.conversation_id !== id || kept.some((item) => item.trigger === progress.trigger)) continue @@ -280,7 +280,7 @@ function useWorkspaceState(platform: Platform, connection: ReadyConnection, open setLiveProgress((previous) => previous.filter((progress) => { if (progress.kind === "native") return next.works.some((item) => item.target_node === progress.target_node && item.work_id === progress.work_id && item.state !== "finished") const job = next.scheduler?.jobs.find((item) => item.trigger === progress.trigger) - return Boolean(job && (!job.reply_message_id || !next.messages.some((message) => message.message_id === job.reply_message_id)) && !["completed", "failed", "cancelled", "interrupted", "rejected"].includes(job.state)) + return Boolean(job && (!job.reply_message_id || !next.messages.some((message) => message.message_id === job.reply_message_id)) && !isReplyEnded(job.state)) })) if (joined && current() && readConnection.current === connection && viewRef.current === "chat" && platform.foreground()) { const seen = notificationsRef.current.filter((item) => notificationForMain(item, id) && !isSessionNotification(item)) @@ -1020,7 +1020,7 @@ function useWorkspaceState(platform: Platform, connection: ReadyConnection, open } if (target.kind === 'channel_member') throw new Error('先选择频道中的具体任务,再请求停止。') const context = parseConversation(await connection.rpc('conversation', { conversation_id: id }), id) - const triggers = target.kind === 'routine' ? [target.trigger] : context.scheduler?.jobs.filter((job) => !['completed', 'failed', 'cancelled', 'interrupted', 'rejected', 'steered'].includes(job.state) && job.scope.kind === 'main').map((job) => job.trigger) ?? [] + const triggers = target.kind === 'routine' ? [target.trigger] : context.scheduler?.jobs.filter((job) => !isTerminalJobState(job.state) && job.scope.kind === 'main').map((job) => job.trigger) ?? [] if (!triggers.length) throw new Error('这个目标当前没有可停止的回复。') const receipts = await Promise.all(triggers.map((trigger) => connection.rpc('cancel_manager', { conversation_id: id, trigger }).then(record))) requireCurrent() diff --git a/packages/client/src/workspace/labels.ts b/packages/client/src/workspace/labels.ts index caa266c4e..db3421591 100644 --- a/packages/client/src/workspace/labels.ts +++ b/packages/client/src/workspace/labels.ts @@ -3,6 +3,7 @@ import type { LiveProgress } from "../lib/progress.ts" import { confirmReason } from "./model.ts" import { awaitsAnswer, workApproval } from "./questions.ts" import type { AgentLane, ContextProjection, ManagerJob, MemoryEntry, NodeInfo, OpenWorkCard, OverviewBot, OverviewCoverage, OverviewJob, OverviewTask, PendingCheck, Question, QueueHoldReason, ResolvedModel, SessionHandoff, WorkCard, WorkRecord, WorkspaceActivity, WorkspaceActivityAttention, WorkspaceActivityFacts, WorkspaceActivityGroup, WorkspaceActivityJob, WorkspaceActivityView, WorkspaceOverview } from "./types.ts" +import { isJobState, isTerminalJobState, jobStates, type JobState } from "./types.ts" export const contextStages = { compiled: { label: '已编译', detail: '内容已准备,尚未确认 AI 开始执行。', tone: 'info' }, @@ -266,12 +267,6 @@ const sameObservedJob = (left: WorkspaceActivityJob, right: WorkspaceActivityJob const leftSource = jobSource(left, localNode), rightSource = jobSource(right, localNode) return leftSource !== null && leftSource === rightSource } -const terminalJobStates = new Set(["completed", "failed", "cancelled", "interrupted", "rejected", "steered"]) -const activeJobStates = new Set(["claimed", "running", "steering", "cancelling", "publishing", "reconciling"]) -const jobStateRank: Record = { - queued: 10, pending_delivery: 10, claimed: 20, running: 30, steering: 35, cancelling: 40, publishing: 50, - completed: 60, interrupted: 60, cancelled: 60, steered: 60, rejected: 65, failed: 65, reconciling: 80, unknown: 100, unverified: 100, -} const jobFailure = (job: ManagerJob | OverviewJob) => "failure" in job ? job.failure : undefined function mergeJobObservations(entries: readonly WorkspaceActivityJob[]): WorkspaceActivityJob { @@ -279,19 +274,19 @@ function mergeJobObservations(entries: readonly WorkspaceActivityJob[]): Workspa if (!first) throw new Error("任务观察不能为空。") const jobs = entries.map((entry) => entry.job) const states = new Set(jobs.map((job) => job.state)) - const terminalStates = new Set(jobs.map((job) => job.state).filter((state) => terminalJobStates.has(state))) + const terminalStates = new Set(jobs.map((job) => job.state).filter(isTerminalJobState)) const outcomes = new Set(jobs.flatMap((job) => job.outcome ? [job.outcome] : [])) const runtimes = jobs.flatMap((job) => job.runtime ? [job.runtime] : []) const runtimeIds = new Set(runtimes.map((runtime) => JSON.stringify([runtime.delivery_id, runtime.generation, runtime.session_generation, runtime.provider, runtime.model, runtime.reasoning_effort]))) const ownerEpochs = new Set(jobs.flatMap((job) => "owner_epoch" in job && job.owner_epoch !== null ? [job.owner_epoch] : [])) - const hasUnknownState = jobs.some((job) => !(job.state in jobStateRank)) + const hasUnknownState = jobs.some((job) => !isJobState(job.state)) const pendingCheck = jobs.some((job) => job.pending_check === true) const queueStates = new Set(["queued", "pending_delivery"]) const recoveryStates = new Set(["claimed", "steering"]) const recoveryRollback = jobs.some((job) => queueStates.has(job.state)) && jobs.some((job) => recoveryStates.has(job.state)) const attemptConflict = runtimeIds.size > 1 || ownerEpochs.size > 1 - const uncertain = pendingCheck || hasUnknownState || states.has("unknown") || states.has("unverified") || terminalStates.size > 1 || outcomes.size > 1 || (terminalStates.size > 0 && jobs.some((job) => activeJobStates.has(job.state))) || recoveryRollback || attemptConflict || jobs.some((job) => job.state === "steered" && job.outcome === "unknown") - const state = uncertain ? "unknown" : [...states].sort((left, right) => jobStateRank[right]! - jobStateRank[left]!)[0] ?? "unknown" + const uncertain = pendingCheck || hasUnknownState || states.has("unknown") || states.has("unverified") || terminalStates.size > 1 || outcomes.size > 1 || (terminalStates.size > 0 && jobs.some((job) => isJobState(job.state) && jobStates[job.state].phase === 'active')) || recoveryRollback || attemptConflict || jobs.some((job) => job.state === "steered" && job.outcome === "unknown") + const state = uncertain ? "unknown" : [...states].filter(isJobState).sort((left, right) => jobStates[right].rank - jobStates[left].rank)[0] ?? "unknown" const primary = entries.find((entry) => entry.job.state === state) ?? entries[0] const failure = jobs.map(jobFailure).find(Boolean) const runtime = primary.job.runtime ?? runtimes[0] @@ -424,13 +419,13 @@ export function parseSummarySteps(summary: string): SummaryStep[] | null { return steps } -const schedulerStates: Record = { +const schedulerStates: Partial> = { queued: { label: "排队中", group: "open" }, claimed: { label: "已领取", group: "open" }, running: { label: "执行中", group: "open" }, steering: { label: "补充中", group: "open" }, cancelling: { label: "停止中", group: "open" }, publishing: { label: "回复同步中", group: "open" }, pending_delivery: { label: "等待设备接收", group: "open" }, reconciling: { label: "正在查询上次回复", group: "open" }, steered: { label: "补充已结算", group: "history" }, completed: { label: "已完成", group: "history" }, failed: { label: "失败", group: "history" }, cancelled: { label: "已取消", group: "history" }, interrupted: { label: "已中断", group: "history" }, } -export const schedulerStatus = (state: string): { label: string; group: "open" | "history" | "unknown" } => schedulerStates[state] ?? { label: "状态未知", group: "unknown" } +export const schedulerStatus = (state: string): { label: string; group: "open" | "history" | "unknown" } => (isJobState(state) ? schedulerStates[state] : undefined) ?? { label: "状态未知", group: "unknown" } /** Muting is independent of the device's reachability and never proves that a provider session is alive. */ export function externalSessionStatus(session: Pick, nodes: readonly NodeInfo[]): "revoked" | "unknown" | "offline" | "reachable" { diff --git a/packages/client/src/workspace/model.ts b/packages/client/src/workspace/model.ts index 87d23a24c..59d6af0c9 100644 --- a/packages/client/src/workspace/model.ts +++ b/packages/client/src/workspace/model.ts @@ -1,6 +1,6 @@ import type { Attachment, PendingMessage, TranscriptMessage } from "../chat/types.ts" import type { OutboxItem } from "../lib/outbox.ts" -import { parseProviderReadiness } from "../lib/node.ts" +import { array, boolean, isRecord, isSafeInteger, isText, parseProviderReadiness } from "../lib/node.ts" import { providerStages, type ProviderStage, type TurnRuntime } from "./types.ts" import type { ReplyStatus } from "../chat/types.ts" import type { @@ -25,25 +25,19 @@ export function inputText(value: string, label: string, maxBytes: number): strin } export function record(value: unknown): Record { - if (!value || typeof value !== "object" || Array.isArray(value)) throw new Error("设备返回的内容格式不对。") - return value as Record + if (!isRecord(value)) throw new Error("设备返回的内容格式不对。") + return value } function text(value: unknown): string { - if (typeof value !== "string") throw new Error("设备返回的内容缺少文字。") + if (!isText(value)) throw new Error("设备返回的内容缺少文字。") return value } function number(value: unknown): number { - if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 0) throw new Error("设备返回的数字不对。") - return value -} -function flag(value: unknown): boolean { - if (typeof value !== "boolean") throw new Error("设备返回的内容缺少开关值。") - return value -} -function list(value: unknown): unknown[] { - if (!Array.isArray(value)) throw new Error("设备返回的内容缺少列表。") + if (!isSafeInteger(value) || value < 0) throw new Error("设备返回的数字不对。") return value } +function flag(value: unknown): boolean { return boolean(value, "设备返回的内容缺少开关值。") } +function list(value: unknown): unknown[] { return array(value, "设备返回的内容缺少列表。") } function texts(value: unknown): string[] { return list(value).map(text) } function conversation(value: unknown): ConversationSummary { const data = record(value) diff --git a/packages/client/src/workspace/overview.test.ts b/packages/client/src/workspace/overview.test.ts index 4a4fafb02..5a0931859 100644 --- a/packages/client/src/workspace/overview.test.ts +++ b/packages/client/src/workspace/overview.test.ts @@ -1,5 +1,6 @@ import assert from "node:assert/strict" import type { SessionScope } from './types.ts' +import { isReplyEnded, isTerminalJobState } from './types.ts' import test from "node:test" import { openWorksFromOverview, overviewFromWorkspace, parseConversation, parseOpenWorks, parseWorkspaceOverview } from "./model.ts" import { cardState, overviewScope, taskGroup, taskLabel, workspaceActivities } from "./labels.ts" @@ -7,6 +8,19 @@ import type { BotRecord, NodeInfo, OpenWork, PendingCheck, Question } from "./ty const mainScope: SessionScope = { kind: 'main' } +test('steered inputs settle without ending their shared reply stream', () => { + for (const state of ['completed', 'failed', 'cancelled', 'interrupted', 'rejected']) { + assert.equal(isReplyEnded(state), true) + assert.equal(isTerminalJobState(state), true) + } + assert.equal(isReplyEnded('steered'), false) + assert.equal(isTerminalJobState('steered'), true) + for (const state of ['publishing', 'pending_delivery', 'future_state', 'toString']) { + assert.equal(isReplyEnded(state), false) + assert.equal(isTerminalJobState(state), false) + } +}) + const bot = (conversation_id: string, bot_id = `bot-${conversation_id}`, name = conversation_id): BotRecord => ({ bot_id, conversation_id, name, instructions: "", models: {}, skills: null, avatar: null, avatar_version: null, event_hash: "event", last_message: null, busy: null, diff --git a/packages/client/src/workspace/records.test.ts b/packages/client/src/workspace/records.test.ts index de70a2577..1e0148583 100644 --- a/packages/client/src/workspace/records.test.ts +++ b/packages/client/src/workspace/records.test.ts @@ -2,7 +2,8 @@ import assert from "node:assert/strict" import type { SessionScope } from './types.ts' import test from "node:test" import { parseChanged } from "../lib/progress.ts" -import { botReadiness, confirmReason, overviewFromWorkspace, parseConversation, parseNodes, parseOpenWorks, parseQuestions, parseRecords, parseScopeContext, parseWorkDetail, parseWorkspaceHistory, parseWorkspaceState, workChecks } from "./model.ts" +import { parsePrincipal, text } from "../lib/node.ts" +import { botReadiness, confirmReason, overviewFromWorkspace, parseConversation, parseNodes, parseNotifications, parseOpenWorks, parseQuestions, parseRecords, parseScopeContext, parseWorkDetail, parseWorkspaceHistory, parseWorkspaceState, workChecks } from "./model.ts" import { answerBody } from './questions.ts' import { mergeBots, mergeConversation, mergeNotifications, mergeOpenWorks, mergeQuestions, mergeWorkspaceState } from "./records.ts" import { taskAgent } from "./labels.ts" @@ -39,6 +40,19 @@ const records = (extra: Record = {}) => { return parseRecords(event.type === "records" ? event.value : null) } +test('response parsers retain their text, numeric and error boundaries', () => { + const principal = { network_id: 'a'.repeat(64), device_id: 'b'.repeat(64), source: 'membership', can_manage: false, epoch: 1 } + assert.equal(parsePrincipal(principal).epoch, 1) + assert.throws(() => parsePrincipal({ ...principal, epoch: 0 }), { message: '设备返回的编号或版本无效。' }) + assert.throws(() => text(''), { message: '文本缺失或格式无效。' }) + assert.equal(parseNotifications({ notifications: [{ notification_id: 'n', conversation_id: 'chat', kind: 'message', text: '' }] })[0].text, '') + const state = { channels: [], tasks: [], assignments: [], inputs: [], coverage: { limit: 0, truncated: false } } + assert.equal(parseWorkspaceState(state).coverage.limit, 0) + assert.throws(() => parseWorkspaceState({ ...state, channels: {} }), { message: '设备返回的内容缺少列表。' }) + assert.throws(() => parseWorkspaceState({ ...state, coverage: { limit: 0, truncated: 'false' } }), { message: '设备返回的内容缺少开关值。' }) + assert.throws(() => parseWorkspaceState([]), { message: '设备返回的内容格式不对。' }) +}) + test("task labels use the accepted provider and leave summaries unknown", () => { const accepted = records({ works: [work("frozen", { kind: "task", resolved: { provider: "claude", model: null } })] }).works[0] assert.equal(taskAgent(accepted), "claude") diff --git a/packages/client/src/workspace/types.ts b/packages/client/src/workspace/types.ts index dd1c7ed0f..aa101e600 100644 --- a/packages/client/src/workspace/types.ts +++ b/packages/client/src/workspace/types.ts @@ -146,6 +146,19 @@ export type TurnRuntime = { stage: ProviderStage; execution: 'not_started' | 'started' | 'unknown'; attempted_at_ms: number switched_from: { provider: ProviderId; model: string | null; reasoning_effort: string | null } | null; switch_reason: FailureKind | null } +export const jobStates = { + queued: { rank: 10, phase: 'waiting' }, pending_delivery: { rank: 10, phase: 'waiting' }, + claimed: { rank: 20, phase: 'active' }, running: { rank: 30, phase: 'active' }, steering: { rank: 35, phase: 'active' }, + cancelling: { rank: 40, phase: 'active' }, publishing: { rank: 50, phase: 'active' }, reconciling: { rank: 80, phase: 'active' }, + completed: { rank: 60, phase: 'ended' }, interrupted: { rank: 60, phase: 'ended' }, cancelled: { rank: 60, phase: 'ended' }, + steered: { rank: 60, phase: 'steered' }, rejected: { rank: 65, phase: 'ended' }, failed: { rank: 65, phase: 'ended' }, + unknown: { rank: 100, phase: 'unknown' }, unverified: { rank: 100, phase: 'unknown' }, +} as const +export type JobState = keyof typeof jobStates +export function isJobState(state: string): state is JobState { return Object.hasOwn(jobStates, state) } +/** A steered input settles separately from the reply whose stream it joined. */ +export const isReplyEnded = (state: string) => isJobState(state) && jobStates[state].phase === 'ended' +export const isTerminalJobState = (state: string) => isReplyEnded(state) || state === 'steered' export type ManagerJob = { trigger: string state: string diff --git a/packages/docs/ci.md b/packages/docs/ci.md index d0133c205..d4b820f8b 100644 --- a/packages/docs/ci.md +++ b/packages/docs/ci.md @@ -160,11 +160,13 @@ controller 在真实 key1 后、C0 快照前开放配对,成员随原节点数 ## 定期、手动和发布检查 -定时检查每 4 小时运行一次,使用 `23 */4 * * *`,每日 UTC 00:23、04:23、08:23、12:23、16:23、20:23 发起。定时和 Actions 中手动运行 `ci` 强制执行扩展检查,不受路径影响。merge queue 执行整个工作区的核心检查,不启用慢测试、MSRV 或三平台打包。开发分支 PR 不触发 `ci`。产品 `v` Tag 由 `release.yml` 并行构建全部产品目标,`www-v` 独立构建官网。`ci` 的手动入口只验证和生成制品,不发布。在 GitHub Actions 选择 `ci` → `Run workflow`,选择待验收分支后运行。 +定时检查保留每 4 小时的 `23 */4 * * *`,UTC 槽位为 00:23、04:23、08:23、12:23、16:23、20:23。GitHub schedule 是 best effort,可能延迟或跳过,不保证每槽运行。定时和 Actions 手动运行 `ci` 强制执行扩展检查,不受路径影响。merge queue 执行整个工作区的核心检查,不启用慢测试、MSRV 或三平台打包。开发分支 PR 不触发 `ci`。产品 `v` Tag 由 `release.yml` 并行构建全部产品目标,`www-v` 独立构建官网。手动 `ci` 只验证和生成制品,不自动补发 Nightly。 -每四小时 Nightly 接在同一 schedule run 的 `ci-gate` 后,不新增第二套定时全量 CI。`daily-nightly` 仅接受 main 的 schedule,以显式 `!cancelled()` 和 `ci-gate` success 作为条件,允许上游不适用的检查正常跳过。协调器使用固定 concurrency 且不取消进行中的协调。每四小时机械版本 PR 自动 squash 已授权并默认开启,需配置专用单仓库 `NIGHTLY_RELEASE_TOKEN`。该 PAT 需 Contents、Pull requests、Actions 所需权限,只处理版本 PR 创建和普通 squash;版本 branch、tag 与精确 workflow dispatch 使用 job 级 `GITHUB_TOKEN`。未配置时明确失败,不修改权限或保护规则。 +`daily-nightly` 接在同一 main schedule run 的 `ci-gate` 后,要求 `!cancelled()` 和 gate success,不增加第二套定时全量 CI。协调器全局串行,只使用 `GITHUB_TOKEN` 创建不可变 Tag 和精确 dispatch,不需要 PAT、版本分支、PR 或 main merge。 -协调脚本先验证精确机械版本 diff 和 `cargo metadata --locked --no-deps`,合并后只接受真实 squash SHA 的 main push CI success,再并行启动产品与官网发布组。不会为版本分支再派发一轮全量检查,也不为合并 SHA 再跑扩展全量 CI。原 run 的 `created_at` 转为 UTC 后向下取整到四小时批次,使用 `bot/daily-nightly/YYYYMMDDHH` 分支;版本日期保留 `YYYYMMDD`。不同批次分配新 PR 和递增版本,Android versionCode 与 iOS buildNumber 各递增一次;延迟启动或跨日重跑复用原批次 PR、版本、构建号、SHA 和留存制品。旧批次若尚无分支与 PR,而 main 的产品或官网 Nightly 日期已更晚,则拒绝初始化,避免版本倒退。quick-check 与 preflight 注册 `python3 scripts/daily-nightly.py --self-test` 及 Python 语法检查,涵盖六个 UTC 批次、同批次恢复、同日递增、旧批次拒绝倒退、真实临时 Git、篡改 diff、完整产品发布门、未知写入对账和原 artifact 恢复。具体发布顺序与停止条件见 [每 4 小时 Nightly 的发布约束](releasing.md#每-4-小时-nightly-的发布约束)。 +版本提交只使用原 schedule 的精确 main SHA 作 parent。共享来源门核对普通单 parent、完整机械 tree、版本 cohort、两个移动构建号及 `cargo metadata --locked --no-deps`。发布 workflow 验证原 run 的仓库、workflow path、main、SHA 与 `ci-gate` success,不等待包含协调 job 的整体 run success。真正的 artifact source 与 provenance 仍是 Tag 的版本提交。main 上的普通 Tag 继续要求同一 SHA 的 main push CI success,Stable/RC 不允许 detached 来源。 + +原 run 的 UTC `created_at` 决定四小时批次;延迟和跨日恢复沿用原版本、构建号、SHA、运行与留存 artifact。构建号从 main 和上一可信产品 Nightly 取数值上界加一,iOS 分段溢出时停止。上一来源完整发布且 npm 成功、main 无新提交时跳过;部分 Tag、Release、npm 先恢复原来源,不新建版本覆盖。quick-check 与 preflight 执行现有 Nightly 自测,覆盖真实临时 Git、机械 tree 篡改、原 schedule CI 门、固定 parent、构建号数值比较、同批次及部分发布恢复、Tag 冲突和未知写入对账。详情见 [Nightly 发布约束](releasing.md#每-4-小时-nightly-的发布约束)。 | 扩展检查 | 执行范围 | | --- | --- | @@ -178,13 +180,19 @@ controller 在真实 key1 后、C0 快照前开放配对,成员随原节点数 | 公共节点、排队与私有 relay 令牌验收 | Linux x64 一次,使用 `--public-node` | | 全网入网、志愿 relay、relay 选择与无网页 Node 验收 | Linux x64 一次,使用 `--mesh` | | 写入保护、状态与诊断包验收 | macOS ARM64 一次,使用 `--phase5` | -| Nightly npm 包(当前声明版本为 Nightly 时) | 组装四个原生包,四个平台分别安装并检查版本,POSIX 平台另执行 `--web-only` 冒烟 | +| Nightly npm 包(当前声明版本为 Nightly 时) | 组装四个原生包,四个平台分别安装并检查版本;macOS ARM64、Linux x64/ARM64 另执行 `--web-only` 和 `--npm-service` 原生服务验收 | + +`--npm-service` 仅允许无既有 Vibeface 数据或默认服务的 GitHub hosted runner,使用真实 launchd 或 systemd user manager,不能在开发者机器运行。Linux job 显式准备 user manager 和 D-Bus;不可用即失败。验收通过公开 npm CLI 执行安装、启动、停止、重启和卸载,核对服务使用 scoped npm vendor 中的原生程序。 + +`npm-package` 每次只读取一次公开 `nightly`,固定版本、registry integrity、完整产品 manifest 及全部 vendor 字节,使用共享发布来源门证明基线业务来源是候选源码的祖先。基线单独上传,三个平台按不可变 artifact ID 下载同一份,不混入候选发布 artifact。来源相同、基线业务来源超前或 data epoch 不兼容均失败。 + +受控 Node 和 App Server 夹具从不可变的公开基线包创建真实非空消息及附件,安装真实服务并核对运行状态与数据,再停服,以独立 prefix 核验的本 CI 候选包替换原 prefix。两包都验证活动 Node、updater 锁、pending journal 和第二 updater 的无修改拒绝。完整的安装、状态、停止、启动、重启换 PID 和卸载生命周期由最终候选包验收;重启后核对原身份、配置、签名历史和附件字节,并确认已完成的受控调用未重放,卸载保留数据。公开基线不承担完整生命周期验收。收据分别记录两包的真实 version、source、摘要和来源关系;候选声明版本相同或较低时如实标注,不称为 SemVer 升级。只验证当前格式,不做跨历史格式迁移,也不据机械版本变化宣称业务升级或真实 Provider 验收。 快、慢测试组均使用 `opt-level=1`,保留调试断言和整数溢出检查,调试信息使用 `line-tables-only` 保留回溯文件和行号,共享独立的 `test-opt1` Rust 缓存;Clippy 使用单独的 `check` 缓存。两组运行前校验每个慢测试名恰好匹配一个已有测试。慢组用 `--include-ignored` 运行列出的用例,其他 `ignored` 用例仍默认忽略。两组测试命令用 `/usr/bin/time -v` 记录 CPU 与 maxRSS;maxRSS 不能当作并发进程树的总峰值。 CI 不使用真实 AI Provider 或模型密钥。各模式都使用受控 SDK 和 App Server 夹具,`--phase5` 不启动 App Server。模拟协议通过不能证明真实模型理解需求或正确选择工具。 -产品 `v` 与官网 `www-v` 分别进入发布路径,必须匹配声明版本与 main 中的固定源码。产品并行构建 Web、四个原生目标、Android 和 iOS,各平台再按自身输入组包并构建桌面 App。每组校验后立即向同一个公开 Release 追加附件,完整审计完成后才追加全局 `release.json`。`publish=false` 手动入口只验证并生成制品。产品与官网制品发布不触发部署;独立部署命令可显式选择已发布的 Stable、RC 或 Nightly Tag;派发成功不代表部署成功。详见 [构建与发布](releasing.md)。 +产品 `v` 与官网 `www-v` 分别进入发布路径,必须匹配声明版本,并通过同一共享来源门验证 main 源码或受信机械 Nightly 提交。产品并行构建 Web、四个原生目标、Android 和 iOS,各平台再按自身输入组包并构建桌面 App。每组校验后立即向同一个公开 Release 追加附件,完整审计完成后才追加全局 `release.json`。`publish=false` 手动入口只验证并生成制品。产品与官网制品发布不触发部署;独立部署命令可显式选择已发布的 Stable、RC 或 Nightly Tag;派发成功不代表部署成功。详见 [构建与发布](releasing.md)。 Windows 专项使用当前源码,不接受历史运行包。只检查原生 Rust 时运行: diff --git a/packages/docs/releasing.md b/packages/docs/releasing.md index 1aecddc07..5d5f63842 100644 --- a/packages/docs/releasing.md +++ b/packages/docs/releasing.md @@ -26,15 +26,15 @@ Core、CLI、Web App、Desktop 和 Mobile 共享一个版本、一个 `v- 自定义 Rust 编译参数通过 `RUSTFLAGS` 或 `CARGO_ENCODED_RUSTFLAGS` 传入,后者优先。发布脚本保留这些参数并追加路径映射;按 Cargo 的优先级,此时不使用配置文件中的 `rustflags`。 -运行任务前须另装并登录相应 Provider。Cursor 管理路径使用已安装的官方 Agent CLI 及其订阅登录,显式环境认证仍可使用。初始化、配对、服务安装和公网 TLS 见 [Node 使用说明](node.md)。 +运行任务前须另装并登录相应 Provider。Cursor 管理路径使用已安装的官方 Agent CLI 和订阅登录,继承的认证环境变量会被清除,显式认证覆盖也被拒绝。初始化、配对、服务安装和公网 TLS 见 [Node 使用说明](node.md)。 ## CI 验证范围 @@ -87,7 +87,7 @@ cd vibeface-- 质量检查按计划选择受影响的 crate,包含 Rust fmt、快慢测试、Clippy、声明的最低 Rust 版本编译检查、工作流和 shell 静态检查、JavaScript 语法、Web TypeScript 检查和静态构建,以及 Atlas 迁移文件校验。Atlas 不打开用户数据库。README 和项目指引不触发构建,`packages/docs/` 触发门户构建;`main` 的纯 App 改动只构建 Linux x64 包并验收网页资源,不运行 Core 库业务测试。`main` 保留相关快速测试、Clippy 和 Linux x64 制品。慢测试、最低 Rust 版本和三平台制品在扩展检查执行。`ci-gate` 要求本次计划中的任务全部成功。 -打包任务校验 SHA256、源码提交、版本、target、密码学资源和存储 epoch。扩展检查在 Linux x64 启动解包后的程序运行无账号网络验收;每个选中的平台独立启动网页进程,检查 SPA 路由、JavaScript、CSS、字体、每段内联脚本的 CSP hash、HEAD、缓存策略和未知资源的 404。Linux x64 还安装包内锁定的 SDK 依赖并导入模块,不开启模型会话。 +打包任务校验 SHA256、源码提交、版本、target、密码学资源和存储 epoch。扩展检查在 Linux x64 启动解包后的程序运行无账号网络验收;每个选中的平台独立启动网页进程,检查 SPA 路由、JavaScript、CSS、字体、每段内联脚本的 CSP hash、HEAD、缓存策略和未知资源的 404。安装包不含 Provider SDK,验收不安装或导入 Provider SDK,也不开启模型会话。 源码打包前先生成密码学资源,再构建网页。安装 Node.js 22.12+、pnpm 11.9.0、Rust 的 `wasm32-unknown-unknown` target 和 `wasm-bindgen-cli 0.2.129`,然后执行: @@ -206,7 +206,7 @@ npm trust github @leaperone/vibeface --repo leaperone/vibeface --file npm-publis GitHub 的同名 environment 及 tag 访问策略由仓库远端设置控制。仓库代码不会自动创建 npm trust,也不能证明设置已生效。保存 trust 后仍需一次真实 tag 发布确认 OIDC、provenance 和公开 registry 结果;本地自检和 dry run 不证明真实发布权限。不要把登录凭证发到聊天或写入仓库。 -手动入口默认 `publish=false`,只接受与输入相同的 tag ref。它要求该 tag 和源码 SHA 只有一个成功的 `release.yml` run,以及一个有效 npm artifact;无法唯一定位时停止。该 tag 必须已经包含本工作流与脚本。dry run 命令如下: +手动入口默认 `publish=false`,只接受与输入相同的 tag ref。它要求该 Tag 和源码 SHA 只有一个已完成的 `release.yml` run,且 `collect`、`publish-product` 均成功,再使用唯一有效的原 npm artifact。总 run 可因 npm 失败而标红,不要求整体 success;无法唯一定位时停止。该 tag 必须已经包含本工作流与脚本。dry run 命令如下: ```sh gh workflow run npm-publish.yml --repo leaperone/vibeface --ref v \ diff --git a/packages/docs/service-deployments.md b/packages/docs/service-deployments.md index 5d93339ce..f14f26e00 100644 --- a/packages/docs/service-deployments.md +++ b/packages/docs/service-deployments.md @@ -31,11 +31,15 @@ python3 scripts/deploy.py release --service all --tag '' ```sh python3 scripts/deploy.py release --service www --tag '' -gh workflow run deploy-services.yml --ref '' -f service=app -f tag='' -F apply=false +gh workflow run deploy-services.yml --ref '' -f service=node -f tag='' -F apply=false ``` 真实激活仅允许从 main 执行。每次运行重新核对最终 `release.json` 的 Tag、源码 SHA、main 归属、附件集合和摘要,再下载需要的组件并核对实际字节与包内来源。容器只包装这些已发布字节,通过 SSH 传输并按 image ID 激活,无需共享镜像仓库。 +激活前同时核对传输归档的 SHA256 和导入后的 image ID。归档中的镜像必须与调用方指定的 `--image` 一致;摘要正确不能替代镜像身份校验。 + +`node` 无论是否 `apply` 都先在 runner 演练。脚本选择候选发布时间之前最近的完整产品 Release,排除候选本身、Draft 和缺少最终 manifest 或必要组件的发布,并核对其 `release.json`、来源和附件摘要。旧 Core 初始化并启动隔离 fixture 后,脚本原子换入候选 Core 生成的完整配置,再激活候选镜像并验证重复激活、原身份和初始化数据保留。fixture 只用回环地址、独立 mesh 和生成的数据,不带 Provider 或部署凭证。没有历史完整产品 Release 时明确输出 `first_release`,只演练首次安装和重复激活;查询、校验或升级失败都停止,不退到更老版本或略过升级。 + ## 配置部署目标 为每个目标创建 GitHub Environment `vibeface-node`、`vibeface-app`、`vibeface-www`。按需设置 Environment 的 main 分支限制和人工审核。仅验证使用不含生产凭证的 `vibeface-validation`。 diff --git a/scripts/build-inputs.py b/scripts/build-inputs.py index 5e0fa9ba8..2b822f93b 100644 --- a/scripts/build-inputs.py +++ b/scripts/build-inputs.py @@ -5,6 +5,7 @@ import fnmatch import hashlib import json +import os import shutil import subprocess import tempfile @@ -185,7 +186,7 @@ def check(repo, component): def self_test(): - scratch = ROOT / ".artifacts/build-inputs-test" + scratch = Path(os.environ.get("VIBEFACE_RELEASE_SCRATCH", ROOT / ".artifacts/build-inputs-test")) scratch.mkdir(parents=True, exist_ok=True) with tempfile.TemporaryDirectory(prefix="fixture-", dir=scratch) as directory: repo = Path(directory) diff --git a/scripts/check-release-workflow.py b/scripts/check-release-workflow.py index 0cc5cecfa..ec2c227c0 100644 --- a/scripts/check-release-workflow.py +++ b/scripts/check-release-workflow.py @@ -80,6 +80,14 @@ def check(source, helpers): for name, block in workflow.items()), 'artifact publication must not deploy services') require('release-${{ inputs.tag || github.ref_name }}-' in source, 'tag pushes and manual publication must share the same concurrency group') + identity = workflow['identity'] + require("command += ['--require-main', '--publish']" in identity + and 'release.require_ci(result)' in identity + and "result['source_commit'] != os.environ['GITHUB_SHA']" in identity + and "os.environ['GITHUB_REPOSITORY'] != release.GITHUB_REPO" in identity, + 'publication identity must prove exact source, repository and shared CI rules') + require('scripts/release.py plan --app product --tag "$RELEASE_TAG" --require-main --publish' in helpers['npm-publish.yml'], + 'npm publication must enforce the same release source rule') producers = ['static', 'android', 'ios', *('native-' + target for target in TARGETS)] for producer in producers: require(needs(workflow[producer]) == {'identity'}, producer + ' waits on an unrelated build') @@ -236,6 +244,8 @@ def self_test(source, helpers): check(source, helpers) helper = helpers['release-publish-part.yml'] broken = [ + source.replace('release.require_ci(result)', 'pass'), + source.replace("command += ['--require-main', '--publish']", "command += ['--publish']"), source + '\n deploy-app:\n uses: ./.github/workflows/deploy-services.yml\n', source.replace('needs: [identity, static, native-macos]', 'needs: [identity, static, native-macos, native-linux-x64]'), source.replace('needs: [identity, compose-macos]', 'needs: [identity, compose-macos, compose-windows]'), diff --git a/scripts/ci.py b/scripts/ci.py index 4c27c155d..215d3026d 100644 --- a/scripts/ci.py +++ b/scripts/ci.py @@ -13,6 +13,8 @@ import tempfile import urllib.request +import release + PACKAGES = ("vibeface-protocol", "vibeface-bridge", "vibeface-secure", "vibeface-core", "vibeface-control", "vibeface", "vibeface-secure-ffi", "vibeface-delivery") DEPENDENTS = { @@ -282,7 +284,7 @@ def release_metadata(ref_type, ref_name): if ref_type == "tag": if ref_name != f"v{version}": raise ValueError("Release tag must exactly match the Cargo version") - git("merge-base", "--is-ancestor", "HEAD", "origin/main") + release.validate_source(git("rev-parse", "HEAD").decode().strip(), "product") return {"version": version, "msrv": package["rust_version"], "channel": channel, "prerelease": channel != "stable"} @@ -482,7 +484,7 @@ def self_test(): for path in ("README.md", "README.zh-CN.md"): assert make_plan([path])["required_jobs"] == ["quick-check"] assert make_plan(["README.zh-CN.md", "apps/core/src/api.rs"])["rust_packages"] == ["vibeface-core"] - for path in ("README.zh-CN.md.rs", ".agents/skills/release/SKILL.md", "scripts/release.py", "scripts/daily-nightly.py", + for path in ("README.zh-CN.md.rs", ".agents/skills/release/SKILL.md", "scripts/release.py", "scripts/daily-nightly.py", "scripts/nightly_source.py", "apps/desktop/electron-builder.cjs", "scripts/build-desktop.mjs"): assert make_plan([path])["full"] assert docs["local_process_modes"] == [] and www["local_process_modes"] == [] @@ -650,7 +652,7 @@ def self_test(): expect_failure(lambda: validate_slow_tests(["other::test", *SLOW_TESTS[:-1]])) expect_failure(lambda: validate_slow_tests(["other::test", *SLOW_TESTS, SLOW_TESTS[0] + "_extra"])) expect_failure(lambda: package_args(["vibeface", "vibeface"])) - scratch = Path(__file__).resolve().parent.parent / ".artifacts/ci-self-test" + scratch = Path(os.environ.get("VIBEFACE_RELEASE_SCRATCH", Path(__file__).resolve().parent.parent / ".artifacts/ci-self-test")) scratch.mkdir(parents=True, exist_ok=True) with tempfile.TemporaryDirectory(prefix="git-", dir=scratch) as directory: git("init", "--quiet", cwd=directory) @@ -697,10 +699,10 @@ def self_test(): coverage_self_test(directory, base, new_base) for version, channel in (("1.2.3", "stable"), ("1.2.3-rc.2", "rc"), ("1.2.3-nightly.20261019.1", "nightly")): cargo_metadata = {"packages": [{"name": "vibeface", "version": version, "rust_version": "1.89"}]} - with patch("subprocess.check_output", return_value=json.dumps(cargo_metadata)), patch(f"{__name__}.git") as ancestor: + with patch("subprocess.check_output", return_value=json.dumps(cargo_metadata)), patch(f"{__name__}.git", return_value=b"a" * 40), patch("release.validate_source") as ancestor: assert release_metadata("branch", "")["channel"] == channel assert release_metadata("tag", f"v{version}")["prerelease"] == (channel != "stable") - ancestor.assert_called_once_with("merge-base", "--is-ancestor", "HEAD", "origin/main") + ancestor.assert_called_once_with("a" * 40, "product") expect_failure(lambda: release_metadata("tag", "v0.0.0")) for version in ("01.2.3", "1.2.3-nightly.20260229.1", "1.2.3-nightly.20261019.0", "1.2.3+build"): cargo_metadata = {"packages": [{"name": "vibeface", "version": version, "rust_version": "1.89"}]} diff --git a/scripts/daily-nightly.py b/scripts/daily-nightly.py index 53e0a9960..7c89da5a0 100644 --- a/scripts/daily-nightly.py +++ b/scripts/daily-nightly.py @@ -1,8 +1,6 @@ #!/usr/bin/env python3 -"""Land one mechanical version PR per four-hour UTC batch, then reconcile both Nightlies.""" +"""Tag one mechanical source per four-hour UTC batch, then reconcile both Nightlies.""" import argparse -from contextlib import contextmanager -from dataclasses import dataclass import datetime import json import os @@ -16,13 +14,14 @@ sys.dont_write_bytecode = True import release +from nightly_source import (DailyPlan, BOT, PATHS, blob, checkout, generated, git, metadata, + next_build_numbers, next_ios_build_number, nightly_order, recovered_plan, + scratch, verify_tree, versions_at) ROOT = Path(__file__).resolve().parent.parent REPO = 'leaperone/vibeface' CI = '.github/workflows/ci.yml' RELEASE = '.github/workflows/release.yml' -PATHS = ('apps/www/package.json', 'apps/app/package.json', 'apps/core/Cargo.toml', - 'apps/cli/Cargo.toml', 'Cargo.lock', 'apps/mobile/app.json', 'apps/desktop/package.json') PRODUCT_APPS = release.PRODUCT_APPS RELEASES = release.RELEASES PRODUCER = 'collect' @@ -38,53 +37,17 @@ 'publish-desktop-x86_64-pc-windows-msvc', 'publish-android', 'publish-ios') PUBLISHERS = ('release-start', *PART_PUBLISHERS, 'publish-product', 'release-failed', 'publish', 'npm') -BOT = '41898282+github-actions[bot]@users.noreply.github.com' require = release.require -@dataclass(frozen=True) -class DailyPlan: - batch: str - base: str - versions: dict - android_code: int - ios_build_number: str - - def __post_init__(self): - slot = datetime.datetime.strptime(self.batch, '%Y%m%d%H') - require(slot.strftime('%Y%m%d%H') == self.batch and slot.hour % 4 == 0, - 'Nightly batch must be a four-hour UTC slot') - - @property - def day(self): - return self.batch[:8] - - @property - def branch(self): - return 'bot/daily-nightly/' + self.batch - - def schedule_batch(created_at): created = datetime.datetime.fromisoformat(created_at.replace('Z', '+00:00')).astimezone(datetime.timezone.utc) return created.replace(hour=created.hour // 4 * 4).strftime('%Y%m%d%H') -def git(*args, root=None, data=None, env=None): - return subprocess.check_output(['git', *args], cwd=root or ROOT, input=data, env=env).decode().strip() - - -def blob(commit, path): - return subprocess.check_output(['git', 'show', f'{commit}:{path}'], cwd=ROOT) - - -def api(path, method='GET', data=None, pat=False, missing=False): +def api(path, method='GET', data=None, missing=False): environment = dict(os.environ) - if pat: - require(environment.get('NIGHTLY_RELEASE_TOKEN'), 'NIGHTLY_RELEASE_TOKEN is missing') - environment['GH_TOKEN'] = environment['NIGHTLY_RELEASE_TOKEN'] args = ['gh', 'api', '--method', method, 'repos/' + REPO + '/' + path] - if path == '/user': - args[-1] = 'user' if data is not None: args += ['--input', '-'] result = subprocess.run(args, input=json.dumps(data) if data is not None else None, @@ -122,41 +85,6 @@ def reconcile_write(write, read, accepted, description): raise ValueError(description + ' has an unknown or rejected outcome; no write was repeated') -def json_value(text, path, value): - data = json.loads(text) - require(json.dumps(data, indent=2, ensure_ascii=False) + '\n' == text, 'version JSON formatting changed; refuse unrelated formatting edits') - parent = data - for key in path[:-1]: - parent = parent[key] - require(path[-1] in parent, 'missing JSON version field') - parent[path[-1]] = value - return json.dumps(data, indent=2, ensure_ascii=False) + '\n' - - -def package_version(text, value): - match = re.search(r'(?ms)^\[package\]\s*\n(.*?)(?=^\[|\Z)', text) - require(match, 'missing Cargo package section') - section, count = re.subn(r'(?m)^(version\s*=\s*)"[^"]+"', lambda m: m[1] + json.dumps(value), match[1]) - require(count == 1, 'Cargo package must own exactly one version') - return text[:match.start(1)] + section + text[match.end(1):] - - -def versions_at(commit): - values = {} - for app in release.APPS: - path = f'apps/{app}/' + ('Cargo.toml' if app in ('core', 'cli') else 'app.json' if app == 'mobile' else 'package.json') - text = blob(commit, path).decode() - if app in ('core', 'cli'): - match = re.search(r'(?ms)^\[package\]\s*\n(.*?)(?=^\[|\Z)', text) - require(match, 'missing Cargo package') - values[app] = re.search(r'(?m)^version\s*=\s*"([^"]+)"', match[1])[1] - else: - info = json.loads(text) - values[app] = info['expo']['extra']['releaseVersion'] if app == 'mobile' else info['version'] - release.version(values[app]) - return values - - def declared_versions(tags): """Versions already claimed by real release tags, per release group.""" values = {group: [] for group in RELEASES} @@ -175,21 +103,7 @@ def next_sequence(prefix, claimed): return prefix + str(max(sequences, default=0) + 1) -def next_ios_build_number(value): - """Increment the last valid CFBundleVersion component without rollover.""" - require(isinstance(value, str), 'iOS build number must be a string') - parts = value.split('.') - require(1 <= len(parts) <= 3 and re.fullmatch(r'[0-9]{1,4}', parts[0]) - and int(parts[0]) > 0 and all(re.fullmatch(r'[0-9]{1,2}', part) for part in parts[1:]), - 'iOS build number must be a valid Apple CFBundleVersion') - maximum = 9999 if len(parts) == 1 else 99 - last = int(parts[-1]) - require(last < maximum, 'iOS build number component cannot be incremented without rollover') - parts[-1] = str(last + 1) - return '.'.join(parts) - - -def plan_versions(batch, base, tags): +def plan_versions(batch, base, tags, run_id, previous=''): day = batch[:8] values = versions_at(base) for app in ('cli', 'www'): @@ -203,207 +117,8 @@ def plan_versions(batch, base, tags): values['www'] = next_sequence(www_prefix, claimed['www'] + [values['www']]) for app in PRODUCT_APPS: values[app] = product_ver - expo = json.loads(blob(base, 'apps/mobile/app.json'))['expo'] - code = expo['android']['versionCode'] - require(type(code) is int and 0 < code < 2100000000, 'Android versionCode cannot be incremented') - ios_build_number = next_ios_build_number(expo['ios']['buildNumber']) - return DailyPlan(batch, base, values, code + 1, ios_build_number) - - -def generated(plan): - old = versions_at(plan.base) - require(set(plan.versions) == set(release.APPS), 'daily plan must contain all release applications') - product_ver = plan.versions['cli'] - for app in PRODUCT_APPS: - require(plan.versions[app] == product_ver, 'product apps must share one canonical version') - for app, value in plan.versions.items(): - release.version(value) - prefix = old[app].split('-', 1)[0] + '-nightly.' + plan.day + '.' - require(value.startswith(prefix), 'daily version changed baseline or date') - if old[app].startswith(prefix): - require(int(value[len(prefix):]) > int(old[app][len(prefix):]), 'daily version did not advance') - result = {path: blob(plan.base, path).decode() for path in PATHS} - for app in ('www', 'app', 'desktop'): - path = f'apps/{app}/package.json' - result[path] = json_value(result[path], ('version',), plan.versions[app]) - for app in ('core', 'cli'): - path = f'apps/{app}/Cargo.toml' - result[path] = package_version(result[path], plan.versions[app]) - blocks = result['Cargo.lock'].split('[[package]]') - found = set() - for index, block in enumerate(blocks): - match = re.search(r'(?m)^name = "(vibeface|vibeface-core)"$', block) - if match: - app = 'cli' if match[1] == 'vibeface' else 'core' - require(app not in found and f'version = "{old[app]}"' in block, 'Cargo.lock local package mismatch') - blocks[index], count = re.subn(r'(?m)^version = "[^"]+"$', 'version = "' + plan.versions[app] + '"', block) - require(count == 1, 'ambiguous local lock version') - found.add(app) - require(found == {'core', 'cli'}, 'Cargo.lock is missing local packages') - result['Cargo.lock'] = '[[package]]'.join(blocks) - mobile_path = 'apps/mobile/app.json' - expo = json.loads(result[mobile_path])['expo'] - require(expo['version'] == plan.versions['mobile'].split('-', 1)[0], 'mobile marketing version mismatch') - require(type(plan.android_code) is int and plan.android_code == expo['android']['versionCode'] + 1 - and 0 < plan.android_code <= 2100000000, 'Android versionCode must advance exactly once') - require(plan.ios_build_number == next_ios_build_number(expo['ios']['buildNumber']), - 'iOS build number must advance exactly once') - result[mobile_path] = json_value(result[mobile_path], ('expo', 'extra', 'releaseVersion'), plan.versions['mobile']) - result[mobile_path] = json_value(result[mobile_path], ('expo', 'android', 'versionCode'), plan.android_code) - result[mobile_path] = json_value(result[mobile_path], ('expo', 'ios', 'buildNumber'), plan.ios_build_number) - return {path: text.encode() for path, text in result.items()} - - -def recovered_plan(batch, head): - parents = git('rev-list', '--parents', '-n', '1', head).split() - require(len(parents) == 2, 'daily version head must be one ordinary commit') - expo = json.loads(blob(head, 'apps/mobile/app.json'))['expo'] - return DailyPlan(batch, parents[1], versions_at(head), expo['android']['versionCode'], expo['ios']['buildNumber']) - - -def expected_tree(plan): - expected = generated(plan) - with tempfile.TemporaryDirectory(dir=scratch(), prefix='index-') as temporary: - environment = dict(os.environ, GIT_INDEX_FILE=str(Path(temporary) / 'index')) - git('read-tree', plan.base, env=environment) - for path, content in expected.items(): - entry = git('ls-tree', plan.base, '--', path).split() - require(entry[:2] == ['100644', 'blob'], 'version source must be an ordinary tracked file') - oid = git('hash-object', '-w', '--stdin', data=content) - git('update-index', '--add', '--cacheinfo', '100644', oid, path, env=environment) - return git('write-tree', env=environment) - - -def verify_tree(plan, head): - require(git('show', '-s', '--format=%ae%n%ce', head).splitlines() == [BOT, BOT], 'unexpected daily commit author') - require(git('rev-list', '--parents', '-n', '1', head).split() == [head, plan.base], 'daily version head changed its original base') - require(expected_tree(plan) == git('rev-parse', head + '^{tree}'), - 'daily PR tree differs from exact mechanical version changes; refusing merge') - - -def scratch(): - directory = ROOT / '.artifacts/daily-nightly' - directory.mkdir(parents=True, exist_ok=True) - return directory - - -@contextmanager -def checkout(commit): - with tempfile.TemporaryDirectory(dir=scratch(), prefix='checkout-') as temporary: - source = Path(temporary) / 'source' - git('worktree', 'add', '--detach', str(source), commit) - try: - yield source - finally: - git('worktree', 'remove', '--force', str(source)) - - -def metadata(source, plan): - before = (source / 'Cargo.lock').read_bytes() - result = subprocess.check_output(['cargo', 'metadata', '--locked', '--no-deps', '--format-version', '1'], cwd=source) - packages = {package['name']: package['version'] for package in json.loads(result)['packages']} - require(packages['vibeface'] == plan.versions['cli'] and packages['vibeface-core'] == plan.versions['core'], - 'Cargo metadata does not match daily versions') - require((source / 'Cargo.lock').read_bytes() == before, 'cargo metadata changed dependency resolution') - - -def daily_pr(branch): - matches = pages('pulls?state=all&head=' + quote('leaperone:' + branch, safe='')) - require(len(matches) <= 1, 'daily branch has multiple PRs; refusing a second daily PR') - return api(f'pulls/{matches[0]["number"]}') if matches else None - - -def branch_sha(branch): - ref = api('git/ref/heads/' + branch, missing=True) - return ref['object']['sha'] if ref else None - - -def validate_pr(pr, plan, head, login): - require(pr['user']['login'] == login and pr['head']['repo']['full_name'] == REPO - and pr['base']['repo']['full_name'] == REPO and pr['base']['ref'] == 'main' - and pr['head']['ref'] == plan.branch and pr['head']['sha'] == head - and pr['commits'] == 1 and not pr['draft'] and pr['title'] == 'chore(release): nightly ' + plan.batch, - 'daily PR identity, author, base or head mismatch') - require(pr['state'] == 'open' or pr.get('merged'), 'daily PR was closed without merge; refusing a replacement') - verify_tree(plan, head) - - -def create_pr(plan, head, login): - def read(): - return daily_pr(plan.branch) - pr = reconcile_write(lambda: api('pulls', 'POST', dict(head=plan.branch, base='main', - title='chore(release): nightly ' + plan.batch, - body='Mechanical four-hour Nightly versions. Only the trusted Nightly coordinator may merge this PR.'), pat=True), - read, lambda value: value is not None, 'daily PR creation') - validate_pr(pr, plan, head, login) - return pr - - -def merge_pr(pr, plan, head, login): - validate_pr(pr, plan, head, login) - if not pr.get('merged'): - git('fetch', 'origin', 'main') - git('merge-base', '--is-ancestor', plan.base, 'origin/main') - with checkout(head) as source: - metadata(source, plan) - for attempt in range(18): - pr = api(f'pulls/{pr["number"]}') - validate_pr(pr, plan, head, login) - if pr['mergeable'] is not None: - break - if attempt < 17: - time.sleep(10) - require(pr['mergeable'] is True, 'daily PR is conflicted or mergeability is unknown; rerun after reconciliation') - number = pr['number'] - pr = reconcile_write(lambda: api(f'pulls/{number}/merge', 'PUT', dict(sha=head, merge_method='squash'), pat=True), - lambda: api(f'pulls/{number}'), lambda value: bool(value.get('merged')), 'daily squash merge') - validate_pr(pr, plan, head, login) - source = pr['merge_commit_sha'] - require(re.fullmatch(r'[a-f0-9]{40}', source or ''), 'merged PR has no actual merge SHA') - git('fetch', 'origin', 'main') - git('merge-base', '--is-ancestor', source, 'origin/main') - parents = git('rev-list', '--parents', '-n', '1', source).split() - require(len(parents) == 2, 'daily squash merge must have one real main parent') - landed = DailyPlan(plan.batch, parents[1], plan.versions, plan.android_code, plan.ios_build_number) - require(git('rev-parse', source + '^{tree}') == expected_tree(landed), - 'actual main merge differs from exact mechanical versions on its real parent; retain original PR and stop') - return source - - -def ensure_landed(batch, login): - branch = 'bot/daily-nightly/' + batch - pr = daily_pr(branch) - remote = branch_sha(branch) - if pr: - head = pr['head']['sha'] - require(remote is None and pr.get('merged') or remote == head, 'daily branch no longer matches original PR head') - git('fetch', 'origin', f'pull/{pr["number"]}/head') - plan = recovered_plan(batch, head) - elif remote: - git('fetch', 'origin', 'refs/heads/' + branch) - head = remote - plan = recovered_plan(batch, head) - verify_tree(plan, head) - pr = create_pr(plan, head, login) - else: - git('fetch', 'origin', 'main', '--tags') - base = git('rev-parse', 'origin/main') - tags = [line.split('\t')[1][10:] for line in git('ls-remote', '--tags', 'origin').splitlines() - if not line.endswith('^{}')] - plan = plan_versions(batch, base, tags) - with checkout(base) as source: - for path, content in generated(plan).items(): - (source / path).write_bytes(content) - metadata(source, plan) - git('add', '--', *PATHS, root=source) - git('-c', 'user.name=github-actions[bot]', '-c', 'user.email=' + BOT, - 'commit', '-m', 'chore(release): nightly ' + batch, root=source) - head = git('rev-parse', 'HEAD', root=source) - verify_tree(plan, head) - reconcile_write(lambda: git('push', 'origin', head + ':refs/heads/' + branch), - lambda: branch_sha(branch), lambda value: value == head, 'daily branch push') - pr = create_pr(plan, head, login) - return plan, merge_pr(pr, plan, head, login), pr['number'] + code, ios = next_build_numbers(base, previous) + return DailyPlan(batch, base, values, code, ios, str(run_id), previous) def matching_runs(workflow, sha, branch, events): @@ -420,24 +135,6 @@ def rerun(run, failed_only): lambda value: value['run_attempt'] > attempt or value['status'] != 'completed', 'original run recovery') -def wait_main_ci(source, deadline): - restored = set() - while time.monotonic() < deadline: - runs = matching_runs(CI, source, 'main', {'push'}) - require(len(runs) <= 1, 'multiple main push CI runs for daily merge; reconcile before releasing') - if runs: - run = runs[0] - if run['status'] == 'completed': - if run['conclusion'] == 'success': - return run - require(run['conclusion'] == 'cancelled' and run['id'] not in restored, - 'exact daily merge main CI did not succeed; refusing tags') - restored.add(run['id']) - rerun(run, False) - time.sleep(15) - raise ValueError('exact daily merge main push CI is missing or pending; refusing tags') - - def tag_sha(tag): ref = api('git/ref/tags/' + tag, missing=True) if not ref: @@ -454,7 +151,7 @@ def tag_sha(tag): def ensure_tag(tag, source): existing = tag_sha(tag) if existing is None: - existing = reconcile_write(lambda: api('git/refs', 'POST', dict(ref='refs/tags/' + tag, sha=source)), + existing = reconcile_write(lambda: git('push', 'origin', source + ':refs/tags/' + tag), lambda: tag_sha(tag), lambda value: value is not None, 'release tag creation') require(existing == source, 'existing immutable tag points to a different SHA') @@ -549,6 +246,7 @@ def finish_release(app, source, deadline): run = runs[0] # A public Release with a failed npm job is not a success; it is a recoverable publisher failure. if state and not state['draft'] and run['conclusion'] == 'success': + require(app != 'product' or npm_complete(run), 'original Nightly npm publisher did not succeed') with tempfile.TemporaryDirectory(dir=scratch(), prefix='public-') as temporary: release.verify_public(app, Path(temporary) / 'assets') return [dict(id=run['id'], url=run['html_url'], conclusion=run['conclusion'])] @@ -559,9 +257,9 @@ def finish_release(app, source, deadline): raise ValueError('release workflow is still pending; rerun the original daily coordinator to reconcile') -def publish_daily(source, deadline): +def publish_daily(source, deadline, groups=RELEASES): receipt = {}; started = []; errors = [] - for app in RELEASES: + for app in groups: started.append(app) try: start_release(app, source) @@ -569,7 +267,7 @@ def publish_daily(source, deadline): errors.append(f'{app}: {error}') break if not errors: - for app in RELEASES: + for app in groups: if app not in started: continue try: @@ -590,49 +288,149 @@ def publish_daily(source, deadline): return receipt +def trusted_nightlies(tags, batch): + """Either group may be the sole surviving tag of a partially published source.""" + trusted = {} + for tag in tags: + try: + app, value, channel = release.parse_tag(tag) + except ValueError: + continue + if channel != 'nightly': + continue + source = git('rev-parse', tag + '^{commit}') + try: + require(set(git('ls-tree', '-r', '--name-only', source, '--', *PATHS).splitlines()) == set(PATHS), + 'Nightly candidate predates the current release layout') + context = release.validate_source(source, app) + if app == 'www' and not context['ci_run_id']: + continue + values = versions_at(source) + require(values['cli' if app == 'product' else 'www'] == value, 'Nightly tag version mismatch') + order = nightly_order(values['cli']) + except ValueError: + require(git('show', '-s', '--format=%s', source) != 'chore(release): nightly ' + batch, + 'existing Nightly batch tag has an invalid immutable source') + continue + trusted[source] = (order, tag, source, context) + return [(tag, source, context) for _, tag, source, context in sorted(trusted.values(), reverse=True)] + + +def npm_complete(run): + jobs = pages(f'actions/runs/{run["id"]}/jobs?filter=latest', 'jobs') + npm = [job for job in jobs if job_leaf(job) == 'npm'] + return len(npm) == 1 and npm[0]['status'] == 'completed' and npm[0]['conclusion'] == 'success' + + +def completed_source(source): + """A complete product includes the original npm publisher, not merely a public Release.""" + with checkout(source) as clean: + previous_root = release.ROOT + release.ROOT = clean + try: + groups = RELEASES if release.validate_source(source, 'product')['ci_run_id'] else ('product',) + for app in groups: + tag = release_dispatch_tag(app) + require(release.parse_tag(tag)[2] == 'nightly', 'recovery only accepts Nightly groups') + if tag_sha(tag) != source: + return False + state = release_state(tag) + runs = matching_runs(RELEASE, source, tag, {'push', 'workflow_dispatch'}) + if not (state and not state['draft'] and len(runs) == 1 + and runs[0]['status'] == 'completed' and runs[0]['conclusion'] == 'success'): + return False + if app == 'product' and not npm_complete(runs[0]): + return False + with tempfile.TemporaryDirectory(dir=scratch(), prefix='complete-') as temporary: + release.verify_public(app, Path(temporary) / 'assets') + return True + finally: + release.ROOT = previous_root + + +def ensure_source(batch, base, run_id): + git('fetch', 'origin', 'main', '--tags') + tags = [line.split('\t')[1][10:] for line in git('ls-remote', '--tags', 'origin').splitlines() + if not line.endswith('^{}')] + # Fetch before reading any immutable ref, including a batch stopped after only its WWW tag. + trusted = trusted_nightlies(tags, batch) + sources = {source for _, source, context in trusted if context['nightly_batch'] == batch} + require(len(sources) <= 1, 'Nightly batch has conflicting immutable sources') + if sources: + source = sources.pop() + plan = recovered_plan(source) + require(plan.base == base and plan.run_id == str(run_id), 'Nightly batch differs from its original schedule source') + return source, completed_source(source) + # Finish the previous source before allocating anything new, even across schedule batches. + previous = '' + if trusted: + _, previous, context = trusted[0] + if not completed_source(previous): + return previous, False + require(subprocess.run(['git', 'merge-base', '--is-ancestor', context['ci_source_commit'], base], + cwd=release.ROOT, capture_output=True).returncode == 0, + 'unstarted Nightly source predates the previous main source') + require(batch[:8] >= versions_at(previous)['cli'].split('-nightly.')[1].split('.')[0], + 'unstarted Nightly batch predates the previous Nightly; refusing a downgrade') + if context['ci_source_commit'] == base: + return previous, True + plan = plan_versions(batch, base, tags, run_id, previous) + with checkout(base) as clean: + for path, content in generated(plan).items(): + (clean / path).write_bytes(content) + metadata(clean, plan) + git('add', '--', *PATHS, root=clean) + git('-c', 'user.name=github-actions[bot]', '-c', 'user.email=' + BOT, + 'commit', '-m', plan.message, root=clean) + source = git('rev-parse', 'HEAD', root=clean) + verify_tree(plan, source) + return source, False + + def run_daily(run_id): - require(os.environ.get('NIGHTLY_RELEASE_TOKEN'), 'NIGHTLY_RELEASE_TOKEN is missing') - require(os.environ.get('GH_TOKEN'), 'GITHUB_TOKEN must be supplied as GH_TOKEN for branch/tag/dispatch writes') + require(os.environ.get('GH_TOKEN'), 'GITHUB_TOKEN must be supplied as GH_TOKEN for tag/dispatch writes') require(os.environ.get('GITHUB_REPOSITORY') == REPO and os.environ.get('GITHUB_EVENT_NAME') == 'schedule' and os.environ.get('GITHUB_REF') == 'refs/heads/main' and str(run_id) == os.environ.get('GITHUB_RUN_ID'), 'daily coordinator only accepts its original main schedule run') original = api(f'actions/runs/{run_id}') + source = os.environ.get('GITHUB_SHA') require(original['event'] == 'schedule' and original['head_branch'] == 'main' and original['path'] == CI - and original['head_sha'] == os.environ.get('GITHUB_SHA') and original['repository']['full_name'] == REPO, + and original['head_sha'] == source and original['repository']['full_name'] == REPO + and original['head_repository']['full_name'] == REPO and str(original['id']) == str(run_id), 'untrusted daily schedule source') batch = schedule_batch(original['created_at']) - login = api('/user', pat=True)['login'] + ci = release.require_ci(dict(ci_source_commit=source, ci_run_id=str(run_id), nightly_batch=batch)) + source, complete = ensure_source(batch, source, run_id) + if complete: + print(json.dumps(dict(batch=batch, source_commit=source, skipped='no new main commits; previous Nightly including npm is complete'))) + return + context = release.validate_source(source, 'product') + ci = release.require_ci(context) + groups = RELEASES if context['ci_run_id'] else ('product',) deadline = time.monotonic() + 5 * 60 * 60 - plan, source, number = ensure_landed(batch, login) - ci = wait_main_ci(source, deadline) with checkout(source) as clean: - previous = release.ROOT + previous_root = release.ROOT release.ROOT = clean try: - require(versions_at(source) == plan.versions, 'landed daily versions changed') - environment = {key: value for key, value in os.environ.items() - if key not in ('GITHUB_EVENT_NAME', 'GITHUB_REF', 'NIGHTLY_RELEASE_TOKEN')} - environment['PYTHONDONTWRITEBYTECODE'] = '1' - for app in RELEASES: - tag = release.release_tag_for(app, plan.versions['cli' if app == 'product' else 'www']) - subprocess.run(['python3', str(clean / 'scripts/release.py'), 'plan', '--app', app, '--tag', tag, '--require-main'], - cwd=clean, env=environment, check=True) - releases = publish_daily(source, deadline) + versions = versions_at(source) + for app in groups: + tag = release_dispatch_tag(app) + require(release.parse_tag(tag)[2] == 'nightly', 'coordinator only publishes Nightly groups') + release.plan(app, tag=tag, require_main=True) + releases = publish_daily(source, deadline, groups) finally: - release.ROOT = previous - print(json.dumps(dict(batch=batch, day=plan.day, pr=number, source_commit=source, versions=plan.versions, - main_ci=ci['html_url'], releases=releases), indent=2)) + release.ROOT = previous_root + print(json.dumps(dict(batch=batch, source_commit=source, versions=versions, + schedule_ci=ci['html_url'], releases=releases), indent=2)) def self_test(): from unittest.mock import patch - with patch.dict(os.environ, {'GH_TOKEN': 'fixture-actions-token', 'NIGHTLY_RELEASE_TOKEN': 'fixture-pr-token'}): + with patch.dict(os.environ, {'GH_TOKEN': 'fixture-actions-token'}): with patch('subprocess.run', return_value=subprocess.CompletedProcess([], 0, '{}', '')) as transport: api('git/refs', 'POST', dict(ref='refs/tags/v0.0.1', sha='a' * 40)) assert transport.call_args.kwargs['env']['GH_TOKEN'] == 'fixture-actions-token' - api('pulls', 'POST', {}, pat=True) - assert transport.call_args.kwargs['env']['GH_TOKEN'] == 'fixture-pr-token' def rejected(call, message=None): try: @@ -648,7 +446,7 @@ def rejected(call, message=None): assert schedule_batch('2026-11-07T00:23:00+08:00') == '2026110616' assert schedule_batch('2026-11-07T00:00:00Z') == '2026110700' for batch in ('20261106', '2026110601', '2026110624', '2026130600'): - rejected(lambda: DailyPlan(batch, '', {}, 1, '3')) + rejected(lambda: DailyPlan(batch, '', {}, 1, '3', '42')) with tempfile.TemporaryDirectory(dir=scratch(), prefix='self-test-') as temporary: root = Path(temporary) / 'repo'; root.mkdir() @@ -668,204 +466,228 @@ def rejected(call, message=None): (root / 'Cargo.toml').write_text('[workspace]\nmembers = ["apps/core", "apps/cli"]\nresolver = "2"\n') (root / 'Cargo.lock').write_text('version = 3\n\n[[package]]\nname = "vibeface"\nversion = "0.0.1-nightly.20261105.2"\n\n[[package]]\nname = "vibeface-core"\nversion = "0.0.1-nightly.20261105.2"\n') git('init', '--initial-branch=main', root=root) + legacy_blob = git('hash-object', '-w', '--stdin', root=root, data=(root / 'Cargo.lock').read_bytes()) + legacy_tree = git('mktree', root=root, data=f'100644 blob {legacy_blob}\tCargo.lock\n'.encode()) + legacy = git('-c', 'user.name=github-actions[bot]', '-c', 'user.email=' + BOT, + 'commit-tree', legacy_tree, '-m', 'old release layout fixture', root=root) + legacy_batch = git('-c', 'user.name=github-actions[bot]', '-c', 'user.email=' + BOT, + 'commit-tree', legacy_tree, '-p', legacy, '-m', 'chore(release): nightly 2026110600', root=root) + git('update-ref', 'refs/heads/main', legacy_batch, root=root) + git('tag', 'v0.0.1-nightly.20261016.1', legacy, root=root) + git('tag', 'v0.0.1-nightly.20261016.2', legacy_batch, root=root) git('add', '.', root=root) git('-c', 'user.name=github-actions[bot]', '-c', 'user.email=' + BOT, 'commit', '-qm', 'fixture', root=root) - real_check_output = subprocess.check_output - - def metadata_cargo_stub(cmd, cwd=None, **kwargs): - if isinstance(cmd, (list, tuple)) and len(cmd) >= 2 and cmd[0] == 'cargo' and cmd[1] == 'metadata': - work = Path(cwd) - cli = re.search(r'version = "([^"]+)"', (work / 'apps/cli/Cargo.toml').read_text())[1] - core = re.search(r'version = "([^"]+)"', (work / 'apps/core/Cargo.toml').read_text())[1] - return json.dumps({'packages': [{'name': 'vibeface', 'version': cli}, - {'name': 'vibeface-core', 'version': core}]}).encode() - return real_check_output(cmd, cwd=cwd, **kwargs) - - with patch(f'{__name__}.ROOT', root), patch('release.ROOT', root), patch(f'{__name__}.time.sleep'), \ - patch('subprocess.check_output', side_effect=metadata_cargo_stub): + with patch('release.ROOT', root), patch(f'{__name__}.time.sleep'): assert RELEASES == ('product', 'www') and PRODUCT_APPS == ('app', 'core', 'cli', 'mobile', 'desktop') base = git('rev-parse', 'HEAD') - # Legacy component tags no longer claim a sequence; only real v/www-v tags and the current cohort do. - plan = plan_versions('2026110600', base, ['v0.0.1-nightly.20261106.7', 'core-v0.0.1-nightly.20261106.9', - 'mobile-v0.0.1-nightly.20261106.9', 'cli-v0.0.1-nightly.20261106.9']) - assert plan.versions['core'] == '0.0.1-nightly.20261106.8' - assert all(plan.versions[app] == plan.versions['cli'] for app in PRODUCT_APPS) + git('update-ref', 'refs/remotes/origin/main', base) + plan = plan_versions('2026110600', base, ['v0.0.1-nightly.20261106.7', 'core-v0.0.1-nightly.20261106.9'], '42') + assert plan.versions['cli'] == '0.0.1-nightly.20261106.8' assert plan.versions['www'] == '0.0.1-nightly.20261106.1' - assert plan.ios_build_number == '4' + assert plan.ios_build_number == '4' and plan.android_code == 250002 assert next_ios_build_number('3.1.9') == '3.1.10' assert next_ios_build_number('3.01') == '3.2' for invalid in ('', '0', '10000', '9999', '3.99', '3.1.99', '3.100', '3.1.100', '3.1.2.3'): rejected(lambda invalid=invalid: next_ios_build_number(invalid)) - assert plan.day == '20261106' and plan.branch == 'bot/daily-nightly/2026110600' - assert plan_versions('2026110500', base, []).versions['core'].endswith('.3') - assert plan_versions('2026110600', base, ['v0.0.1-nightly.20261106.9']).versions['cli'].endswith('.10') - assert plan_versions('2026110600', base, ['www-v0.0.1-nightly.20261106.4']).versions['www'].endswith('.5') - assert plan_versions('2026110600', base, ['www-v0.0.1-nightly.20261106.4']).versions['cli'].endswith('.1') contents = generated(plan) - mobile_generated = json.loads(contents['apps/mobile/app.json'])['expo'] - assert mobile_generated['ios']['buildNumber'] == '4' - assert mobile_generated['ios']['supportsTablet'] is True - assert json.loads(contents['apps/desktop/package.json'])['version'] == plan.versions['cli'] + assert json.loads(contents['apps/mobile/app.json'])['expo']['ios']['supportsTablet'] is True assert contents['Cargo.lock'].count(b'[[package]]') == 2 - def commit_files(changes): - git('checkout', '--detach', '-q', base) - for path, content in contents.items(): + def commit_files(changes, candidate=plan): + git('checkout', '--detach', '-q', candidate.base) + for path, content in generated(candidate).items(): (root / path).write_bytes(content) for path, content in changes.items(): (root / path).write_bytes(content) git('add', '--all') - git('-c', 'user.name=github-actions[bot]', '-c', 'user.email=' + BOT, 'commit', '-qm', 'daily fixture') + git('-c', 'user.name=github-actions[bot]', '-c', 'user.email=' + BOT, 'commit', '-qm', candidate.message) return git('rev-parse', 'HEAD') head = commit_files({}) - assert recovered_plan(plan.batch, head) == plan + assert recovered_plan(head) == plan verify_tree(plan, head) metadata(root, plan) + assert release.validate_source(head, 'product') == dict(ci_source_commit=base, ci_run_id='42', nightly_batch=plan.batch) + assert release.validate_source(head, 'www')['ci_source_commit'] == base for path, content in ( ('apps/app/package.json', contents['apps/app/package.json'].replace(b'"private": true', b'"private": false')), ('apps/desktop/package.json', contents['apps/desktop/package.json'].replace(b'"private": true', b'"private": false')), ('Cargo.lock', contents['Cargo.lock'] + b'\n[[package]]\nname = "unexpected"\nversion = "1.0.0"\n'), ('apps/core/Cargo.toml', contents['apps/core/Cargo.toml'] + b'\n[dependencies]\nserde = "1"\n'), - ('apps/cli/Cargo.toml', contents['apps/cli/Cargo.toml'].replace(b'0.0.1-nightly.20261106.8', b'0.0.1-nightly.20261106.9')), + ('apps/cli/Cargo.toml', contents['apps/cli/Cargo.toml'].replace(b'20261106.8', b'20261106.9')), ('apps/mobile/app.json', contents['apps/mobile/app.json'].replace(b'250002', b'1')), - ('apps/mobile/app.json', contents['apps/mobile/app.json'].replace(b'"buildNumber": "4"', b'"buildNumber": "3"')), ('apps/mobile/app.json', contents['apps/mobile/app.json'].replace(b'"buildNumber": "4"', b'"buildNumber": "5"')), ('apps/mobile/app.json', contents['apps/mobile/app.json'].replace(b'"supportsTablet": true', b'"supportsTablet": false')), ('extra.txt', b'not a version file\n'), ): bad = commit_files({path: content}) rejected(lambda: verify_tree(plan, bad), 'mechanical') - git('checkout', '--detach', '-q', base) - # An old per-app operation is refused outright instead of being reinterpreted as a new cohort. - stale = commit_files({'apps/mobile/app.json': contents['apps/mobile/app.json'].replace( - b'"releaseVersion": "0.0.1-nightly.20261106.8"', b'"releaseVersion": "0.0.1-nightly.20261106.9"')}) - rejected(lambda: verify_tree(plan, stale), 'mechanical') - rejected(lambda: verify_tree(recovered_plan(plan.batch, stale), stale), 'canonical version') - git('checkout', '--detach', '-q', base) - head = commit_files({}) + rejected(lambda: release.validate_source(bad, 'product')) + stale = commit_files({'apps/mobile/app.json': contents['apps/mobile/app.json'].replace(b'20261106.8', b'20261106.9')}) + rejected(lambda: release.validate_source(stale, 'product'), 'canonical version') git('checkout', '--detach', '-q', base) (root / 'business.txt').write_text('main advanced independently\n') git('add', 'business.txt') git('-c', 'user.name=github-actions[bot]', '-c', 'user.email=' + BOT, 'commit', '-qm', 'independent main change') advanced = git('rev-parse', 'HEAD') - landed = DailyPlan(plan.batch, advanced, plan.versions, plan.android_code, plan.ios_build_number) - merged = git('-c', 'user.name=github-actions[bot]', '-c', 'user.email=' + BOT, - 'commit-tree', expected_tree(landed), '-p', advanced, '-m', 'actual squash fixture') - assert merged != head - assert blob(merged, 'business.txt') == b'main advanced independently\n' git('update-ref', 'refs/remotes/origin/main', advanced) - pr = dict(number=17, user=dict(login='nightly-owner'), head=dict(repo=dict(full_name=REPO), ref=plan.branch, sha=head), - base=dict(repo=dict(full_name=REPO), ref='main'), commits=1, draft=False, - title='chore(release): nightly ' + plan.batch, state='open', merged=False, mergeable=True) - state = {'pr': None, 'merge_checks': 0}; writes = [] + assert recovered_plan(head).base == base and release.validate_source(head, 'product')['ci_source_commit'] == base + assert release.validate_source(advanced, 'product')['ci_source_commit'] == advanced + merge = git('-c', 'user.name=github-actions[bot]', '-c', 'user.email=' + BOT, + 'commit-tree', git('rev-parse', head + '^{tree}'), '-p', base, '-p', advanced, '-m', plan.message) + rejected(lambda: release.validate_source(merge, 'product'), 'ordinary commit') + foreign = git('-c', 'user.name=github-actions[bot]', '-c', 'user.email=' + BOT, + 'commit-tree', git('rev-parse', head + '^{tree}'), '-m', plan.message) + rejected(lambda: release.validate_source(foreign, 'product'), 'ordinary commit') + orphan_base = git('-c', 'user.name=github-actions[bot]', '-c', 'user.email=' + BOT, + 'commit-tree', git('rev-parse', base + '^{tree}'), '-m', 'foreign root') + orphan = DailyPlan(plan.batch, orphan_base, plan.versions, plan.android_code, plan.ios_build_number, '42') + orphan_head = commit_files({}, orphan) + rejected(lambda: release.validate_source(orphan_head, 'product'), 'parent must be on main') + stable = commit_files({'apps/www/package.json': contents['apps/www/package.json'].replace(b'0.0.1-nightly.20261106.1', b'0.0.1')}) + rejected(lambda: release.validate_source(stable, 'www'), 'only Nightly') + for value in (b'0.0.1', b'0.0.1-rc.1'): + ordinary_channel = commit_files({path: content.replace(plan.versions['cli'].encode(), value) + for path, content in contents.items()}) + rejected(lambda: release.validate_source(ordinary_channel, 'product'), 'only Nightly') + git('tag', 'v' + plan.versions['cli'], head) + # A new business parent still starts with main's old build values; the prior tag supplies the upper bounds. + next_plan = plan_versions('2026110604', advanced, ['v' + plan.versions['cli']], '43', head) + assert next_plan.android_code == 250003 and next_plan.ios_build_number == '5' + next_head = commit_files({}, next_plan) + assert release.validate_source(next_head, 'product')['ci_source_commit'] == advanced + assert recovered_plan(next_head).previous == head + legacy_tag = 'v0.0.1-nightly.20261016.1' + current_tag = 'v' + plan.versions['cli'] + git('merge-base', '--is-ancestor', legacy, 'origin/main') + for tags in ([legacy_tag, current_tag], [current_tag, legacy_tag]): + assert [source for _, source, _ in trusted_nightlies(tags, plan.batch)] == [head] + rejected(lambda: trusted_nightlies(['v0.0.1-nightly.20261016.2'], plan.batch), 'invalid immutable source') + with patch('release.validate_source', side_effect=subprocess.CalledProcessError(1, ['cargo', 'metadata'])): + try: + trusted_nightlies([current_tag], plan.batch) + except subprocess.CalledProcessError as error: + assert error.cmd == ['cargo', 'metadata'] + else: + raise AssertionError('candidate filtering swallowed a metadata environment failure') + rollback_plan = plan_versions('2026110608', base, ['v' + next_plan.versions['cli']], '44', next_head) + rollback = commit_files({}, rollback_plan) + for _ in range(2): + rejected(lambda: release.validate_source(rollback, 'product'), 'previous Nightly main source must be an ancestor') + nested_plan = plan_versions('2026110612', advanced, ['v' + rollback_plan.versions['cli']], '45', rollback) + nested = commit_files({}, nested_plan) + rejected(lambda: release.validate_source(nested, 'product'), 'previous Nightly main source must be an ancestor') + future_main_plan = plan_versions('2026110608', base, [current_tag], '44', advanced) + future_main = commit_files({}, future_main_plan) + rejected(lambda: release.validate_source(future_main, 'product'), 'previous Nightly main source must be an ancestor') + # Numeric CFBundleVersion ordering must select 3.1.10 rather than 3.1.9. + with patch('nightly_source.blob', side_effect=lambda commit, path: json.dumps({'expo': { + 'android': {'versionCode': 9 if commit == base else 10}, + 'ios': {'buildNumber': '3.1.9' if commit == base else '3.1.10'}}}).encode()): + assert next_build_numbers(base, head) == (11, '3.1.11') + with patch('nightly_source.blob', side_effect=lambda commit, path: json.dumps({'expo': { + 'android': {'versionCode': 10}, 'ios': {'buildNumber': '4' if commit == base else '3.99'}}}).encode()): + assert next_build_numbers(base, head) == (11, '5') + rejected(lambda: plan_versions('2026110400', base, [], '42'), 'current cli version') + + # Recovery never allocates a new version or switches to the newly advanced main. + remote_tags = f'{head}\trefs/tags/v{plan.versions["cli"]}' real_git = git - def fixture_git(*args, **kwargs): - return '' if args[0] == 'fetch' else real_git(*args, **kwargs) - - def pr_api(path, method='GET', data=None, pat=False, missing=False): - if method == 'POST': - assert path == 'pulls' and pat - assert data['head'] == plan.branch and data['base'] == 'main' and data['title'] == pr['title'] - writes.append((path, method, data)); state['pr'] = dict(pr) - raise ValueError('PR created but response lost') - if method == 'PUT': - assert path == 'pulls/17/merge' and pat and data == dict(sha=head, merge_method='squash') - assert state['merge_checks'] >= 2 - writes.append((path, method, data)) - state['pr'] = dict(pr, merged=True, state='closed', merge_commit_sha=merged) - real_git('update-ref', 'refs/remotes/origin/main', merged) - raise ValueError('merge completed but response lost') - assert path == 'pulls/17' - if not state['pr'].get('merged'): - state['merge_checks'] += 1 - if state['merge_checks'] == 1: - return dict(state['pr'], mergeable=None) - return state['pr'] - - with patch(f'{__name__}.api', side_effect=pr_api), patch(f'{__name__}.daily_pr', side_effect=lambda branch: state['pr']), \ - patch(f'{__name__}.git', side_effect=fixture_git): - created = create_pr(plan, head, 'nightly-owner') - assert merge_pr(created, plan, head, 'nightly-owner') == merged - assert len(writes) == 2 - assert merge_pr(state['pr'], plan, head, 'nightly-owner') == merged and len(writes) == 2 - with patch(f'{__name__}.branch_sha', return_value=head), patch(f'{__name__}.plan_versions', side_effect=AssertionError('same-batch reallocation')): - assert ensure_landed(plan.batch, 'nightly-owner') == (plan, merged, 17) - rejected(lambda: validate_pr(dict(pr, user=dict(login='outsider')), plan, head, 'nightly-owner'), 'identity') - next_batch = plan_versions('2026110604', merged, []) - assert next_batch.day == plan.day and next_batch.branch != plan.branch - assert next_batch.versions['cli'] == '0.0.1-nightly.20261106.9' - assert next_batch.versions['www'] == '0.0.1-nightly.20261106.2' - assert next_batch.android_code == plan.android_code + 1 - assert next_batch.ios_build_number == '5' - new_remote = {} - def next_batch_git(*args, **kwargs): - if args[0] in ('fetch', 'ls-remote'): - return '' - if args[0] == 'push': - pushed, ref = args[2].split(':') - assert ref == 'refs/heads/' + next_batch.branch - new_remote['head'] = pushed + if args[0] == 'fetch': return '' + if args[0] == 'ls-remote': + return remote_tags return real_git(*args, **kwargs) - with patch(f'{__name__}.git', side_effect=next_batch_git), \ - patch(f'{__name__}.daily_pr', side_effect=lambda branch: state['pr'] if branch == plan.branch else None) as lookup, \ - patch(f'{__name__}.branch_sha', side_effect=lambda branch: new_remote.get('head')), \ - patch(f'{__name__}.create_pr', return_value=dict(number=18)) as new_pr, \ - patch(f'{__name__}.merge_pr', side_effect=lambda pr, plan, head, login: head): - allocated, next_head, number = ensure_landed(next_batch.batch, 'nightly-owner') - assert allocated == next_batch and number == 18 - lookup.assert_called_once_with(next_batch.branch) - new_pr.assert_called_once_with(next_batch, next_head, 'nightly-owner') - assert recovered_plan(next_batch.batch, next_head) == next_batch - verify_tree(next_batch, next_head) - next_day = plan_versions('2026110700', next_head, []) - pv = next_day.versions['cli'] - assert pv == '0.0.1-nightly.20261107.1' and all(next_day.versions[a] == pv for a in PRODUCT_APPS) - assert next_day.versions['www'] == '0.0.1-nightly.20261107.1' - assert next_day.android_code == next_batch.android_code + 1 - assert next_day.ios_build_number == '6' - future_main = git('-c', 'user.name=github-actions[bot]', '-c', 'user.email=' + BOT, - 'commit-tree', expected_tree(next_day), '-p', next_head, '-m', 'next-day main fixture') - git('update-ref', 'refs/remotes/origin/main', future_main) - with patch(f'{__name__}.git', side_effect=next_batch_git), patch(f'{__name__}.daily_pr', return_value=None), \ - patch(f'{__name__}.branch_sha', return_value=None), patch(f'{__name__}.api') as no_write, \ - patch(f'{__name__}.checkout', side_effect=AssertionError('stale batch generated a branch')) as no_checkout: - rejected(lambda: ensure_landed('2026110620', 'nightly-owner'), 'current cli version') - no_checkout.assert_not_called() - no_write.assert_not_called() - newer_www = versions_at(next_head) - newer_www['www'] = next_day.versions['www'] - with patch(f'{__name__}.versions_at', return_value=newer_www): - rejected(lambda: plan_versions('2026110620', next_head, []), 'current www version') - with patch(f'{__name__}.git', side_effect=fixture_git), patch(f'{__name__}.daily_pr', return_value=state['pr']), \ - patch(f'{__name__}.branch_sha', return_value=head), \ + with patch(f'{__name__}.git', side_effect=fixture_git), patch(f'{__name__}.completed_source', return_value=False), \ + patch(f'{__name__}.plan_versions', side_effect=AssertionError('original source reallocated')): + assert ensure_source(plan.batch, base, 42) == (head, False) + assert ensure_source('2026110604', base, 43) == (head, False) + rejected(lambda: ensure_source(plan.batch, advanced, 42), 'original schedule source') + with patch(f'{__name__}.git', side_effect=fixture_git), patch(f'{__name__}.completed_source', return_value=True), \ + patch(f'{__name__}.plan_versions', side_effect=AssertionError('unchanged main reallocated')): + assert ensure_source('2026110604', base, 43) == (head, True) + git('tag', 'www-v' + plan.versions['www'], head) + saved_tags = remote_tags + remote_tags = f'{head}\trefs/tags/www-v{plan.versions["www"]}' + with patch(f'{__name__}.git', side_effect=fixture_git), patch(f'{__name__}.completed_source', return_value=False), \ + patch(f'{__name__}.plan_versions', side_effect=AssertionError('WWW-only source reallocated')): + assert ensure_source('2026110604', advanced, 43) == (head, False) + remote_tags = saved_tags + git('tag', 'v' + next_plan.versions['cli'], next_head) + remote_tags += f'\n{next_head}\trefs/tags/v{next_plan.versions["cli"]}' + with patch(f'{__name__}.git', side_effect=fixture_git), patch(f'{__name__}.completed_source', return_value=True), \ + patch(f'{__name__}.plan_versions', side_effect=AssertionError('stale batch allocated a new version')): + rejected(lambda: ensure_source('2026110504', advanced, 44), 'predates the previous Nightly') + rejected(lambda: ensure_source('2026110608', base, 44), 'predates the previous main source') + with patch(f'{__name__}.git', side_effect=fixture_git), patch(f'{__name__}.completed_source', return_value=False), \ patch(f'{__name__}.plan_versions', side_effect=AssertionError('old batch was reallocated')): - assert ensure_landed(plan.batch, 'nightly-owner') == (plan, merged, 17) - git('update-ref', 'refs/remotes/origin/main', merged) - - good = dict(id=1, run_attempt=1, head_sha=merged, head_branch='main', event='push', path=CI, + assert ensure_source(plan.batch, base, 42) == (head, False) + remote_tags = saved_tags + # The same shared source rule runs at the real plan and publisher boundaries. + git('checkout', '--detach', '-q', head) + assert release.plan('product', tag='v' + plan.versions['cli'], require_main=True, publishing=True)['ci_source_commit'] == base + release.require_product_publishing() + bad = commit_files({'apps/app/package.json': contents['apps/app/package.json'].replace(b'"private": true', b'"private": false')}) + git('update-ref', 'refs/tags/v' + plan.versions['cli'], bad) + remote_tags = f'{bad}\trefs/tags/v{plan.versions["cli"]}' + with patch(f'{__name__}.git', side_effect=fixture_git), \ + patch(f'{__name__}.plan_versions', side_effect=AssertionError('conflicting tag allocated a new version')): + rejected(lambda: ensure_source(plan.batch, base, 42), 'invalid immutable source') + remote_tags = saved_tags + rejected(lambda: release.plan('product', require_main=True), 'mechanical') + rejected(release.require_product_publishing, 'mechanical') + with patch('release.verify_release', return_value=release.identity('product')), patch('release.release_view', side_effect=AssertionError('untrusted publication wrote remotely')): + rejected(lambda: release.publish('product', root), 'mechanical') + git('update-ref', 'refs/tags/v' + plan.versions['cli'], head) + remote_tags = '' + with patch(f'{__name__}.git', side_effect=fixture_git): + pinned_source, complete = ensure_source('2026110600', base, 42) + assert not complete and recovered_plan(pinned_source).base == base + assert git('ls-tree', pinned_source, '--', 'business.txt') == '' + remote_tags = saved_tags + good = dict(id=1, run_attempt=1, head_sha=head, head_branch='main', event='push', path=CI, repository=dict(full_name=REPO), head_repository=dict(full_name=REPO), status='completed', conclusion='success', html_url='https://example.invalid/main-ci') - with patch(f'{__name__}.pages', return_value=[good]): - assert wait_main_ci(merged, time.monotonic() + 1)['id'] == 1 - for change in ({'head_sha': 'b' * 40}, {'event': 'workflow_dispatch'}, {'head_branch': 'feature'}, {'path': RELEASE}, - {'status': 'in_progress', 'conclusion': None}, {'conclusion': 'failure'}): - with patch(f'{__name__}.pages', return_value=[dict(good, **change)]), \ - patch(f'{__name__}.time.monotonic', side_effect=[0, 1]), patch(f'{__name__}.api') as no_write: - rejected(lambda: wait_main_ci(merged, 0.5)) - no_write.assert_not_called() - cancelled = {'run': dict(good, conclusion='cancelled')}; writes = [] - def cancelled_api(path, method='GET', **kwargs): - if method == 'POST': - writes.append(path); cancelled['run'] = dict(good, run_attempt=2) - raise ValueError('CI recovery response lost') - return cancelled['run'] - with patch(f'{__name__}.pages', side_effect=lambda *args: [cancelled['run']]), \ - patch(f'{__name__}.api', side_effect=cancelled_api): - assert wait_main_ci(merged, time.monotonic() + 1)['run_attempt'] == 2 - assert writes == ['actions/runs/1/rerun'] + schedule = dict(good, id=42, event='schedule', head_sha=base, created_at='2026-11-06T00:23:00Z', + status='in_progress', conclusion=None) + ci_context = dict(ci_source_commit=base, ci_run_id='42', nightly_batch=plan.batch) + jobs = [dict(name='ci-gate', status='completed', conclusion='success')] + def ci_api(*args, **kwargs): + return json.dumps([dict(jobs=jobs)] if args[-1].endswith('per_page=100') else schedule) + with patch('release.run', side_effect=ci_api): + assert release.require_ci(ci_context)['id'] == 42 + for change in ({'id': 43}, {'head_sha': advanced}, {'event': 'workflow_dispatch'}, {'head_branch': 'feature'}, + {'path': RELEASE}, {'repository': dict(full_name='other/repo')}, + {'head_repository': dict(full_name='other/repo')}, {'created_at': '2026-11-06T04:23:00Z'}): + with patch.dict(schedule, change): + rejected(lambda: release.require_ci(ci_context)) + for job_list in ([], [dict(jobs[0], conclusion='failure')], [dict(jobs[0], status='in_progress')], jobs * 2): + with patch('release.run', side_effect=lambda *args: json.dumps([dict(jobs=job_list)] if args[-1].endswith('per_page=100') else schedule)): + rejected(lambda: release.require_ci(ci_context), 'ci-gate') + with patch('release.run', return_value=json.dumps([dict(workflow_runs=[dict(good, head_sha=base)])])): + assert release.require_ci(dict(ci_source_commit=base, ci_run_id='', nightly_batch=''))['id'] == 1 + for change in ({'event': 'schedule'}, {'head_sha': advanced}, {'path': RELEASE}, + {'status': 'in_progress'}, {'conclusion': 'failure'}): + with patch('release.run', return_value=json.dumps([dict(workflow_runs=[dict(good, **({'head_sha': base} | change))])])): + rejected(lambda: release.require_ci(dict(ci_source_commit=base, ci_run_id='', nightly_batch=''))) + with patch(f'{__name__}.tag_sha', return_value=head), patch(f'{__name__}.release_state', return_value=dict(draft=False)), \ + patch(f'{__name__}.matching_runs', return_value=[good]), patch('release.verify_public') as verify_public: + for state in ('success', 'failure', 'skipped'): + with patch(f'{__name__}.pages', return_value=[dict(name='npm / npm', status='completed', conclusion=state)]): + assert completed_source(head) == (state == 'success') + assert verify_public.call_count == 2 + with patch(f'{__name__}.pages', return_value=[]): + assert not completed_source(head) + with patch(f'{__name__}.tag_sha', return_value=None): + assert not completed_source(head) + with patch(f'{__name__}.tag_sha', return_value=base), patch(f'{__name__}.release_state', return_value=dict(draft=False)), \ + patch(f'{__name__}.matching_runs', return_value=[good]), patch(f'{__name__}.npm_complete', return_value=True), \ + patch('release.verify_public') as verified: + assert completed_source(base) + assert verified.call_count == 1 and verified.call_args.args[0] == 'product' + merged = head + git('checkout', '--detach', '-q', advanced) assert release_dispatch_tag('product') == 'v0.0.1-nightly.20261105.2' assert release_dispatch_tag('www') == 'www-v0.0.1' @@ -881,9 +703,7 @@ def cancelled_api(path, method='GET', **kwargs): def release_api(path, method='GET', data=None, **kwargs): if method != 'GET': writes.append((path, method, data)) - if path == 'git/refs': - remote['tag'] = data['sha'] - elif path.endswith('/dispatches'): + if path.endswith('/dispatches'): # A real daily release must request publication, not a build-only dry run. assert data == dict(ref=tag, inputs=dict(app='product', tag=tag, publish='true')) remote['runs'] = [dict(good, head_branch=tag, event='workflow_dispatch', path=RELEASE)] @@ -895,7 +715,12 @@ def release_api(path, method='GET', data=None, **kwargs): assert path.startswith('releases/tags/') return dict(draft=False) if remote['public'] else None - with patch(f'{__name__}.api', side_effect=release_api), \ + def tag_push(*args, **kwargs): + assert args == ('push', 'origin', merged + ':refs/tags/' + tag) + writes.append(('tag-push', 'git', args)); remote['tag'] = merged + raise subprocess.CalledProcessError(1, ['git', 'push']) + + with patch(f'{__name__}.api', side_effect=release_api), patch(f'{__name__}.git', side_effect=tag_push), \ patch(f'{__name__}.matching_runs', side_effect=lambda *args: remote['runs']), \ patch(f'{__name__}.release_dispatch_tag', return_value=tag): ensure_tag(tag, merged); ensure_tag(tag, merged) @@ -969,7 +794,7 @@ def recovery_pages(path, key): rejected(lambda: recover_publisher('mobile', failing, merged), 'product and www') assert len(writes) == 1 + len(product_publishers) - with patch(f'{__name__}.tag_sha', return_value=merged), patch(f'{__name__}.release_state', return_value=dict(draft=False)), \ + with patch(f'{__name__}.npm_complete', return_value=True), patch(f'{__name__}.tag_sha', return_value=merged), patch(f'{__name__}.release_state', return_value=dict(draft=False)), \ patch('release.verify_public') as verified, patch(f'{__name__}.recover_publisher') as no_retry, \ patch(f'{__name__}.release_dispatch_tag', return_value='v' + plan.versions['cli']): with patch(f'{__name__}.matching_runs', return_value=[]): @@ -986,13 +811,13 @@ def recovery_pages(path, key): assert finish_release('product', merged, time.monotonic() + 1)[0]['conclusion'] == 'success' verified.assert_called_once() - with patch(f'{__name__}.tag_sha', return_value=merged), patch(f'{__name__}.release_state', return_value=dict(draft=False)), \ + with patch(f'{__name__}.npm_complete', return_value=True), patch(f'{__name__}.tag_sha', return_value=merged), patch(f'{__name__}.release_state', return_value=dict(draft=False)), \ patch('release.verify_public', side_effect=ValueError('canonical completion manifest is missing')), \ patch(f'{__name__}.release_dispatch_tag', return_value='v' + plan.versions['cli']), \ patch(f'{__name__}.matching_runs', return_value=[good]): rejected(lambda: finish_release('product', merged, time.monotonic() + 1), 'completion manifest is missing') - with patch(f'{__name__}.tag_sha', return_value=merged), patch(f'{__name__}.release_state', return_value=None), \ + with patch(f'{__name__}.npm_complete', return_value=True), patch(f'{__name__}.tag_sha', return_value=merged), patch(f'{__name__}.release_state', return_value=None), \ patch('release.verify_public') as verified, patch(f'{__name__}.recover_publisher') as no_retry, \ patch(f'{__name__}.release_dispatch_tag', return_value='v' + plan.versions['cli']), \ patch(f'{__name__}.matching_runs', return_value=[failing]), patch(f'{__name__}.time.sleep'): @@ -1034,24 +859,18 @@ def drain(*args): assert not any(t.startswith('mobile-v') or t.startswith('app-v') or t.startswith('core-v') for t in groups) environment = dict(GITHUB_REPOSITORY=REPO, GITHUB_EVENT_NAME='schedule', GITHUB_REF='refs/heads/main', - GITHUB_RUN_ID='42', GITHUB_SHA=merged, NIGHTLY_RELEASE_TOKEN='fixture-only', GH_TOKEN='fixture-only') - original = dict(good, event='schedule', created_at='2026-11-06T00:23:00Z') + GITHUB_RUN_ID='42', GITHUB_SHA=base, GH_TOKEN='fixture-only') observed_batches = [] - def observe_batch(batch, login): - observed_batches.append(batch) + def observe_batch(batch, pinned, run_id): + observed_batches.append((batch, pinned, run_id)) raise ValueError('batch recovered from original run') - for now, attempt in ((datetime.datetime(2026, 11, 6, 23, tzinfo=datetime.timezone.utc), 1), - (datetime.datetime(2026, 11, 7, 12, tzinfo=datetime.timezone.utc), 2)): - original['run_attempt'] = attempt - with patch.dict(os.environ, environment), \ - patch(f'{__name__}.api', side_effect=lambda path, **kwargs: dict(login='owner') if path == '/user' else original), \ - patch(f'{__name__}.ensure_landed', side_effect=observe_batch), \ - patch(f'{__name__}.datetime.datetime', wraps=datetime.datetime) as clock: - clock.now.return_value = now + for attempt in (1, 2): + schedule['run_attempt'] = attempt + with patch.dict(os.environ, environment), patch(f'{__name__}.api', return_value=schedule), \ + patch('release.require_ci', return_value=schedule), patch(f'{__name__}.ensure_source', side_effect=observe_batch): rejected(lambda: run_daily(42), 'batch recovered') - clock.now.assert_not_called() - assert observed_batches == ['2026110600', '2026110600'] - for changes in ({'NIGHTLY_RELEASE_TOKEN': ''}, {'GITHUB_EVENT_NAME': 'workflow_dispatch'}): + assert observed_batches == [('2026110600', base, 42)] * 2 + for changes in ({'GH_TOKEN': ''}, {'GITHUB_EVENT_NAME': 'workflow_dispatch'}): with patch.dict(os.environ, dict(environment, **changes)), patch(f'{__name__}.api') as no_write: rejected(lambda: run_daily(42)) no_write.assert_not_called() @@ -1062,6 +881,8 @@ def observe_batch(batch, login): npm = (ROOT / '.github/workflows/npm-publish.yml').read_text() assert 'id-token: write' in npm and 'environment: npm-release' in npm and 'default: false' in npm assert 'NPM_TOKEN' not in npm and 'secrets.' not in npm + nightly_job = workflow.split(' daily-nightly:', 1)[1] + assert 'NIGHTLY_RELEASE_TOKEN' not in nightly_job and 'pull-requests:' not in nightly_job print('four-hour Nightly real-Git, batch identity, reconciliation and original-artifact self-tests passed') diff --git a/scripts/deploy-activate.py b/scripts/deploy-activate.py index cc1272706..e768376e7 100644 --- a/scripts/deploy-activate.py +++ b/scripts/deploy-activate.py @@ -628,6 +628,36 @@ def self_test(): with patch(__name__ + '.docker', return_value=result): with TestCase().assertRaises(ValueError, msg='accepted an ambiguous imported image'): import_image(str(archive), digest) + settings = Path(temporary) / 'config.json' + settings.write_text(json.dumps({'project': 'official-app', 'role': 'app', + 'deploy_dir': '/srv/service-control/app', 'listen': '127.0.0.1:8088', + 'nginx_config': '/etc/nginx/conf.d/app.conf', + 'server_name': 'app.example.invalid'})) + arguments = ['deploy-activate.py', '--config', str(settings), '--tag', 'v0.0.1', + '--version', '0.0.1', '--source-sha', 'a' * 40, '--archive', str(archive)] + for pin, checksum, loaded, accepted in ( + (imported, digest, 'Loaded image ID: ' + imported, True), + ('sha256:' + 'b' * 64, digest, 'Loaded image ID: ' + imported, False), + (imported, '0' * 64, 'Loaded image ID: ' + imported, False), + (imported, digest, 'Loaded image: moving:tag', False), + (imported, digest, 'Loaded image ID: ' + imported + '\nLoaded image ID: ' + imported, False)): + with patch.object(sys, 'argv', arguments + ['--image', pin, '--archive-sha256', checksum]), \ + patch(__name__ + '.docker', return_value=loaded) as load, \ + patch(__name__ + '.activate', return_value={}) as activation, \ + patch('builtins.print'): + try: + main() + except ValueError: + assert not accepted, 'matching pinned archive was refused' + else: + assert accepted, 'CLI activated an archive that differs from its pins' + if accepted: + activation.assert_called_once_with(json.loads(settings.read_text()), pin, + 'v0.0.1', 'a' * 40, '0.0.1') + else: + activation.assert_not_called() + if checksum != digest: + load.assert_not_called() config = {'project': 'official-node', 'role': 'node', 'deploy_dir': '/srv/service-control/node', 'listen': '0.0.0.0:4792', 'data_dir': '/var/lib/official-node', 'tls_dir': '/etc/official-node/tls', 'device_id': 'a' * 64, @@ -828,7 +858,8 @@ def main(): require(args.tag == ('www-v' if config['role'] == 'www' else 'v') + args.version, 'tag and version differ') require(bool(args.archive) == bool(args.archive_sha256), 'archive and its digest must be supplied together') if args.archive: - args.image = import_image(args.archive, args.archive_sha256) + require(import_image(args.archive, args.archive_sha256) == args.image, + 'imported image differs from the pinned image ID') print(json.dumps(activate(config, args.image, args.tag, args.source_sha, args.version))) diff --git a/scripts/deploy-run.py b/scripts/deploy-run.py index 984c45a54..6e3c55afb 100644 --- a/scripts/deploy-run.py +++ b/scripts/deploy-run.py @@ -8,6 +8,7 @@ import re import shlex import shutil +import sqlite3 import subprocess import sys import tempfile @@ -258,12 +259,16 @@ def activate(service, output): def rehearse(output): - """Activate the prepared Core on this runner as a fixture host: fresh install, handover, then a no-op.""" - release.require(os.geteuid() == 0, 'the activation rehearsal runs as root on a disposable runner') + """Start the previous published Core, stage the candidate config, then upgrade and repeat.""" + release.require(os.geteuid() == 0 and sys.platform == 'linux', + 'the activation rehearsal requires root on a disposable Linux runner') output = Path(output).resolve() info = json.loads((output / 'deployment.json').read_text()) - identity = info['images']['core'] - host = Path(tempfile.mkdtemp(prefix='vibeface-rehearsal-')) + previous = deploy.previous(info['tag']) + if previous: + previous = build('node', previous['tag'], output / 'previous') + seed = previous or info + host = Path(tempfile.mkdtemp(dir=output, prefix='rehearsal-')) data, tls = host / 'state', host / 'tls' public_url = 'https://127.0.0.1:4792' options = {'data_dir': '/data', 'listen': '0.0.0.0:4792', 'public_url': public_url, 'tls': True, @@ -272,51 +277,86 @@ def rehearse(output): 'relay_max_clients': 128, 'relay_max_clients_per_ip': 8, 'forward_any': True, 'forward_sessions': 32, 'forward_queue': 192, 'forward_rate': 262144, 'forward_per_ip': 8, 'mesh_network': 'deploy-fixture'} - # The host initialization in deploy/README.md, step by step. - data.mkdir() - result = json.loads(subprocess.check_output( - ['docker', 'run', '--rm', '-i', '--network', 'none', '-v', f'{data}:/data', '--entrypoint', - '/opt/vibeface/vibeface-core', identity, '--control'], - input=json.dumps({'command': 'init', 'options': options}), text=True)) - release.require(result.get('ok') is True, 'fixture Node initialization failed') - (data / 'node.lock').touch() - checked(['chown', '-R', '1001:1001', str(data)]) - os.chmod(data, 0o700) - checked(['install', '-d', '-o', 'root', '-g', '1001', '-m', '0750', str(tls)]) - checked(['install', '-o', 'root', '-g', 'root', '-m', '0644', str(data / 'tls-cert.pem'), str(tls / 'tls-cert.pem')]) - checked(['install', '-o', 'root', '-g', '1001', '-m', '0640', str(data / 'tls-key.pem'), str(tls / 'tls-key.pem')]) - device = json.loads(subprocess.check_output( - ['docker', 'run', '--rm', '-i', '--network', 'none', '-v', f'{data}:/data', '--user', '1001:1001', - '--entrypoint', '/opt/vibeface/vibeface-core', identity, '--control'], - input='{"command":"inspect","data_dir":"/data"}', text=True))['result']['device_id'] - # The image pins the official issuer, so the rehearsal pins this fixture Node instead. - anchor = host / 'public-bootstrap.json' - anchor.write_text(json.dumps({'issuer': device, 'udp_host': '127.0.0.1', 'udp_port': 4792, 'gateway_url': public_url})) - container = subprocess.check_output(['docker', 'create', identity], text=True).strip() - try: - checked(['docker', 'cp', str(anchor), container + ':/opt/vibeface/public-bootstrap.json']) - fixture = subprocess.check_output(['docker', 'commit', container], text=True).strip() - finally: - checked(['docker', 'rm', container], stdout=subprocess.DEVNULL) - config = {'project': 'vibeface-rehearsal', 'role': 'node', 'deploy_dir': str(host / 'deploy'), - 'listen': '0.0.0.0:4792', 'data_dir': str(data), 'tls_dir': str(tls), 'device_id': device, - 'uid': 1001, 'gid': 1001} - receipts = [] + project = 'vibeface-rehearsal-' + uuid.uuid4().hex[:24] + fixtures, receipts = {}, [] + + def initialize(identity, directory): + directory.mkdir() + result = json.loads(subprocess.check_output( + ['docker', 'run', '--rm', '-i', '--network', 'none', '-v', f'{directory}:/data', '--entrypoint', + '/opt/vibeface/vibeface-core', identity, '--control'], + input=json.dumps({'command': 'init', 'options': options}), text=True)) + release.require(result.get('ok') is True, 'fixture Node initialization failed') + return result['result']['device_id'] + + def state(): + connection = sqlite3.connect((data / 'node.sqlite3').as_uri() + '?mode=ro', uri=True) + try: + bootstrap = connection.execute("SELECT value FROM identity_meta WHERE key='bootstrap_capability_hash'").fetchone() + release.require(bootstrap is not None, 'fixture bootstrap database record is missing') + finally: + connection.close() + return ([release.digest(data / name) for name in ('identity.json', 'bootstrap.json', 'data-epoch.json')], bootstrap) + + def activate_fixture(released, expected): + identity = released['images']['core'] + if identity not in fixtures: + container = subprocess.check_output(['docker', 'create', identity], text=True).strip() + try: + checked(['docker', 'cp', str(anchor), container + ':/opt/vibeface/public-bootstrap.json']) + fixtures[identity] = subprocess.check_output(['docker', 'commit', container], text=True).strip() + finally: + checked(['docker', 'rm', container], stdout=subprocess.DEVNULL) + fixture = fixtures[identity] + result = subprocess.run([sys.executable, '-B', str(ROOT / 'scripts/deploy-activate.py'), '--config', '-', + '--image', fixture, '--tag', released['tag'], '--source-sha', released['source_sha'], + '--version', released['version']], + input=json.dumps(config), capture_output=True, text=True) + release.require(result.returncode == 0, 'rehearsal ' + (result.stderr.strip() or 'activation failed')) + receipt = json.loads(result.stdout) + release.require(receipt['status'] == expected and receipt['image'] == fixture + and receipt['source_sha'] == released['source_sha'] and receipt['version'] == released['version'], + 'rehearsal receipt differs') + release.require(state() == preserved, 'rehearsal changed the existing Node identity or data') + receipts.append(receipt['status']) + try: - for settings, expected in ((config, 'activated'), (dict(config, public_url=public_url), 'activated'), - (dict(config, public_url=public_url), 'already_active')): - result = subprocess.run([sys.executable, '-B', str(ROOT / 'scripts/deploy-activate.py'), '--config', '-', - '--image', fixture, '--tag', info['tag'], '--source-sha', info['source_sha'], - '--version', info['version']], - input=json.dumps(settings), capture_output=True, text=True) - release.require(result.returncode == 0, 'rehearsal ' + (result.stderr.strip() or 'activation failed')) - receipt = json.loads(result.stdout) - release.require(receipt['status'] == expected and receipt['image'] == fixture, 'rehearsal receipt differs') - receipts.append(receipt['status']) + # Initialize with the old binary; only its fixture anchor differs from published bytes. + device = initialize(seed['images']['core'], data) + preserved = state() + (data / 'node.lock').touch() + checked(['chown', '-R', '1001:1001', str(data)]) + os.chmod(data, 0o700) + checked(['install', '-d', '-o', 'root', '-g', '1001', '-m', '0750', str(tls)]) + checked(['install', '-o', 'root', '-g', 'root', '-m', '0644', str(data / 'tls-cert.pem'), str(tls / 'tls-cert.pem')]) + checked(['install', '-o', 'root', '-g', '1001', '-m', '0640', str(data / 'tls-key.pem'), str(tls / 'tls-key.pem')]) + anchor = host / 'public-bootstrap.json' + anchor.write_text(json.dumps({'issuer': device, 'udp_host': '127.0.0.1', 'udp_port': 4792, + 'gateway_url': public_url})) + config = {'project': project, 'role': 'node', 'deploy_dir': str(host / 'deploy'), + 'listen': '0.0.0.0:4792', 'data_dir': str(data), 'tls_dir': str(tls), 'device_id': device, + 'uid': 1001, 'gid': 1001, 'public_url': public_url} + activate_fixture(seed, 'activated') + if previous: + # Reproduce a host whose running old Core now sees the candidate's complete config format. + candidate = host / 'candidate-state' + initialize(info['images']['core'], candidate) + replacement = data / '.config.rehearsal.json' + shutil.copy2(candidate / 'config.json', replacement) + os.chown(replacement, 1001, 1001) + os.replace(replacement, data / 'config.json') + activate_fixture(info, 'activated') + activate_fixture(info, 'already_active') finally: - subprocess.run(['docker', 'compose', '--project-name', 'vibeface-rehearsal', 'down'], - stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) - return {'service': 'node', 'tag': info['tag'], 'rehearsal': receipts} + checked(['docker', 'compose', '--project-name', project, '-f', str(ROOT / 'deploy/compose.yaml'), 'down'], + env=dict(os.environ, VIBEFACE_IMAGE=seed['images']['core']), + stdout=subprocess.DEVNULL) + if fixtures: + subprocess.run(['docker', 'image', 'rm', *fixtures.values()], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + return {'service': 'node', 'tag': info['tag'], 'previous_tag': previous['tag'] if previous else None, + 'mode': 'upgrade' if previous else 'first_release', 'rehearsal': receipts, + 'identity_and_data_preserved': True} def self_test(): @@ -327,7 +367,106 @@ def self_test(): ('DEPLOY_PORT', '0'), ('DEPLOY_PORT', '65536'), ('DEPLOY_SSH_KEY', '')): with TestCase().assertRaises(ValueError, msg='accepted invalid transport'): transport(dict(base, **{key: value})) - print('deployment transport self-test passed') + import runpy + from unittest.mock import patch + validate_config = runpy.run_path(str(ROOT / 'scripts/deploy-activate.py'))['validate_config'] + for has_previous, failure in ((True, None), (False, None), (True, 'activation'), (True, 'data')): + with tempfile.TemporaryDirectory() as temporary: + output = Path(temporary).resolve() + host = output / 'host'; host.mkdir() + current = {'tag': 'v0.0.1-nightly.20261108.1', 'version': '0.0.1-nightly.20261108.1', + 'source_sha': 'a' * 40, 'images': {'core': 'sha256:' + 'a' * 64}} + old = {'tag': 'v0.0.1-nightly.20261107.1', 'version': '0.0.1-nightly.20261107.1', + 'source_sha': 'b' * 40, 'images': {'core': 'sha256:' + 'b' * 64}} + (output / 'deployment.json').write_text(json.dumps(current)) + initialized, activated, cleaned = [], [], [] + images = {current['images']['core']: 'sha256:' + 'c' * 64, + old['images']['core']: 'sha256:' + 'd' * 64} + containers = {} + def response(args, **kwargs): + if args[:2] == ['docker', 'run']: + request = json.loads(kwargs['input']) + assert request['command'] == 'init' and request['options']['mesh_network'] == 'deploy-fixture' + assert args[args.index('--network') + 1] == 'none' + identity = args[-2] + initialized.append(identity) + assert initialized[0] == (old if has_previous else current)['images']['core'], \ + 'rehearsal initialized the candidate instead of the previous release' + data = Path(args[args.index('-v') + 1].removesuffix(':/data')) + for name in ('identity.json', 'bootstrap.json', 'data-epoch.json', 'tls-cert.pem', 'tls-key.pem'): + (data / name).write_text(identity + ':' + name) + (data / 'config.json').write_text(json.dumps({'format': identity})) + connection = sqlite3.connect(data / 'node.sqlite3') + connection.execute('CREATE TABLE identity_meta(key TEXT PRIMARY KEY,value TEXT NOT NULL)') + connection.execute("INSERT INTO identity_meta VALUES('bootstrap_capability_hash','preserved')") + connection.commit(); connection.close() + return json.dumps({'ok': True, 'result': {'device_id': 'e' * 64}}) + if args[:2] == ['docker', 'create']: + container = 'fixture-' + str(len(containers)); containers[container] = args[-1] + return container + if args[:2] == ['docker', 'commit']: + return images[containers[args[-1]]] + raise AssertionError('unexpected fixture command: ' + repr(args)) + def command(args, **kwargs): + if args[0] == 'install': + if '-d' in args: + Path(args[-1]).mkdir() + else: + shutil.copyfile(args[-2], args[-1]) + elif args[:2] == ['docker', 'compose']: + assert '-f' in args and args[-1] == 'down' + cleaned.append(args[args.index('--project-name') + 1]) + elif args[0] != 'chown' and args[:2] not in (['docker', 'cp'], ['docker', 'rm']): + raise AssertionError('unexpected fixture command: ' + repr(args)) + def activation(args, **kwargs): + if args[:3] == ['docker', 'image', 'rm']: + return subprocess.CompletedProcess(args, 0) + assert args[2] == str(ROOT / 'scripts/deploy-activate.py') + settings = json.loads(kwargs['input']) + validate_config(settings) + identity = args[args.index('--image') + 1] + expected = old if has_previous and not activated else current + assert identity == images[expected['images']['core']] + data = Path(settings['data_dir']) + assert json.loads((data / 'config.json').read_text())['format'] == expected['images']['core'] + assert (data / 'identity.json').read_text() == (old if has_previous else current)['images']['core'] + ':identity.json' + if failure == 'activation' and activated: + return subprocess.CompletedProcess(args, 1, '', 'fixture cross-version activation refused') + if failure == 'data' and activated: + connection = sqlite3.connect(data / 'node.sqlite3') + connection.execute("UPDATE identity_meta SET value='lost previous data'") + connection.commit(); connection.close() + status = 'already_active' if activated and activated[-1] == identity else 'activated' + activated.append(identity) + return subprocess.CompletedProcess(args, 0, json.dumps({'status': status, 'image': identity, + 'source_sha': expected['source_sha'], 'version': expected['version']}), '') + with patch.object(os, 'geteuid', return_value=0), patch.object(sys, 'platform', 'linux'), \ + patch.object(os, 'chown'), patch.object(tempfile, 'mkdtemp', return_value=str(host)) as directory, \ + patch.object(deploy, 'previous', return_value={'tag': old['tag']} if has_previous else None), \ + patch(__name__ + '.build', return_value=old) as previous_build, \ + patch(__name__ + '.checked', side_effect=command), \ + patch('subprocess.check_output', side_effect=response), \ + patch('subprocess.run', side_effect=activation): + try: + result = rehearse(output) + except ValueError as error: + assert failure and ('cross-version activation refused' in str(error) or 'identity or data' in str(error)) + else: + assert failure is None, 'failed upgrade or lost data was accepted' + assert result['mode'] == ('upgrade' if has_previous else 'first_release') + assert result['previous_tag'] == (old['tag'] if has_previous else None) + assert result['rehearsal'] == (['activated', 'activated', 'already_active'] if has_previous + else ['activated', 'already_active']) + assert result['identity_and_data_preserved'] is True + directory.assert_called_once_with(dir=output, prefix='rehearsal-') + if has_previous: + previous_build.assert_called_once_with('node', old['tag'], output / 'previous') + assert initialized == [old['images']['core'], current['images']['core']] + else: + previous_build.assert_not_called() + assert initialized == [current['images']['core']] + assert len(cleaned) == 1, 'fixture containers were not cleaned up' + print('deployment transport and rehearsal self-test passed') def main(): diff --git a/scripts/deploy.py b/scripts/deploy.py index 29f06686b..72face1da 100644 --- a/scripts/deploy.py +++ b/scripts/deploy.py @@ -1,6 +1,7 @@ #!/usr/bin/env python3 """Select published product artifacts independently of publishing a release.""" import argparse +import datetime import hashlib import json import re @@ -81,11 +82,49 @@ def plan(tag): else: raise ValueError('tag nesting is too deep') source = obj['sha'] - comparison = json.loads(gh('api', f'repos/{REPOSITORY}/compare/{source}...main')) - release.require(comparison['merge_base_commit']['sha'] == source, 'release is not on main') + fetch = ['git', 'fetch', '--tags'] + if release.run('git', 'rev-parse', '--is-shallow-repository') == 'true': + fetch.append('--unshallow') + subprocess.run([*fetch, 'origin', 'main', 'refs/tags/' + tag], cwd=release.ROOT, check=True) + release.require(release.run('git', 'rev-parse', tag + '^{commit}') == source, 'fetched release tag source mismatch') + release.validate_source(source, kind) return verify(tag, manifest, published, source) +def previous(tag): + """Select the last complete product release published no later than the candidate.""" + kind, _, _ = release.parse_tag(tag) + release.require(kind == 'product', 'Core rehearsal requires a product release') + selected = json.loads(gh('api', f'repos/{REPOSITORY}/releases/tags/{tag}')) + release.require(selected['tag_name'] == tag and not selected['draft'], 'candidate release is not published') + cutoff = datetime.datetime.fromisoformat(selected['published_at'].replace('Z', '+00:00')) + pages = json.loads(gh('api', '--paginate', '--slurp', f'repos/{REPOSITORY}/releases?per_page=100')) + release.require(isinstance(pages, list) and all(isinstance(page, list) for page in pages), + 'invalid release listing') + published_releases = [published for page in pages for published in page] + release.require(any(published['tag_name'] == tag for published in published_releases), + 'candidate is missing from the release listing') + candidates = [] + for published in published_releases: + name = published['tag_name'] + if published['draft'] or name == tag or not re.fullmatch('v' + release.VERSION, name): + continue + _, version, _ = release.parse_tag(name) + names = {asset['name'] for asset in published['assets']} + required = {f'vibeface-core-{version}-x86_64-unknown-linux-gnu.tar.gz', + f'vibeface-app-{version}.tar.gz'} + required |= {asset_name + '.sha256' for asset_name in required} | {'release.json'} + if not required <= names: + continue + date = datetime.datetime.fromisoformat(published['published_at'].replace('Z', '+00:00')) + if date <= cutoff: + candidates.append((date, name)) + if not candidates: + return None + # A final manifest claims completeness; failed verification must not fall back to an older release. + return plan(max(candidates)[1]) + + def selection(service, tag): kind, _, _ = release.parse_tag(tag) release.require((service == 'www') == (kind == 'www'), 'website deployments require a www tag; other services require a product tag') @@ -197,17 +236,76 @@ def self_test(): f'repos/{REPOSITORY}/releases/assets/1': content, f'repos/{REPOSITORY}/git/ref/tags/{tag}': json.dumps({'object': {'type': 'tag', 'sha': 'd' * 40}}), f'repos/{REPOSITORY}/git/tags/{"d" * 40}': json.dumps({'object': {'type': 'commit', 'sha': sha}}), - f'repos/{REPOSITORY}/compare/{sha}...main': json.dumps({'merge_base_commit': {'sha': sha}}), } - with patch(__name__ + '.gh', side_effect=lambda *args: routes[args[-1]]): + with patch(__name__ + '.gh', side_effect=lambda *args: routes[args[-1]]), \ + patch('release.run', return_value=sha), patch('subprocess.run'), patch('release.validate_source') as source_gate: assert plan(tag)['source_sha'] == sha - for route, bad in ((f'repos/{REPOSITORY}/releases/assets/1', content + ' '), - (f'repos/{REPOSITORY}/compare/{sha}...main', - json.dumps({'merge_base_commit': {'sha': 'e' * 40}}))): + source_gate.assert_called_once_with(sha, 'product') + for route, bad in ((f'repos/{REPOSITORY}/releases/assets/1', content + ' '),): saved = routes[route]; routes[route] = bad with TestCase().assertRaises(ValueError, msg='accepted changed manifest or non-main source'): plan(tag) routes[route] = saved + def listed(name, date, draft=False, complete=True): + names = {f'vibeface-core-{name[1:]}-x86_64-unknown-linux-gnu.tar.gz', + f'vibeface-app-{name[1:]}.tar.gz'} + names |= {name + '.sha256' for name in names} + if complete: + names.add('release.json') + return {'tag_name': name, 'draft': draft, 'published_at': date, + 'assets': [{'name': value} for value in sorted(names)]} + baseline = 'v0.0.1-nightly.20261107.1' + candidate = listed(tag, '2026-11-08T12:00:00Z') + pages = [[listed('v0.0.1-nightly.20261110.1', '2026-11-10T12:00:00Z'), candidate, + listed('v0.0.1-nightly.20261108.2', '2026-11-08T11:00:00Z', draft=True), + listed('v0.0.1-nightly.20261108.3', '2026-11-08T10:00:00Z', complete=False), + listed('www-v0.0.1', '2026-11-08T11:00:00Z')], + [listed('v0.0.1-nightly.20261106.1', '2026-11-06T12:00:00Z'), + listed(baseline, '2026-11-07T12:00:00Z')]] + missing_component = listed('v0.0.1-nightly.20261108.4', '2026-11-08T11:30:00Z') + missing_component['assets'].pop(0) + pages[0].append(missing_component) + for response, expected in ((pages, baseline), ([pages[0]], None)): + with patch(__name__ + '.gh', side_effect=[json.dumps(candidate), json.dumps(response)]) as query, \ + patch(__name__ + '.plan', return_value={'tag': baseline}) as verify_previous: + assert previous(tag) == ({'tag': expected} if expected else None) + query.assert_called_with('api', '--paginate', '--slurp', f'repos/{REPOSITORY}/releases?per_page=100') + if expected: + verify_previous.assert_called_once_with(expected) + else: + verify_previous.assert_not_called() + with patch(__name__ + '.gh', side_effect=[json.dumps(candidate), json.dumps(pages)]), \ + patch(__name__ + '.plan', side_effect=ValueError('invalid previous manifest')) as verify_previous: + try: + previous(tag) + except ValueError as error: + assert str(error) == 'invalid previous manifest' + else: + raise AssertionError('fell back after a previous release verification failure') + verify_previous.assert_called_once_with(baseline) + for response in (dict(candidate, published_at='invalid'), dict(candidate, draft=True)): + with patch(__name__ + '.gh', return_value=json.dumps(response)): + try: + previous(tag) + except ValueError: + pass + else: + raise AssertionError('invalid release metadata accepted as a first release') + for response in ([], [[]], ['invalid page']): + with patch(__name__ + '.gh', side_effect=[json.dumps(candidate), json.dumps(response)]): + try: + previous(tag) + except ValueError: + pass + else: + raise AssertionError('invalid release listing accepted as a first release') + with patch(__name__ + '.gh', side_effect=ValueError('offline GitHub failure')): + try: + previous(tag) + except ValueError as error: + assert str(error) == 'offline GitHub failure' + else: + raise AssertionError('GitHub failure accepted as a first release') with patch(__name__ + '.plan', return_value={'tag': tag, 'source_sha': sha}), \ patch(__name__ + '.gh') as command: execute('plan', 'all', tag) diff --git a/scripts/install-node-service.sh b/scripts/install-node-service.sh index 837382b43..a36c5e13b 100755 --- a/scripts/install-node-service.sh +++ b/scripts/install-node-service.sh @@ -87,6 +87,38 @@ different_data() { exit 1 } +launchd_node_action() { + [[ -f "$service" ]] || { echo 'Installed node service is required.' >&2; return 1; } + if [[ "$action" == stop || "$action" == restart ]]; then + if launchctl print "$domain/$label" >/dev/null 2>&1; then + launchctl bootout "$domain/$label" || return + fi + fi + if [[ "$action" == restart ]]; then + local attempt observation + # Keep the normal stop path so Core can finish shutting down its managed process trees. + for ((attempt = 0; attempt < 300; attempt++)); do + if observation=$(launchctl print "$domain/$label" 2>&1); then + sleep 0.1 + elif [[ "$observation" == *'Could not find service'* ]]; then + launchctl bootstrap "$domain" "$service" || return + launchctl kickstart "$domain/$label" + return + else + echo 'Could not confirm node service removal.' >&2 + return 1 + fi + done + echo 'Timed out waiting for node service removal.' >&2 + return 1 + elif [[ "$action" == start ]]; then + if ! launchctl print "$domain/$label" >/dev/null 2>&1; then + launchctl bootstrap "$domain" "$service" || return + fi + launchctl kickstart "$domain/$label" + fi +} + case "$(uname -s)" in Darwin) if [[ "$service_id" == node ]]; then @@ -177,18 +209,7 @@ PLIST printf 'Installed %s\n' "$service" ;; start|stop|restart) - [[ -f "$service" ]] || { echo 'Installed node service is required.' >&2; exit 1; } - if [[ "$action" == stop || "$action" == restart ]]; then - if launchctl print "$domain/$label" >/dev/null 2>&1; then - launchctl bootout "$domain/$label" - fi - fi - if [[ "$action" == start || "$action" == restart ]]; then - if ! launchctl print "$domain/$label" >/dev/null 2>&1; then - launchctl bootstrap "$domain" "$service" - fi - launchctl kickstart "$domain/$label" - fi + launchd_node_action ;; status) status_code=0 diff --git a/scripts/nightly_source.py b/scripts/nightly_source.py new file mode 100644 index 000000000..a8b04ed08 --- /dev/null +++ b/scripts/nightly_source.py @@ -0,0 +1,289 @@ +"""Exact mechanical Nightly trees shared by every release source gate.""" +from contextlib import contextmanager +from dataclasses import dataclass +import datetime +import json +import os +from pathlib import Path +import re +import subprocess +import tempfile + +import release + +PATHS = ('apps/www/package.json', 'apps/app/package.json', 'apps/core/Cargo.toml', + 'apps/cli/Cargo.toml', 'Cargo.lock', 'apps/mobile/app.json', 'apps/desktop/package.json') +PRODUCT_APPS = release.PRODUCT_APPS +BOT = '41898282+github-actions[bot]@users.noreply.github.com' +require = release.require + + +@dataclass(frozen=True) +class DailyPlan: + batch: str + base: str + versions: dict + android_code: int + ios_build_number: str + run_id: str + previous: str = '' + + def __post_init__(self): + slot = datetime.datetime.strptime(self.batch, '%Y%m%d%H') + require(slot.strftime('%Y%m%d%H') == self.batch and slot.hour % 4 == 0, + 'Nightly batch must be a four-hour UTC slot') + require(re.fullmatch(r'[1-9][0-9]*', self.run_id), 'Nightly source must name its original schedule run') + require(not self.previous or re.fullmatch(release.COMMIT, self.previous), 'invalid previous Nightly source') + + @property + def day(self): + return self.batch[:8] + + @property + def message(self): + return (f'chore(release): nightly {self.batch}\n\nNightly-CI-Run: {self.run_id}\n' + f'Nightly-Previous: {self.previous or "none"}') + + +def git(*args, root=None, data=None, env=None): + return subprocess.check_output(['git', *args], cwd=root or release.ROOT, input=data, env=env).decode().strip() + + +def blob(commit, path): + return subprocess.check_output(['git', 'show', f'{commit}:{path}'], cwd=release.ROOT) + + +def json_value(text, path, value): + data = json.loads(text) + require(json.dumps(data, indent=2, ensure_ascii=False) + '\n' == text, 'version JSON formatting changed; refuse unrelated formatting edits') + parent = data + for key in path[:-1]: + parent = parent[key] + require(path[-1] in parent, 'missing JSON version field') + parent[path[-1]] = value + return json.dumps(data, indent=2, ensure_ascii=False) + '\n' + + +def package_version(text, value): + match = re.search(r'(?ms)^\[package\]\s*\n(.*?)(?=^\[|\Z)', text) + require(match, 'missing Cargo package section') + section, count = re.subn(r'(?m)^(version\s*=\s*)"[^"]+"', lambda m: m[1] + json.dumps(value), match[1]) + require(count == 1, 'Cargo package must own exactly one version') + return text[:match.start(1)] + section + text[match.end(1):] + + +def versions_at(commit): + values = {} + for app in release.APPS: + path = f'apps/{app}/' + ('Cargo.toml' if app in ('core', 'cli') else 'app.json' if app == 'mobile' else 'package.json') + text = blob(commit, path).decode() + if app in ('core', 'cli'): + match = re.search(r'(?ms)^\[package\]\s*\n(.*?)(?=^\[|\Z)', text) + require(match, 'missing Cargo package') + values[app] = re.search(r'(?m)^version\s*=\s*"([^"]+)"', match[1])[1] + else: + info = json.loads(text) + values[app] = info['expo']['extra']['releaseVersion'] if app == 'mobile' else info['version'] + release.version(values[app]) + return values + + +def next_ios_build_number(value): + """Increment the last valid CFBundleVersion component without rollover.""" + ios_number(value) + parts = value.split('.') + maximum = 9999 if len(parts) == 1 else 99 + last = int(parts[-1]) + require(last < maximum, 'iOS build number component cannot be incremented without rollover') + parts[-1] = str(last + 1) + return '.'.join(parts) + + +def generated(plan): + old = versions_at(plan.base) + require(set(plan.versions) == set(release.APPS), 'daily plan must contain all release applications') + product_ver = plan.versions['cli'] + for app in PRODUCT_APPS: + require(plan.versions[app] == product_ver, 'product apps must share one canonical version') + for app, value in plan.versions.items(): + release.version(value) + prefix = old[app].split('-', 1)[0] + '-nightly.' + plan.day + '.' + require(value.startswith(prefix), 'daily version changed baseline or date') + if release.version(old[app]) == 'nightly': + require(nightly_order(value) > nightly_order(old[app]), 'daily version did not advance') + result = {path: blob(plan.base, path).decode() for path in PATHS} + for app in ('www', 'app', 'desktop'): + path = f'apps/{app}/package.json' + result[path] = json_value(result[path], ('version',), plan.versions[app]) + for app in ('core', 'cli'): + path = f'apps/{app}/Cargo.toml' + result[path] = package_version(result[path], plan.versions[app]) + blocks = result['Cargo.lock'].split('[[package]]') + found = set() + for index, block in enumerate(blocks): + match = re.search(r'(?m)^name = "(vibeface|vibeface-core)"$', block) + if match: + app = 'cli' if match[1] == 'vibeface' else 'core' + require(app not in found and f'version = "{old[app]}"' in block, 'Cargo.lock local package mismatch') + blocks[index], count = re.subn(r'(?m)^version = "[^"]+"$', 'version = "' + plan.versions[app] + '"', block) + require(count == 1, 'ambiguous local lock version') + found.add(app) + require(found == {'core', 'cli'}, 'Cargo.lock is missing local packages') + result['Cargo.lock'] = '[[package]]'.join(blocks) + mobile_path = 'apps/mobile/app.json' + expo = json.loads(result[mobile_path])['expo'] + require(expo['version'] == plan.versions['mobile'].split('-', 1)[0], 'mobile marketing version mismatch') + code, ios = next_build_numbers(plan.base, plan.previous) + require(plan.android_code == code, 'Android versionCode must advance exactly once') + require(plan.ios_build_number == ios, 'iOS build number must advance exactly once') + result[mobile_path] = json_value(result[mobile_path], ('expo', 'extra', 'releaseVersion'), plan.versions['mobile']) + result[mobile_path] = json_value(result[mobile_path], ('expo', 'android', 'versionCode'), plan.android_code) + result[mobile_path] = json_value(result[mobile_path], ('expo', 'ios', 'buildNumber'), plan.ios_build_number) + return {path: text.encode() for path, text in result.items()} + + +def recovered_plan(head): + parents = git('rev-list', '--parents', '-n', '1', head).split() + require(len(parents) == 2, 'daily version head must be one ordinary commit') + message = git('show', '-s', '--format=%B', head) + match = re.fullmatch(r'chore\(release\): nightly ([0-9]{10})\n\nNightly-CI-Run: ([1-9][0-9]*)\nNightly-Previous: (none|[a-f0-9]{40})', message) + require(match, 'Nightly source is missing its original schedule identity') + expo = json.loads(blob(head, 'apps/mobile/app.json'))['expo'] + return DailyPlan(match[1], parents[1], versions_at(head), expo['android']['versionCode'], + expo['ios']['buildNumber'], match[2], '' if match[3] == 'none' else match[3]) + + +def expected_tree(plan): + expected = generated(plan) + with tempfile.TemporaryDirectory(dir=scratch(), prefix='index-') as temporary: + environment = dict(os.environ, GIT_INDEX_FILE=str(Path(temporary) / 'index')) + git('read-tree', plan.base, env=environment) + for path, content in expected.items(): + entry = git('ls-tree', plan.base, '--', path).split() + require(entry[:2] == ['100644', 'blob'], 'version source must be an ordinary tracked file') + oid = git('hash-object', '-w', '--stdin', data=content) + git('update-index', '--add', '--cacheinfo', '100644', oid, path, env=environment) + return git('write-tree', env=environment) + + +def verify_tree(plan, head): + require(git('show', '-s', '--format=%ae%n%ce', head).splitlines() == [BOT, BOT], 'unexpected daily commit author') + require(git('rev-list', '--parents', '-n', '1', head).split() == [head, plan.base], 'daily version head changed its original base') + require(expected_tree(plan) == git('rev-parse', head + '^{tree}'), + 'Nightly tree differs from exact mechanical version changes') + + +def scratch(): + directory = Path(os.environ.get('VIBEFACE_RELEASE_SCRATCH', release.ROOT / '.artifacts/nightly-source')) + directory.mkdir(parents=True, exist_ok=True) + return directory + + +@contextmanager +def checkout(commit): + with tempfile.TemporaryDirectory(dir=scratch(), prefix='checkout-') as temporary: + source = Path(temporary) / 'source' + git('worktree', 'add', '--detach', str(source), commit) + try: + yield source + finally: + git('worktree', 'remove', '--force', str(source)) + + +def metadata(source, plan): + before = (source / 'Cargo.lock').read_bytes() + result = subprocess.check_output(['cargo', 'metadata', '--locked', '--no-deps', '--format-version', '1'], cwd=source) + packages = {package['name']: package['version'] for package in json.loads(result)['packages']} + require(packages['vibeface'] == plan.versions['cli'] and packages['vibeface-core'] == plan.versions['core'], + 'Cargo metadata does not match daily versions') + require((source / 'Cargo.lock').read_bytes() == before, 'cargo metadata changed dependency resolution') + + +def nightly_order(value): + require(release.version(value) == 'nightly', 'previous product source must be Nightly') + baseline, suffix = value.split('-nightly.') + day, sequence = suffix.split('.') + return (*map(int, baseline.split('.')), int(day), int(sequence)) + + +def ios_number(value): + require(isinstance(value, str), 'iOS build number must be a string') + parts = value.split('.') + require(1 <= len(parts) <= 3 and re.fullmatch(r'[0-9]{1,4}', parts[0]) + and int(parts[0]) > 0 and all(re.fullmatch(r'[0-9]{1,2}', part) for part in parts[1:]), + 'iOS build number must be a valid Apple CFBundleVersion') + numeric = tuple(map(int, parts)) + return numeric + (0,) * (3 - len(numeric)) + + +def next_build_numbers(base, previous=''): + versions = [json.loads(blob(commit, 'apps/mobile/app.json'))['expo'] + for commit in (base, previous) if commit] + codes = [expo['android']['versionCode'] for expo in versions] + require(all(type(code) is int and 0 < code < 2100000000 for code in codes), + 'Android versionCode cannot be incremented') + ios = max((expo['ios']['buildNumber'] for expo in versions), key=ios_number) + return max(codes) + 1, next_ios_build_number(ios) + + +_verified_sources = {} + + +def source_context(source, release_app): + require(release_app in release.RELEASES and re.fullmatch(release.COMMIT, source), 'invalid release source') + context = dict(ci_source_commit=source, ci_run_id='', nightly_batch='') + ancestry = subprocess.run(['git', 'merge-base', '--is-ancestor', source, 'origin/main'], + cwd=release.ROOT, capture_output=True) + if ancestry.returncode == 0: + return context + require(ancestry.returncode == 1, 'cannot verify release main ancestry') + origin = source + repository = str((release.ROOT / git('rev-parse', '--git-common-dir')).resolve()) + main = git('rev-parse', 'origin/main') + pending = [] + previous_links = [] + seen = set() + while True: + key = (repository, main, source, release_app) + if key in _verified_sources: + if source == origin: + context = _verified_sources[key] + break + require(source not in seen, 'cyclic previous Nightly source') + seen.add(source) + ancestry = subprocess.run(['git', 'merge-base', '--is-ancestor', source, 'origin/main'], + cwd=release.ROOT, capture_output=True) + if ancestry.returncode == 0: + pending.append((key, dict(ci_source_commit=source, ci_run_id='', nightly_batch=''))) + break + require(ancestry.returncode == 1, 'cannot verify release main ancestry') + values = versions_at(source) + require(release.version(values['cli' if release_app == 'product' else 'www']) == 'nightly', + 'only Nightly mechanical sources may be outside main') + plan = recovered_plan(source) + require(subprocess.run(['git', 'merge-base', '--is-ancestor', plan.base, 'origin/main'], + cwd=release.ROOT, capture_output=True).returncode == 0, + 'Nightly parent must be on main') + if plan.previous: + require(nightly_order(versions_at(plan.previous)['cli']) < nightly_order(values['cli']), + 'previous Nightly must precede this product version') + previous_links.append((plan.previous, plan.base)) + verify_tree(plan, source) + with checkout(source) as clean: + metadata(clean, plan) + current = dict(ci_source_commit=plan.base, ci_run_id=plan.run_id, nightly_batch=plan.batch) + pending.append((key, current)) + if source == origin: + context = current + if not plan.previous: + break + source = plan.previous + release_app = 'product' + resolved = _verified_sources | dict(pending) + for previous, parent in previous_links: + previous_context = resolved[(repository, main, previous, 'product')] + require(subprocess.run(['git', 'merge-base', '--is-ancestor', previous_context['ci_source_commit'], parent], + cwd=release.ROOT, capture_output=True).returncode == 0, + 'previous Nightly main source must be an ancestor of the current parent') + _verified_sources.update(pending) + return context diff --git a/scripts/progressive_publication_fixtures.py b/scripts/progressive_publication_fixtures.py index d91153680..262e7cb4c 100644 --- a/scripts/progressive_publication_fixtures.py +++ b/scripts/progressive_publication_fixtures.py @@ -1,6 +1,7 @@ #!/usr/bin/env python3 """Real producer directories for progressive publication tests (local fixtures only).""" import json +import os import shutil import subprocess import tempfile @@ -69,7 +70,7 @@ def build_collected_tree(source=None): expo = json.loads((ROOT / 'apps/mobile/app.json').read_text(encoding='utf-8'))['expo'] badging = (f"package: name='{expo['android']['package']}' versionCode='{expo['android']['versionCode']}'" f" versionName='{expo['version']}'") - scratch = ROOT / '.artifacts/progressive-publication-tests' + scratch = Path(os.environ.get('VIBEFACE_RELEASE_SCRATCH', ROOT / '.artifacts/progressive-publication-tests')) scratch.mkdir(parents=True, exist_ok=True) installers = scratch / 'installers' installers.mkdir(exist_ok=True) diff --git a/scripts/release.py b/scripts/release.py index e4cbae7ff..c4268c6ed 100644 --- a/scripts/release.py +++ b/scripts/release.py @@ -19,6 +19,8 @@ import zipfile sys.dont_write_bytecode = True +if __name__ == '__main__': + sys.modules.setdefault('release', sys.modules[__name__]) ROOT = Path(__file__).resolve().parent.parent APPS = ('www', 'app', 'core', 'cli', 'mobile', 'desktop') @@ -126,6 +128,52 @@ def identity(release_app, root=None): 'source_commit': run('git', 'rev-parse', 'HEAD', cwd=root), 'channel': version(value)} +def validate_source(source, release_app, root=None): + """Return the main CI source after proving the entire detached Nightly tree.""" + import nightly_source + global ROOT + previous = ROOT + ROOT = root or ROOT + try: + return nightly_source.source_context(source, release_app) + finally: + ROOT = previous + + +def require_ci(context): + """Main tags require push success; detached Nightlies require their original schedule gate.""" + source, run_id = context['ci_source_commit'], context['ci_run_id'] + def api(path): + return json.loads(run('gh', 'api', f'repos/{GITHUB_REPO}/' + path)) + if run_id: + candidates = [api(f'actions/runs/{run_id}')] + else: + batches = json.loads(run('gh', 'api', '--paginate', '--slurp', + f'repos/{GITHUB_REPO}/actions/workflows/ci.yml/runs?head_sha={source}&branch=main&event=push&per_page=100')) + candidates = [item for page in batches for item in page['workflow_runs']] + for candidate in candidates: + if not (candidate['head_sha'] == source and candidate['head_branch'] == 'main' + and candidate['path'] == '.github/workflows/ci.yml' + and candidate['repository']['full_name'] == GITHUB_REPO + and candidate['head_repository']['full_name'] == GITHUB_REPO): + continue + if not run_id: + if candidate['event'] == 'push' and candidate['status'] == 'completed' and candidate['conclusion'] == 'success': + return candidate + continue + require(str(candidate['id']) == run_id and candidate['event'] == 'schedule', 'untrusted original Nightly CI run') + created = datetime.datetime.fromisoformat(candidate['created_at'].replace('Z', '+00:00')).astimezone(datetime.timezone.utc) + batch = created.replace(hour=created.hour // 4 * 4).strftime('%Y%m%d%H') + require(batch == context['nightly_batch'], 'Nightly CI run does not match its original batch') + pages = json.loads(run('gh', 'api', '--paginate', '--slurp', + f'repos/{GITHUB_REPO}/actions/runs/{run_id}/jobs?filter=latest&per_page=100')) + gates = [job for page in pages for job in page['jobs'] if job['name'] == 'ci-gate'] + require(len(gates) == 1 and gates[0]['status'] == 'completed' and gates[0]['conclusion'] == 'success', + 'original Nightly schedule ci-gate did not succeed') + return candidate + raise ValueError('release requires the exact trusted main push CI or original Nightly schedule ci-gate') + + def expected_release_kind_for_component(app): return 'www' if app == 'www' else 'product' @@ -664,7 +712,7 @@ def require_product_publishing(): tag = expected['tag'] require(run('git', 'rev-parse', tag + '^{commit}') == expected['source_commit'], 'publishing requires the existing release tag at this source commit') - subprocess.run(['git', 'merge-base', '--is-ancestor', expected['source_commit'], 'origin/main'], check=True) + validate_source(expected['source_commit'], 'product') return expected @@ -1077,7 +1125,7 @@ def publish(release_app, directory, repo=GITHUB_REPO): info = verify_release(release_app, directory, expected) tag = info['tag'] require(run('git', 'rev-parse', tag + '^{commit}') == info['source_commit'], 'release tag source mismatch') - subprocess.run(['git', 'merge-base', '--is-ancestor', info['source_commit'], 'origin/main'], check=True) + validate_source(info['source_commit'], release_app) if release_app == 'product': return finalize_progressive_product(directory, expected, info, repo) view = subprocess.run(['gh', 'release', 'view', tag, f'--repo={repo}', '--json', 'isDraft,assets'], @@ -1174,7 +1222,7 @@ def plan(release_app, tag=None, require_main=False, publishing=False, github_out 'tag does not match the declared version') if require_main or publishing: require(not run('git', 'status', '--porcelain'), 'release source must be clean') - subprocess.run(['git', 'merge-base', '--is-ancestor', 'HEAD', 'origin/main'], check=True) + data.update(validate_source(data['source_commit'], release_app)) if publishing: require(run('git', 'rev-parse', data['tag'] + '^{commit}') == data['source_commit'], 'publishing requires the existing release tag at this source commit') @@ -1276,7 +1324,7 @@ def boundary(*args, **kwargs): return json.dumps(dict(sha=public_view['source'])) return real_run(*args, **kwargs) - scratch = ROOT / '.artifacts/release-tests' + scratch = Path(os.environ.get('VIBEFACE_RELEASE_SCRATCH', ROOT / '.artifacts/release-tests')) scratch.mkdir(parents=True, exist_ok=True) # The per-platform user service installers are repository files; the real compose job proves they # exist, so the self-test composes against fixtures instead of the checkout's scripts directory. diff --git a/scripts/verify-node.py b/scripts/verify-node.py index 7e45b3d09..9f50206ec 100755 --- a/scripts/verify-node.py +++ b/scripts/verify-node.py @@ -14,6 +14,8 @@ --memory-sync checks signed user Memory controls, concurrent Agent changes, receipts, provenance and replay across two isolated Nodes with the same controlled fixtures. --web-only checks packaged Web resources through one isolated Node process. +--npm-service checks native npm service management and published-to-candidate replacement +only on fresh GitHub-hosted POSIX runners, never a developer's user services. --auto-route checks the entry: joins, origins, revocation, unknown writes, recovery and forwarding through another member Gateway. """ @@ -30,6 +32,7 @@ import ipaddress import json import os +import plistlib import re import runpy from html.parser import HTMLParser @@ -4547,6 +4550,428 @@ def provenance(node): raise RuntimeError("Memory fixture cleanup failed; see " + str(run / "report.json")) +def pin_npm_baseline(options): + import release + + source = Path(__file__).resolve().parents[1] + output = options.artifacts.resolve() + require(output.is_relative_to(source / ".artifacts") and not output.exists(), + "npm baseline pinning requires a fresh checkout artifact directory") + candidate = release.run("git", "rev-parse", "HEAD", cwd=source) + require(os.environ.get("GITHUB_SHA", candidate) == candidate, "npm baseline pins this exact candidate source") + with urllib.request.urlopen("https://registry.npmjs.org/@leaperone%2fvibeface", timeout=30) as response: + encoded = response.read(4 * 1024 * 1024 + 1) + require(len(encoded) <= 4 * 1024 * 1024, "npm metadata is bounded") + registry = json.loads(encoded) + version = registry["dist-tags"]["nightly"] + require(release.version(version) == "nightly", "the public baseline is a real Nightly") + metadata = registry["versions"][version] + require(metadata["name"] == "@leaperone/vibeface" and metadata["version"] == version, + "the pinned registry version matches its package") + tarball = f"https://registry.npmjs.org/@leaperone/vibeface/-/vibeface-{version}.tgz" + integrity = metadata["dist"]["integrity"] + require(metadata["dist"]["tarball"] == tarball and re.fullmatch(r"sha512-[A-Za-z0-9+/]{86}==", integrity), + "the baseline has an immutable registry tarball and SHA-512 integrity") + tag = "v" + version + record = release.release_record(tag) + require(not record["draft"] and record["published_at"] and record["tag_name"] == tag + and not any(marker in (record.get("body") or "") for marker in (release.BUILDING_MARKER, release.FAILED_MARKER)), + "the baseline product release is public and complete") + remote = release.remote_assets_by_name(record["assets"]) + require("release.json" in remote and all(release.remote_asset_sha256(asset) for asset in remote.values()), + "baseline product assets require immutable GitHub SHA-256 metadata") + baseline_source = json.loads(release.run("gh", "api", f"repos/{release.GITHUB_REPO}/commits/{tag}"))["sha"] + context = release.validate_source(baseline_source, "product", root=source) + require(baseline_source != candidate, "baseline and candidate must have different real source commits") + require(subprocess.run(["git", "merge-base", "--is-ancestor", context["ci_source_commit"], candidate], + cwd=source, capture_output=True).returncode == 0, + "the baseline business source must be an ancestor of the candidate; future baselines are refused") + output.mkdir(parents=True, mode=0o700) + with tempfile.TemporaryDirectory(dir=output, prefix=".verify-") as scratch: + scratch = Path(scratch) + + def download_asset(name): + for _ in range(3): + (scratch / name).unlink(missing_ok=True) + result = subprocess.run(["gh", "release", "download", tag, f"--repo={release.GITHUB_REPO}", + "--pattern", name, "--dir", str(scratch)], cwd=source, capture_output=True) + if result.returncode == 0: + require(file_sha256(scratch / name) == release.remote_asset_sha256(remote[name]), + "the pinned public product asset matches its immutable digest") + return + raise AssertionError("cannot download pinned public product asset " + name) + + expected = {"app": "product", "tag": tag, "version": version, "source_commit": baseline_source, "channel": "nightly"} + download_asset("release.json") + release.verify_remote_product_manifest(tag, release.GITHUB_REPO, expected, remote, scratch) + archive = output / "baseline.tgz" + for attempt in range(3): + digest = hashlib.sha512() + archive.unlink(missing_ok=True) + try: + with urllib.request.urlopen(tarball, timeout=90) as response, archive.open("xb") as stream: + require(response.geturl() == tarball, "the pinned npm download does not redirect") + size = 0 + for chunk in iter(lambda: response.read(1024 * 1024), b""): + size += len(chunk) + require(size <= release.MAX_ARCHIVE, "the npm baseline archive is bounded") + stream.write(chunk) + digest.update(chunk) + break + except (OSError, http.client.HTTPException): + require(attempt < 2, "cannot download the pinned public npm baseline") + require("sha512-" + base64.b64encode(digest.digest()).decode() == integrity, "the baseline tarball matches registry integrity") + unpacked = release.unpack(archive, scratch / "npm") + npm = json.loads((unpacked / "package.json").read_text()) + require(npm["name"] == "@leaperone/vibeface" and npm["version"] == version + and npm["bin"] == {"vibeface": "bin/vibeface.mjs"} + and not any(npm.get(key) for key in ("scripts", "dependencies", "optionalDependencies")) + and {path.name for path in (unpacked / "vendor").iterdir()} == set(release.TARGETS), + "the published npm baseline is a complete package without install hooks or downloaded dependencies") + builds = {} + for target in release.TARGETS: + vendor = unpacked / "vendor" / target + build = json.loads((vendor / "BUILD-INFO.json").read_text()) + release.verify_build_info(build) + require(build["source_commit"] == baseline_source and build["version"] == version and build["target"] == target, + "every baseline vendor belongs to the complete product source and current data epoch") + name = f"vibeface-{version}-{target}.tar.gz" + download_asset(name) + native = release.unpack(scratch / name, scratch / target) + require(release.files(native) == release.files(vendor), "npm vendor bytes equal the published native product bundle") + builds[target] = build + shutil.copyfile(scratch / "release.json", output / "release.json") + pin = {"version": version, "source_commit": baseline_source, "source_context": context, + "candidate_source": candidate, "published_at": registry["time"][version], "release_id": record["id"], + "integrity": integrity, "tarball": tarball, "sha256": file_sha256(archive), + "manifest_sha256": file_sha256(output / "release.json"), "builds": builds, + "business_source_relation": "same" if context["ci_source_commit"] == candidate else "ancestor"} + private_json(output / "pin.json", pin) + print("PASS pinned public npm baseline " + json.dumps({key: pin[key] for key in + ("version", "source_commit", "source_context", "candidate_source", "sha256", "manifest_sha256")}), flush=True) + + +def npm_service_acceptance(options): + require(os.environ.get("GITHUB_ACTIONS") == "true" + and os.environ.get("RUNNER_ENVIRONMENT") == "github-hosted", + "npm service acceptance requires a disposable GitHub-hosted runner") + require(sys.platform in ("darwin", "linux"), "npm service acceptance requires launchd or systemd") + import fcntl + import tarfile + + source = Path(__file__).resolve().parents[1] + artifacts = options.artifacts.resolve() + require(artifacts.is_relative_to(source / ".artifacts"), "npm service artifacts stay in the checkout") + binary, core = options.binary.resolve(strict=True), options.core_binary.resolve(strict=True) + root, vendor = binary.parent.parent, core.parent + require(binary.name == "vibeface.mjs" and root.name == "vibeface" + and root.parent.name == "@leaperone" and root.parents[1].name == "node_modules" + and root.parents[2].name == "lib" and root.is_relative_to(source / ".artifacts") + and vendor.parent == root / "vendor", "service acceptance uses an isolated scoped npm installation") + prefix = root.parents[3] + native = vendor / "vibeface" + module = options.crypto_module.resolve(strict=True) + require(module == vendor / "web/vibeface_secure.js", "the npm client uses its own packaged WASM") + require(options.npm_package is not None, "npm service acceptance requires the immutable CI npm artifact") + package = options.npm_package.resolve(strict=True) + require(package.is_relative_to(source / ".artifacts") and package.suffix == ".tgz", + "npm replacement uses the downloaded CI artifact") + metadata = json.loads((root / "package.json").read_text()) + build = json.loads((vendor / "BUILD-INFO.json").read_text()) + target = {("darwin", "arm64"): "aarch64-apple-darwin", + ("linux", "x86_64"): "x86_64-unknown-linux-gnu", + ("linux", "aarch64"): "aarch64-unknown-linux-gnu"}.get((sys.platform, os.uname().machine)) + require(target is not None and build["target"] == target and metadata["name"] == "@leaperone/vibeface" + and metadata["version"] == build["version"] and build["data_epoch"] == DATA_EPOCH + and all(build[field] == os.environ.get("GITHUB_SHA") + for field in ("source_commit", "core_source_commit", "app_source_commit")), + "the npm artifact pins this runner, source commit and current data format") + home = Path.home() + require(not list(home.glob(".vibeface*")), "the disposable runner has no existing Vibeface data") + manager_env = os.environ.copy() + + def command(arguments, *, check=True): + result = subprocess.run([str(argument) for argument in arguments], capture_output=True, text=True, + timeout=90, cwd=source, env=manager_env) + require(not check or result.returncode == 0, + f"{Path(arguments[0]).name} {arguments[1]} succeeds: {result.stderr[-400:]}") + return result + + def manager_absent(unit): + if sys.platform == "darwin": + result = command(["/bin/launchctl", "print", f"{domain}/{unit}"], check=False) + return result.returncode != 0 and "Could not find service" in result.stderr + result = command(["/usr/bin/systemctl", "--user", "show", unit, "--property=LoadState", "--value"], check=False) + return result.stdout.strip() == "not-found" + + if sys.platform == "darwin": + domain = f"gui/{os.getuid()}" + service_dir = home / "Library/LaunchAgents" + units = ("com.leaperone.vibeface.node", "com.leaperone.vibeface.update") + service_files = [service_dir / (unit + ".plist") for unit in units] + command(["/bin/launchctl", "print", domain]) + else: + runtime = Path(f"/run/user/{os.getuid()}") + require(os.environ.get("XDG_CONFIG_HOME", str(home / ".config")) == str(home / ".config") + and os.environ.get("XDG_RUNTIME_DIR") == str(runtime) + and os.environ.get("DBUS_SESSION_BUS_ADDRESS") == f"unix:path={runtime}/bus" + and (runtime / "bus").is_socket(), "the real disposable user's D-Bus is available") + service_dir = home / ".config/systemd/user" + units = ("vibeface.service", "vibeface-update.service", "vibeface-update.timer") + service_files = [service_dir / unit for unit in units] + command(["/usr/bin/systemctl", "--user", "show-environment"]) + require(all(manager_absent(unit) for unit in units), "the disposable manager has no existing default Vibeface service") + require(not any(path.exists() or path.is_symlink() for path in service_files), + "the disposable runner has no existing default service files") + require(options.npm_baseline is not None, "npm service acceptance requires a pinned public baseline") + baseline = options.npm_baseline.resolve(strict=True) + require(baseline.is_relative_to(source / ".artifacts"), "the pinned npm baseline stays in checkout artifacts") + pin = json.loads((baseline / "pin.json").read_text()) + require(pin["candidate_source"] == build["source_commit"] and pin["source_commit"] != build["source_commit"] + and file_sha256(baseline / "baseline.tgz") == pin["sha256"] + and file_sha256(baseline / "release.json") == pin["manifest_sha256"] + and pin["builds"][target]["data_epoch"] == build["data_epoch"], + "the immutable baseline pins a different real source, this exact candidate and the current data format") + + artifacts.mkdir(parents=True, exist_ok=True) + run = artifacts / ("npm-service-" + uuid.uuid4().hex[:10]) + run.mkdir(mode=0o700) + workspace = run / "workspace" + workspace.mkdir(mode=0o700) + helper = run / "client.mjs" + helper.write_text(CLIENT_SOURCE.replace("MODULE_URL", module.as_uri())) + cleanup_script = run / "cleanup-service.sh" + candidate_checksums = {name: file_sha256(vendor / name) for name in + ("vibeface", "vibeface-core", "BUILD-INFO.json", "FILES.sha256", "scripts/install-node-service.sh")} + nightly_order = lambda value: tuple(map(int, re.fullmatch(r"(\d+)\.(\d+)\.(\d+)-nightly\.(\d{8})\.(\d+)", value).groups())) + baseline_version, candidate_version = nightly_order(pin["version"]), nightly_order(build["version"]) + version_order = (candidate_version > baseline_version) - (candidate_version < baseline_version) + report = {"status": "running", "target": target, + "candidate": {"version": build["version"], "source_commit": build["source_commit"], + "package_sha256": file_sha256(package), "binary_sha256": file_sha256(native), "core_sha256": file_sha256(core)}, + "manager": "launchd" if sys.platform == "darwin" else "systemd --user", "cases": [], + "native_path": str(native.relative_to(root)), "native_pids": [], + "baseline": {key: pin[key] for key in ("version", "source_commit", "source_context", "sha256", "manifest_sha256")}, + "baseline_artifact_id": os.environ.get("NPM_BASELINE_ARTIFACT_ID", ""), + "candidate_declared_version": {1: "higher", 0: "same", -1: "lower"}[version_order], + "business_source_relation": pin["business_source_relation"], "real_provider_calls": 0, + "replacement": "public current-format baseline to a different-source immutable CI candidate", + "not_checked": ["historical-format migration", "real AI Providers", + "full lifecycle of the immutable public baseline; only the candidate lifecycle is required"]} + node, owned = None, False + + def service(action, *, check=True): + entry = {"action": action, "source_commit": json.loads((vendor / "BUILD-INFO.json").read_text())["source_commit"]} + report.setdefault("service_actions", []).append(entry) + result = command([binary, "service", action, "--data-dir", node.directory], check=check) + entry["returncode"] = result.returncode + return result + + def passed(name): + report["cases"].append(name) + print("PASS npm service " + name, flush=True) + + def reinstall(destination, archive): + command(["npm", "install", "--global", "--prefix", destination, "--cache", run / "npm-cache", + "--offline", "--ignore-scripts", "--no-audit", "--no-fund", "--force", archive]) + + def persistent(): + files = {name: file_sha256(node.directory / name) for name in + ("identity.json", "config.json", "data-epoch.json", "node.lock", "update.lock", "update.json", "update-pending.json") + if (node.directory / name).exists()} + with sqlite3.connect(f"file:{node.directory / 'node.sqlite3'}?mode=ro", uri=True) as database: + events = dict(database.execute("SELECT hash,envelope FROM events")) + return files, events + + def rejected(action, expected): + before = persistent() + service_before = [path.read_bytes() if path.exists() else None for path in service_files] + result = service(action, check=False) + require(result.returncode != 0 and expected in result.stderr, "npm service refuses " + expected) + require(persistent() == before and service_before == [path.read_bytes() if path.exists() else None + for path in service_files], + "refused npm service operations preserve existing data and service files") + + def idle_guards(): + rejected("install-updater", "npm installations are updated with npm") + with (node.directory / "update.lock").open("r+b") as lock: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + rejected("install", "wait for the updater before installing the npm service") + private_json(node.directory / "update-pending.json", {"fixture": "pending"}) + rejected("install", "recover the pending update") + (node.directory / "update-pending.json").unlink() + before = persistent() + update = json.loads(command([binary, "update", "status", "--data-dir", node.directory]).stdout) + require(update["distribution"] == "npm" and update["automatic"] is False, + "npm owns updates and disables the built-in updater") + command([binary, "update", "apply", "--automatic", "--data-dir", node.directory]) + require(persistent() == before, "npm automatic update exits without changing persistent data") + + def stopped(): + wait_for("npm service stops its Node", lambda: node_status(node)["status"] == "stopped") + if sys.platform == "darwin": + wait_for("launchd unloads the stopped Node", lambda: manager_absent(units[0])) + else: + state = dict(line.split("=", 1) for line in command([ + "/usr/bin/systemctl", "--user", "show", units[0], "--property=MainPID,LoadState"], check=False).stdout.splitlines()) + require(state.get("LoadState") == "not-found" or state.get("MainPID") == "0", + "systemd has no running Node PID") + + def running(): + wait_for("native npm service starts", lambda: node_status(node)["status"] == "running") + require(node.client.request("/api/info")["device_id"] == node.device_id, "the service retains the fixture identity") + if sys.platform == "darwin": + with service_files[0].open("rb") as stream: + require(plistlib.load(stream)["ProgramArguments"] == [str(native), "serve", "--data-dir", str(node.directory)], + "launchd runs the scoped npm vendor CLI with the fixture data") + state = command(["/bin/launchctl", "print", f"{domain}/{units[0]}"]).stdout + pid = int(re.search(r"(?m)^\s*pid = (\d+)$", state)[1]) + else: + state = dict(line.split("=", 1) for line in command([ + "/usr/bin/systemctl", "--user", "show", units[0], "--property=MainPID,ExecStart,FragmentPath"]).stdout.splitlines()) + require(state["FragmentPath"] == str(service_files[0]) and f"path={native} ;" in state["ExecStart"], + "systemd runs the scoped npm vendor CLI from the fixture unit") + pid = int(state["MainPID"]) + require(Path(f"/proc/{pid}/exe").resolve(strict=True) == core, "the running Core is the npm vendor executable") + process = command(["/bin/ps", "-ww", "-p", str(pid), "-o", "command="]).stdout.strip() + require(process == f"{core} --data-dir {node.directory}", "the manager PID executes the exact scoped npm Core") + require(not any(path.exists() for path in service_files[1:]) and all(manager_absent(unit) for unit in units[1:]), + "npm never installs or loads its built-in updater") + node.client.connect() + node.client.authenticate() + report["native_pids"].append(pid) + return pid + + try: + candidate = run / "candidate-prefix" + reinstall(candidate, package) + candidate_vendor = candidate / "lib/node_modules/@leaperone/vibeface/vendor" / target + require(candidate_checksums == {name: file_sha256(candidate_vendor / name) for name in candidate_checksums}, + "the independent npm prefix contains the exact immutable CI candidate") + reinstall(prefix, baseline / "baseline.tgz") + require(json.loads((root / "package.json").read_text())["version"] == pin["version"] + and json.loads((vendor / "BUILD-INFO.json").read_text()) == pin["builds"][target], + "the first installed service package is the verified public baseline") + import release + with tarfile.open(baseline / "baseline.tgz", "r:gz") as reference: + checksums = reference.extractfile(f"package/vendor/{target}/FILES.sha256").read().decode().splitlines() + expected_files = {name: digest for digest, name in (line.split(" ", 1) for line in checksums)} + require(release.files(vendor, ("FILES.sha256",)) == expected_files, + "every installed baseline vendor byte matches the immutable published package") + shutil.copy2(vendor / "scripts/install-node-service.sh", cleanup_script) + node = Node(run, "NPM", binary, workspace, helper) + node.configure(bins={"codex": str(fixture_wrapper(run, "fixture-codex", "--fixture-codex"))}, + manager=fixture_manager_settings(run, workspace), app_server=SPAWN_APP_SERVER) + node.start() + network = node.bootstrap() + conversation = "npm-service-history" + node.client.rpc("create_bot", bot_id="npm-service-bot", conversation_id=conversation, name="Npm fixture", + instructions="", project={"target_node": node.device_id, "cwd": str(workspace)}) + attachment = b"npm service attachment\x00\xff\n" + digest = hashlib.sha256(attachment).hexdigest() + uploaded = node.client.rpc("upload_attachment", conversation_id=conversation, hash=digest, size=len(attachment), + offset=0, data=base64.b64encode(attachment).decode()) + require(uploaded == {"offset": len(attachment), "complete": True}, "the fixture persists the complete attachment") + receipt = node.client.rpc("message", conversation_id=conversation, message_id="npm-service-message", + text=json.dumps({"chat_agent": True, "label": "npm-service-message"}), + attachments=[{"hash": digest, "size": len(attachment), "name": "fixture.bin"}]) + wait_for("the controlled fixture completes its actual message", lambda: + (scheduler_record(node, network, receipt["trigger"]) or {}).get("state") == "completed") + messages = node.client.rpc("conversation", conversation_id=conversation)["messages"] + require(any(row["message_id"] == "npm-service-message" and row["attachments"] for row in messages) + and any(row["role"] == "assistant" for row in messages), "preservation starts with real nonempty message history") + invocation_count = len(fixture_records(run)) + require(invocation_count == 1, "one controlled fixture call creates the message history") + update = json.loads(command([binary, "update", "status", "--data-dir", node.directory]).stdout) + require(update["distribution"] == "npm" and update["automatic"] is False, + "npm owns updates and disables the built-in updater") + rejected("install", "stop the node before installing its npm service") + passed("public baseline active Node refuses install without changing data or services") + node.client.close() + node.stop() + stopped() + idle_guards() + passed("public baseline updater lock, pending journal and second updater refuse without mutation") + saved_files, saved_events = persistent() + + def preserved(): + files, events = persistent() + require(files == saved_files and saved_events and all(events.get(key) == value for key, value in saved_events.items()), + "npm lifecycle preserves identity, configuration and every original signed event") + require(node.client.rpc("conversation", conversation_id=conversation)["messages"] == messages, + "npm lifecycle preserves the exact nonempty message history") + require(len(fixture_records(run)) == invocation_count, "service restarts never replay a completed fixture turn") + page = node.client.rpc("read_artifact", conversation_id=conversation, blob_hash=digest, binary=True) + require(page["size"] == len(attachment) and page["next_offset"] is None + and base64.b64decode(page["data"]) == attachment, "npm lifecycle returns the original attachment bytes") + + owned = True + service("install") + running() + service("status") + preserved() + service("stop") + stopped() + passed("immutable public baseline installs a real service and stops before candidate replacement") + node.client.close() + definition = service_files[0].read_bytes() + previous_inode = core.stat().st_ino + reinstall(prefix, package) + require(core.stat().st_ino != previous_inode and candidate_checksums == {name: file_sha256(vendor / name) for name in candidate_checksums}, + "npm replaces the stopped public baseline with the exact different-source CI candidate") + require(file_sha256(package) == report["candidate"]["package_sha256"] and service_files[0].read_bytes() == definition, + "npm replacement preserves the immutable input and installed service definition") + stopped() + service("start") + running() + preserved() + passed("published-to-candidate npm replacement preserves identity, config, signed history and attachment bytes") + rejected("install", "stop the node before installing its npm service") + service("stop") + stopped() + idle_guards() + service("install") + running() + service("status") + service("stop") + stopped() + service("start") + previous = running() + service("restart") + require(running() != previous, "the candidate's native manager restart replaces its PID") + preserved() + passed("candidate install, stop, start, restart and concurrency guards preserve baseline data") + service("uninstall") + stopped() + require(not any(path.exists() or path.is_symlink() for path in service_files) and all(manager_absent(unit) for unit in units), + "uninstall removes the fixture services from disk and the native manager") + files, events = persistent() + require(files == saved_files and all(events.get(key) == value for key, value in saved_events.items()) + and (node.directory / "blobs/sha256" / digest[:2] / digest).read_bytes() == attachment, + "uninstall retains the fixture identity, configuration, history and attachment") + passed("uninstall removes the native service and retains data") + report["status"] = "passed" + except Exception as error: + report.update(status="failed", failure=f"{type(error).__name__}: {error}") + raise + finally: + cleanup = ([lambda: command([cleanup_script, "uninstall", "--data-dir", node.directory])] if owned else []) + cleanup += ([node.client.close, node.stop] if node else []) + [lambda: stop_fixture_processes(run)] + failures = [] + for action in cleanup: + try: + action() + except Exception as error: + failures.append(f"{type(error).__name__}: {error}") + if failures: + report.update(status="failed", cleanup_failures=failures) + report["controlled_fixture_calls"] = len(fixture_records(run)) + private_json(run / "report.json", report) + print("REPORT " + json.dumps(report, sort_keys=True), flush=True) + if failures and "failure" not in report: + raise RuntimeError("npm service cleanup failed") + + def web_acceptance(options): binary = options.binary.resolve(strict=True) module = options.crypto_module.resolve(strict=True) @@ -9154,7 +9579,7 @@ def stop(*_): fixture_codex(sys.argv[1:]) else: parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--binary", type=Path, required=True, + parser.add_argument("--binary", type=Path, help="the vibeface CLI binary used to create and run the isolated Nodes") parser.add_argument("--core-binary", type=Path, help="the sibling vibeface-core binary to verify; defaults to --binary's sibling") @@ -9162,8 +9587,12 @@ def stop(*_): help="development static output directory; packages discover sibling web/ automatically") parser.add_argument("--artifacts", type=Path, required=True) parser.add_argument("--crypto-module", type=Path, default=Path(__file__).resolve().parents[1] / "apps/app/dist/client/vibeface_secure.js") + parser.add_argument("--npm-package", type=Path, help="the immutable CI npm artifact for --npm-service replacement") + parser.add_argument("--npm-baseline", type=Path, help="the public baseline artifact pinned once for this candidate") modes = parser.add_mutually_exclusive_group() modes.add_argument("--web-only", action="store_true", help="check packaged Web resources without Node business or provider scenarios") + modes.add_argument("--npm-service", action="store_true", help="check real npm user services only on a fresh GitHub-hosted POSIX runner") + modes.add_argument("--npm-baseline-pin", action="store_true", help="pin one public current-format npm baseline and its complete product provenance") modes.add_argument("--phase1", action="store_true", help="run focused durable Manager queue/follow-up process checks with controlled fixtures") modes.add_argument("--chat-agent", action="store_true", help="run direct App Server work, triage, decisions, cancellation and native observation process checks") modes.add_argument("--memory-sync", action="store_true", help="check signed user Memory controls, cross-Node conflicts, provenance and replay") @@ -9190,8 +9619,9 @@ def stop(*_): arguments = parser.parse_args() try: needs_process = not (arguments.delivery_ui_selfcheck or arguments.native_provider_selfcheck - or arguments.native_clipboard_selfcheck) + or arguments.native_clipboard_selfcheck or arguments.npm_baseline_pin) if needs_process: + require(arguments.binary is not None, "--binary is required for Node process acceptance") program = shutil.which(str(arguments.binary)) if len(arguments.binary.parts) == 1 else str(arguments.binary) require(program is not None, "the Vibeface CLI binary exists on PATH or at its explicit path") launcher = Path(program) @@ -9212,7 +9642,9 @@ def stop(*_): require(arguments.app_dir.is_dir(), "the explicit App output directory exists") os.environ["VIBEFACE_APP_DIR"] = str(arguments.app_dir) require(30_000 <= arguments.delivery_ui_timeout_ms <= 86_400_000, "UI acceptance has a bounded explicit execution deadline") - if arguments.native_provider_selfcheck: + if arguments.npm_baseline_pin: + pin_npm_baseline(arguments) + elif arguments.native_provider_selfcheck: native_provider_selfcheck(arguments) elif arguments.native_clipboard_selfcheck: native_clipboard_selfcheck(arguments) @@ -9235,6 +9667,8 @@ def stop(*_): phase4_acceptance(arguments) elif arguments.auto_route: auto_route_acceptance(arguments) + elif arguments.npm_service: + npm_service_acceptance(arguments) elif arguments.web_only: web_acceptance(arguments) elif arguments.chat_agent: