Skip to content

Tooling: maintained live-proxy deploy and rollback scripts that follow the documented procedure #80

Closed

Description

@anuzzolilli-oss

(ai: anthropic/claude-fable-5-2)

Follow-up to #78 / #79. The deploy procedure in reference/cindy-project.md now follows the cutover order of the kept Deploy-Master.ps1, but the kept scripts still depart from it (a failed rollback re-enables supervision unchecked, one fixed version, stale defaults, a two-line launcher edit, a rollback that checks nothing) and they cannot be tested anywhere but against the live proxy. The owner asked on 2026-11-17 for a spec that closes these departures with maintained scripts in the repository. The spec below is the scope of this issue. It went through four independent reviews and one second-round review; nothing is implemented yet.

Triage: p3, enhancement; topic category K (launcher / self-update / CI tooling). No host lane applies. No deployment is requested by this issue.


Live-proxy deploy and rollback scripts aligned with the documented procedure

Handoff Status: READY FOR EXECUTION — accepted by the owner on 2026-11-17; PR #79 is merged. The last round of fixes was checked by the owner's read, not by a third independent review.

Owner: repository owner (anuzzolilli-oss)
Last Updated: 2026-11-17
Spec Version: v4 (four independent reviews, one second-round review, one final read by the author)


One-Line Description

Add maintained, parameterised, CI-tested deploy and rollback scripts for the live proxy to the repository, behaving as the section "Deploy only with an explicit request" in reference/cindy-project.md describes.

Terms

Term Meaning
runtime root The installation folder holding the launcher, receipts, releases\, backups\, config\, state\. The live one is %APPDATA%\CindyGlobal\billion-context-proxy
launcher Start-BillionContextProxy.ps1 in the runtime root; run by the scheduled task under Windows PowerShell 5.1; a running instance is the supervisor
hold maintenance.inhibit present, scheduled task disabled, no supervisor running. Lift = inhibit removed, task enabled, task started
candidate The new build being deployed, in a folder outside the runtime root. Previous = the release the launcher names when the script starts
receipts tuned-deployment.json and supervision-receipt.json in the runtime root
backup folder One folder per deployment under <runtime root>\backups
kept scripts The unreviewed one-off scripts this work replaces: tooling\Deploy-Master.ps1 in %APPDATA%\CindyGlobal\billion-context-proxy\backups\before-master-20261114-214204, and Rollback-MasterDeploy.ps1 in that backup's root (a file the deploy script generated)
sandbox A temporary runtime root built by the tests, with a stand-in launcher and proxy

Motivation

Why this exists: The kept scripts are the only deploy tooling, and they depart from the documented procedure:

# Kept scripts Documented procedure
D1 After a rollback, supervision is re-enabled whether or not the previous release is healthy (deploy script only prints the result; rollback script checks nothing) A failed rollback ends with the hold in place and a report to the owner (step 8)
D2 Health is compared with one fixed version string, 0.1.213; the same literal is written into a receipt Version comes from the candidate (step 5)
D3 Release names, commits and hashes are parameter defaults from a past deployment Values come from the candidate and the current launcher (step 5)
D4 The launcher edit rewrites exactly ProxyPath and ExpectedCommit ExpectedVersion changes too when the version changes (step 5)
D5 The rollback script is generated text with machine paths, written only after a successful cutover One procedure usable after any failure
D6 An error inside the rollback branch leaves an undefined state Fail closed: hold in place
D7 Runtime root, port and task name are fixed The scripts cannot be tested anywhere but against the live proxy; in this repository CI is the authority

What happens if we don't: The failed-rollback path runs for the first time during a real failure, in an unreviewed script that puts an unhealthy release back under automatic restart.

Triggering context: Issue #78 / PR #79 made the documented steps follow the script's cutover order and recorded where the scripts still depart. The owner asked on 2026-11-17 for a spec to close the departures.

Owner decisions (2026-11-17)

Decision Answer
Where the aligned scripts live In the repository; the backup-folder copies stay untouched
Scope All known departures D1–D7; the functional checks of step 7 stay manual
Proof Sandbox tests in CI; nothing runs against port 8787 or the live runtime root
Failed-rollback end state Hold back on; any proxy process left as it is; distinct exit code; evidence printed
Rollback form One maintained rollback script reading a deployment record saved in the backup folder
Expected config hash Supplied by the operator as a parameter; the receipt's deployedConfigSha256 is stale (the live config was changed on 2026-11-15, after the last deployment) and is neither trusted nor rewritten

Goals

  • A deployment run with the repository script follows the documented steps without the operator editing the script.
  • Every failure from the moment the inhibit is created ends in one of two defined states: previous release healthy under supervision, or hold in place.
  • Rolling back a completed or half-completed deployment is one reviewed script.
  • All paths, including the failure paths, are exercised by tests that cannot reach the live proxy.

Non-Goals

  • Deploying, or running anything against the live proxy or port 8787 — this work ends at a merged PR.
  • Editing or deleting the kept scripts or anything else under the live runtime root — read-only for this work.
  • Importing the real launcher, build.mjs, live-entry.ts, iso-test.mjs or fold-test.mjs — separate scope. No content of the real launcher is copied into fixtures.
  • Automating step 7's functional checks (forwarding, folding, retrieval, checker logs, provider request) — owner decision.
  • Any change to src/, kernel/, pi-subagents/, bili configuration, environment variables or the CLI.
  • Registering or modifying the scheduled task — the scripts only disable, enable and start it.
  • Reading or printing bili.log — the kept script's "last exposure line" output is dropped on purpose (log content does not belong in script output).

Step coverage

Step Covered by
1, 3 Manual (build and isolated tests), unchanged
2 Deploy-LiveProxy.ps1 preflight, release directory, backup
4, 5, 6 Deploy-LiveProxy.ps1
7 Receipt rewrite: Deploy-LiveProxy.ps1. Functional checks: manual, after exit 0
8 Deploy-time: Deploy-LiveProxy.ps1. Later (step 7 failed): Rollback-LiveProxy.ps1
9 Both scripts, as part of their success test (AC-9)
10 Receipts and the printed report: both scripts. The owner-facing report stays the operator's

Acceptance Criteria

Deploy script

  • AC-1 Input validation (exit 2, nothing created). Refuse unless: -CandidateVersion matches ^\d+\.\d+\.\d+(-[0-9A-Za-z.]+)?$; -CandidateCommit matches ^[0-9a-f]{7,12}(-[a-z0-9]+)?$; -CandidateSourceCommit is 40 lower-case hex; -CandidateBundleSha256 and -ExpectedConfigSha256 are 64 hex; -ReleaseName matches ^[A-Za-z0-9][A-Za-z0-9_-]*(\.[A-Za-z0-9_-]+)*$, is at most 64 characters, and <runtime root>\releases\<name> resolves to an immediate child of <runtime root>\releases; -TaskName contains no wildcard character; no value or resolved path contains ', a backtick, $, CR, LF or a non-ASCII character; -RuntimeRoot and -CandidateRoot are absolute, exist, and no component of either is a reparse point; -CandidateRoot is not inside the runtime root.
  • AC-2 Preflight (read-only except one private temp file; exit 2). Every check runs and prints one CHECK PASS|FAIL line with expected and observed; the script refuses if any fails:
    1. Candidate has dist\index.mjs, dist\build-info.json, package.json, LICENSE, none a reparse point; build-info.json commit and sourceCommit equal the supplied values; package.json version equals -CandidateVersion; SHA-256 of dist\index.mjs equals -CandidateBundleSha256, and so does bundleSha256 in build-info.json when that field is present. All hash comparisons are case-insensitive.
    2. Launcher is plain ASCII without BOM and contains exactly one line of each shape [string]$ProxyPath = '<v>',, [string]$ExpectedCommit = '<v>',, [string]$ExpectedVersion = '<v>',, and one [int]$Port = <n>,. These give the previous release's path, commit and version. The previous ProxyPath is <runtime root>\releases\<x>\dist\index.mjs and exists.
    3. -Port equals the launcher's $Port.
    4. The task exists exactly once, is enabled, and its action's arguments contain the full path of the launcher in the runtime root (ordinal, case-insensitive).
    5. SHA-256 of config\billion-context\billion-context.json equals -ExpectedConfigSha256. The value in the receipt is not consulted.
    6. Health is ok with the previous version and commit, reported from the PID of the single listener; that listener is the single process running the previous ProxyPath.
    7. Exactly one supervisor, and supervisor.lock is held.
    8. maintenance.inhibit is absent; the new release directory is absent.
    9. The previous release's node_modules is a junction or symbolic link whose target is an existing directory.
    10. The edited launcher text differs from the current text only in the ProxyPath line, the ExpectedCommit line when the commit differs, and the ExpectedVersion line when the version differs (positional line comparison; 1 to 3 lines).
    11. Dry-run: the edited text, written to a file in a newly created private temp directory, hashed after writing and again before running, run under Windows PowerShell 5.1 with -RuntimeRoot <root> -ReportOnly, exits 0 and prints JSON with an action field. The temp directory is removed in finally.
    12. -ToolingDir, if given, exists, contains no reparse point, and is not inside releases\ or the backup folder this run would create (the kept tooling folder under an older backup is allowed).
    13. Client connections to the port: reported, not refused, under -PreflightOnly; refused in a full run.
      Probes for listeners, connections and processes must propagate inspection errors, never read an error as "free" or "idle".
  • AC-3 -PreflightOnly runs AC-1 and AC-2, prints the plan (previous and candidate identity, release directory, backup folder name, launcher lines that would change), creates nothing in the runtime root, and exits 0 or 2.
  • AC-4 Preparation. Creates releases\<ReleaseName> (without -Force), copies the four candidate files, re-verifies all four hashes on the copies against hashes taken in AC-2, and creates node_modules there as the same link type and target as the previous release's. Creates the backup folder backups\before-deploy-<yyyyMMdd-HHmmss>-<ReleaseName>, copies into it Start-BillionContextProxy.ps1, tuned-deployment.json, supervision-receipt.json, proxy.identity.json, supervisor-status.json, and -ToolingDir (if given; top-level files only) as tooling\. Writes deploy-record.json at stage prepared, including launcherSha256After (the hash of the edited text, known since AC-2.10) and this process's PID and creation time, and prints BACKUP: <full path>. Runs the dry-run of AC-2.11 once more, now expecting the action to be conflict (the previous proxy is listening and does not match the edited identity). Any failure here: remove the release directory this run created (the node_modules link is deleted as a link, never recursed into), keep the backup folder, set stage aborted, exit 6.
  • AC-5 Hold. Write stage holding first. Then create maintenance.inhibit with create-new semantics (fails if the file exists), content deploy <deploymentId>. If that creation fails, nothing was changed: set stage aborted, clean up as in AC-4 and exit 6; someone else's inhibit is never touched. The hold window starts once this run's inhibit exists: a top-level trap is armed at that moment, and from then until AC-9 passes every error goes to rollback (AC-10). Order after the inhibit: task disabled → supervisor stopped (by verified PID and creation time) → no supervisor process and supervisor.lock free → previous proxy still healthy.
  • AC-6 Launcher edit. Re-check that the launcher's SHA-256 equals the one recorded at preflight. If it does not, someone else edited it: go straight to AC-11 (no restore, reason foreign-launcher-edit), never overwrite that edit. Otherwise write stage held, then write the edited text (UTF-8 without BOM) to a temp file in the same directory, verify it against launcherSha256After, rename.
  • AC-7 Idle re-check. If a client connection is open now, roll back with reason not-idle; the previous proxy is never stopped.
  • AC-8 Cutover. Write stage cut before acting. Stop only the previous proxy (verified listener PID and creation time), wait for the port, confirm zero supervisors and the lock free, then lift: inhibit removed → task enabled → task started. The script starts no proxy.
  • AC-9 Success test (also used for the previous release in rollback, with its identity). All of the following, polled until true within -HealthWaitSeconds: health ok with the expected version and commit from the PID of the single listener; that listener's executable arguments name the expected ProxyPath; exactly one process runs that ProxyPath; exactly one supervisor; supervisor-status.json has a heartbeatAt later than the lift, supervisorPid equal to that supervisor, state adopted or spawned, healthMatch true, inhibit false, childPid equal to the listener; the task is enabled; maintenance.inhibit is absent; the config file's SHA-256 is still the one checked in preflight. Once these hold, supervisor.lock is probed once and must be held, and the task's executable, arguments and trigger count are compared with those captured in preflight and must be unchanged. When AC-9 passes for the candidate the rollback trap is disarmed: any later error is exit 5, never a rollback of a healthy release.
  • AC-10 Rollback (deploy-time). The record keeps the stage reached in reachedStage before the stage changes to a rollback value.
    • Reached holding (the launcher was never edited; for example the task could not be disabled): do not re-hold and do not restore; verify the launcher still has launcherSha256Before, then lift and apply AC-9 with the previous identity. Reason hold-failed.
    • Reached held or later: rollback hold = ensure this deployment's inhibit exists (content rollback <deploymentId>; one already carrying this id is accepted) → task disabled → every supervisor found now is stopped (zero is fine) → no supervisor process and the lock free. No health check in this hold. Stop every process running the candidate ProxyPath (there is none before cut), and wait for the port if one was the listener. Restore the launcher from the bytes read at preflight (temp file, verify against launcherSha256Before, rename, verify again). Lift. Apply AC-9 with the previous identity.
    • Pass → stage rolled-back, remove the release directory this run created (the node_modules link is deleted as a link, never recursed into), exit 3 with reason=hold-failed, not-idle or cutover-failed.
  • AC-11 Failed rollback. If AC-9 fails for the previous release, or any step of AC-10 throws, or AC-6 found a foreign edit: re-apply the hold (inhibit → task disabled → every supervisor found is stopped), from that point stop no proxy process and start nothing, set stage rollback-failed (keeping reachedStage), write the evidence block to the console and to <backup folder>\evidence-<timestamp>.txt, exit 4. If part of the hold could not be applied, exit 4 anyway and state which of the three parts is not in place.
  • AC-12 Receipts. After AC-9 passes (trap disarmed; every failure from here is exit 5): stage healthy; write the intended content of both receipts to <backup folder>\tuned-deployment.pending.json and supervision-receipt.pending.json; then replace each receipt (temp file in the same directory + rename; UTF-8 without BOM; ConvertTo-Json -Depth 6). Stage receipts-written, exit 0. If a replace fails: no rollback, the new release stays live, exit 5 naming the receipt(s) not replaced and the copy command that finishes the job from the pending files.

Rollback script

  • AC-13 Preconditions (exit 2, nothing changed). Refuse unless: -BackupRoot resolves to an immediate child of <runtime root>\backups with no reparse point on the folder, the record or the three files to restore; the record passes strict schema validation (unknown fields rejected, AC-1 formats applied); record.runtimeRoot equals the resolved -RuntimeRoot; both proxyPath values are <runtime root>\releases\<name>\dist\index.mjs and the previous one exists; the backup launcher's SHA-256 equals launcherSha256Before and its three identity defaults and $Port equal the record's previous values and port; the task named in the record exists exactly once and its action's arguments contain the full path of the launcher in the runtime root (it must be enabled unless maintenance.inhibit carries this record's deploymentId, in which case a disabled task is expected); the process recorded as the deploy script (PID and creation time) is no longer running; and the stage rule below allows action. It prints ROLLBACK: from <candidate> to <previous> before acting.

    Record stage Live launcher hash Action
    prepared, aborted launcherSha256Before Refuse: nothing was changed
    holding, held, cut, healthy, receipts-written, rollback-failed launcherSha256After or launcherSha256Before Roll back (a launcher that already has the Before hash is verified, not rewritten)
    rolled-back any Refuse: already rolled back
    any neither hash Refuse: the launcher was changed after this deployment; this is not the current deployment's backup

    An existing maintenance.inhibit is accepted only when its content names this record's deploymentId; any other inhibit is someone else's maintenance → refuse. There is no override switch.

  • AC-14 Action. As AC-10 and AC-11, with these differences: identities, port and task name come from the validated record; the launcher is restored from the backup file; the hold is the rollback hold of AC-10 for every stage; every process running the candidate ProxyPath is stopped whatever the stage; each receipt is restored from the backup whenever its hash differs from the backup copy, and verified byte-identical afterwards; the candidate is stopped even when clients are connected (the operator has decided to roll back), and the count of open connections is printed first; the release directory is left in place. Success → stage rolled-back, exit 0. Failure → AC-11, exit 4. The script writes rollback-result.json and never overwrites the deploy script's result.json; -PreflightOnly writes nothing.

Shared

  • AC-15 Process matching. A supervisor is a powershell.exe process whose command line contains the full resolved launcher path as a quoted token; a proxy is a node.exe process whose command line contains the full resolved ProxyPath as a quoted token. Comparison is ordinal case-insensitive Contains, never -like/-match on interpolated text. Every stop re-checks PID and creation time immediately before Stop-Process. Wherever zero supervisors are required, the process count must be zero and then supervisor.lock is probed once and must be free. The probe opens the existing file for exclusive access and closes it at once; it never creates the file and is never run in a polling loop, because a probe that coincides with a starting supervisor would make that supervisor exit. Paths are resolved once (long form, final target) and that one string is used everywhere.
  • AC-16 Sandbox seam. -TestTaskDriver is accepted only when <runtime root>\sandbox.marker exists; the default (real) task driver is refused when the marker exists; -TestTaskDriver is refused when the resolved runtime root is under the roaming application-data folder (obtained from the known-folder API, not from $env:APPDATA) in a CindyGlobal folder. The driver must be a regular file without reparse points. When active, every run prints TEST SEAM ACTIVE and repeats it in RESULT. The guard's path comparison is a pure function in the common file that takes the roaming base as an argument; it is unit-tested with injected strings (different case, trailing separator, 8.3 form, \\?\ prefix). No test creates anything under the real roaming folder, so that one rule has no end-to-end case. The marker rules and the regular-file rule each have an end-to-end failing-case test.
  • AC-17 Output. Stdout, one line per event: <ISO 8601 with offset> <KIND> <text>, kinds CHECK, STEP, WAIT (at least every 10 s during a wait), BACKUP, ROLLBACK, NEXT, RESULT. Final lines: NEXT: <exact command or instruction> and RESULT: <name> (exit <n>) [reason=…]; -PreflightOnly ends with RESULT: preflight-ok (exit 0) or refused. The same result, with stage, backup folder and both identities, is written to <backup folder>\result.json whenever a backup folder exists. Health output is limited to ok, version, commit, pid. Paths inside the runtime root are printed relative to it, except on the BACKUP line. No command lines, config contents, environment variables or tokens are printed.
  • AC-18 No literals. A test scans the three script files and fails on: a 40- or 64-hex string, a semver-shaped string, 8787, Users\, the live task name, or a releases\<name> literal. CindyGlobal may appear once, in the AC-16 guard; the loopback address 127.0.0.1 is exempt; help examples use <placeholder> values.
  • AC-19 No test hooks. The scripts have exactly the parameters listed under Interface Contracts and read no environment variable for behaviour; a test asserts the parameter lists. Fault injection happens only through the test task driver's state file, the stand-in proxy's control file, and harness-side file locks (the harness holding a file open so that a write or rename by the script fails). The executor may extend the two fixture file formats; the scripts gain no hook.
  • AC-20 Non-interactive. Both scripts carry comment-based help with one full example and are documented as pwsh -NoProfile -NonInteractive -File …; a missing mandatory parameter then fails instead of prompting.

Quality and operations

  • AC-21 Tests. Each refusal condition listed in AC-1, AC-2 and AC-13 has its own failing-case test asserting the reason printed, and every AC has a passing case. Required scenarios beyond those: commit unchanged but path changed (same-SHA rebuild); version changed and unchanged; candidate never healthy → exit 3 cutover-failed; client connects after the hold → exit 3 not-idle with the previous proxy's PID unchanged; candidate and previous both unhealthy → exit 4 with the unhealthy proxy's PID still alive; launcher restore fails → exit 4; hold re-application fails → exit 4 naming the missing part; task disable fails during the first hold → exit 3 hold-failed with the previous proxy's PID unchanged and supervision running; someone else's inhibit appears after preflight → exit 6 with that inhibit untouched and the task still enabled; foreign launcher edit before AC-6 → exit 4 with that edit preserved; an error after AC-9 passed → exit 5, candidate still live; retry of the standalone rollback after its own exit 4 on a receipts-written deployment restores the receipts; receipt replace fails → exit 5 with pending files present; record ordering (record at prepared exists before the inhibit; stage cut written before the previous proxy stops); standalone rollback at each stage in the AC-13 table; foreign inhibit → refuse; the evidence file contains no content of the sandbox config file; the scripts write nothing under config\ or state\ and never register or modify the task.
  • AC-22 Isolation. With a second sandbox running, a full deploy and rollback in the first leaves the second's supervisor and proxy PIDs unchanged; root names that are prefixes of each other (…-a1, …-a10) are covered. The harness builds every sandbox from the resolved long form of the temp path (CI runners may report an 8.3 form) and refuses a root that is not <temp>\bili-live-deploy-<random>, a root under the roaming application-data folder, port 8787 anywhere (including inside a record), and tears down only PIDs it started or that its sandbox supervisor reported, with creation-time checks.
  • AC-23 CI. tests/live-deploy-scripts.test.ts is registered in LIVE_FILES in scripts/live-files.mjs and has an entry in scripts/ci-shard-weights.json; it is appended last to LIVE_FILES (the list is append-only); it runs in the test (windows-latest, 24, n) jobs; each sandbox port is obtained by binding port 0, reading the assigned port and closing the socket before the launcher is written (inside the CI zone band where the zone-slot preload provides one); off Windows it skips with a visible reason; on Windows a missing pwsh or powershell.exe is a failure, not a skip. All waits are parameters; no case sleeps for a fixed time.
  • AC-24 Repository checks. Green: npm run typecheck, npm run check:unused-exports, npm run check:test-import-graph, node tools/check-architecture-docs.mjs, git diff --check. npm pack --dry-run lists none of the new files (expected without change: package.json files is an allowlist).
  • AC-25 No personal paths. A test asserts that all committed files contain no user-profile path, and that script output contains none after the sandbox root itself has been replaced by a placeholder (the temp folder is under a profile on every machine).
  • AC-26 Documentation. See Task 7.

Architecture & Approach

Components

Component Responsibility
scripts/live-deploy/Deploy-LiveProxy.ps1 AC-1 to AC-12
scripts/live-deploy/Rollback-LiveProxy.ps1 AC-13, AC-14
scripts/live-deploy/LiveProxy.Common.ps1 Dot-sourced by both: validation, path resolution, probes, process matching, hold/lift, real task driver, launcher restore, success test, rollback core, record and result writing, evidence block
tests/live-deploy-scripts.test.ts Drives both scripts with pwsh against sandboxes
tests/fixtures/live-deploy/ Stand-in launcher, stand-in proxy, test task driver

Data flow

Deploy-LiveProxy.ps1
  validate ─────────────── fail ─► exit 2
  preflight (read-only) ── fail ─► exit 2
  release dir + backup + record(prepared) ── fail ─► clean up ─► exit 6
  record(holding) → inhibit (create-new) ── fail ─► exit 6
  task off → supervisor stopped                          ┐
  record(held) → launcher edited                         │ any error
  idle? ── no ──────────────────────────────────────────►│ ─► ROLLBACK
  record(cut) → stop previous proxy → port free → LIFT   │
  success test (candidate) ── fail ─────────────────────►┘
  [trap disarmed] record(healthy) → receipts ── fail ─► exit 5
                        └─ ok ─► exit 0

ROLLBACK (core shared with Rollback-LiveProxy.ps1)
  HOLD → stop candidate proxy (if cut) → restore + verify launcher
  [standalone: restore receipts] → LIFT → success test (previous)
     pass ─► exit 3 (deploy script) / exit 0 (rollback script)
     fail or any error ─► HOLD, touch no proxy, evidence ─► exit 4

Key decisions

Decision Rationale Rejected
Previous identity is read from the launcher's current defaults It is what the supervisor enforces now Parameters (stale by design, D3); receipts (top-level proxyRelease in supervision-receipt.json is known stale)
The operator supplies candidate version, commit, source commit and bundle hash, taken from the step 1 record, and the script checks the candidate against them The operator asserts what was tested; copying them from the candidate folder would prove nothing Deriving them from the candidate
The reported version is the version of the package.json beside dist\ Verified: src/version.ts reads it at runtime and the kept build.mjs does not bake it in; /__bili/health returns it (src/server/admin.ts) —
Success requires the supervisor's own status file and lock, not only health and counts Health alone is already true when the previous proxy was never stopped; a process count can pass while the supervisor sits in a conflict state. This is step 9's test Counts only (kept script)
The hold window starts at inhibit creation A failure while applying the hold must not fall between "nothing changed" and "rollback" Hold counted from supervisor stop
Inhibit carries the deployment id and is created with create-new Two runs cannot both pass; the rollback can tell its own hold from someone else's Operator switch to override an existing hold
The stage is written before the action it names A kill between action and write would make the record understate what happened Write after
Deploy-time rollback restores the launcher from bytes held in memory The backup file is not re-trusted within one run Re-read the backup
Task control goes through a driver; the test driver needs a marker file in the root, and the real driver refuses a marked root Opt-in seam that cannot be pointed at the live installation, and a sandbox that cannot reach the real task Path deny-list only
Scripts need PowerShell 7; the launcher runs under Windows PowerShell 5.1 in the dry-run and in the sandbox Health call uses -NoProxy; the real task runs 5.1 (verified read-only: its action is powershell.exe … -File "<launcher path>", multiple instances IgnoreNew, a logon trigger and a time trigger) One version for both
The expected config hash is a parameter Owner decision 2026-11-17: the receipt value is stale, and writing a corrected one would be a write under the live root outside a deployment Reading deployedConfigSha256; checking only that the hash does not change
Exit 1 is never used deliberately PowerShell returns 1 for uncaught errors and binding failures; it means "unknown, read the record" "Exit 1 = nothing changed"

API / Interface Contracts

Parameters

Deploy-LiveProxy.ps1
  -RuntimeRoot <path>               mandatory
  -Port <int>                       mandatory; must equal the launcher's $Port
  -TaskName <string>                mandatory
  -CandidateRoot <path>             mandatory
  -ReleaseName <string>             mandatory
  -CandidateVersion <string>        mandatory  ┐ from the step 1 record,
  -CandidateCommit <string>         mandatory  │ not copied from the
  -CandidateSourceCommit <string>   mandatory  │ candidate folder
  -CandidateBundleSha256 <string>   mandatory  ┘
  -ExpectedConfigSha256 <string>    mandatory; the config hash the operator verified in step 2
  -ToolingDir <path>                optional; without it the backup has no tooling\ (a STEP line says so)
  -HealthWaitSeconds <int>          default 90
  -StopWaitSeconds <int>            default 20; applies to every stop and to the port wait
  -TestTaskDriver <path>            sandbox only (AC-16)
  -PreflightOnly                    switch

Rollback-LiveProxy.ps1
  -RuntimeRoot <path>               mandatory
  -BackupRoot <path>                mandatory
  -HealthWaitSeconds <int>          default 90
  -StopWaitSeconds <int>            default 20
  -TestTaskDriver <path>            sandbox only
  -PreflightOnly                    switch; runs AC-13 and prints the action it would take

Health URL is always http://127.0.0.1:<validated port>/__bili/health, 5-second timeout, -NoProxy, no redirects. Poll interval 500 ms.

Exit codes and what to do next

Code Name State left behind Release dir NEXT: line
0 deployed Candidate healthy under supervision kept Run the step 7 checks; the ready-to-paste Rollback-LiveProxy.ps1 command with the real backup path
0 restored (rollback script) Previous healthy under supervision kept Report per step 10; a later deployment needs a new -ReleaseName
2 refused Nothing changed not created The failing checks
3 rolled-back Previous healthy under supervision removed not-idle: retry later with the same parameters. hold-failed, cutover-failed: investigate before retrying
4 rollback-failed Hold in place; proxy processes as they were; no release confirmed live kept Report to the owner with the evidence file, the paths of supervisor.log and state\billion-context\bili.log; the rollback command to retry; the three manual commands that lift the hold if the owner judges the proxy fine; a later deployment needs a new -ReleaseName
5 deployed-receipts-incomplete Candidate healthy under supervision; receipt(s) not replaced kept The copy command(s) from the pending files
6 error-before-hold Nothing changed in the running system removed The error; backup folder kept for the record
1 (unplanned) Unknown unknown Read deploy-record.json stage; run Rollback-LiveProxy.ps1 -PreflightOnly

Deployment record

<backup folder>\deploy-record.json, UTF-8 without BOM, updated by temp file + rename:

{
  "schema": 1,
  "deploymentId": "<backup folder name>",
  "createdAt": "2026-11-17T10:00:00.0000000+00:00",
  "runtimeRoot": "<path>",
  "port": 0,
  "taskName": "<string>",
  "previous":  { "proxyPath": "<path>", "version": "<s>", "commit": "<s>" },
  "candidate": { "releaseName": "<s>", "proxyPath": "<path>", "version": "<s>", "commit": "<s>", "sourceCommit": "<s>",
                 "sha256": { "index.mjs": "<hex>", "build-info.json": "<hex>", "package.json": "<hex>", "LICENSE": "<hex>" } },
  "nodeModules": { "linkType": "Junction|SymbolicLink", "target": "<path>" },
  "configSha256": "<hex>",
  "launcherSha256Before": "<hex>",
  "launcherSha256After": "<hex>",
  "deployProcess": { "pid": 0, "created": "<ISO 8601>" },
  "created": { "releaseDir": true, "backupDir": true },
  "stage": "prepared|aborted|holding|held|cut|healthy|receipts-written|rolled-back|rollback-failed",
  "reachedStage": "<highest of prepared..receipts-written reached; unchanged by rollback stages>"
}

The record holds machine paths: it stays in the backup folder and is never committed or posted.

Receipts (same fields as the kept script; two literals become computed)

deployedConfigSha256 is left as it is (owner decision; see Open Questions).

tuned-deployment.json: set masterDeployedAt (local ISO), masterBackupRoot, masterPreviousReleaseRoot, masterPreviousBuild, masterPreviousBundleSha256, masterPreviousSourceCommit from the old values; then releaseRoot, bundleSha256, build (candidate commit), previousBuild, sourceCommit, livePid, verifiedAt (UTC ISO), deployedSupervisorSha256.

supervision-receipt.json: set masterDeploy = { at, launcherSha256 (lower-case), change, proxyRelease, backup } where change lists the launcher defaults actually changed (for example ProxyPath and ExpectedCommit defaults only) and proxyRelease is releases/<ReleaseName> (<candidate version>, <candidate commit>).

No other field is added, renamed or removed.

Task driver

A .ps1 file dot-sourced by the common file, defining four functions that take the task name: Get-LiveProxyTaskInfo (returns Count, Enabled, Execute, ActionArguments, TriggerCount), Disable-LiveProxyTask, Enable-LiveProxyTask, Start-LiveProxyTask. The real driver in the common file wraps Get-/Disable-/Enable-/Start-ScheduledTask.

Test driver (tests/fixtures/live-deploy/TestTaskDriver.ps1): state in <root>\sandbox-task.json. Start does nothing while disabled, and nothing while supervisor.lock is held (models IgnoreNew); otherwise it launches the launcher with powershell.exe -NoProfile -NonInteractive -ExecutionPolicy Bypass -File "<launcher>" detached. The task's one-minute repeat is modelled by the test helper calling Start again, not by a timer process. The state file can also carry fault instructions, each with a call index so that the first and a later call can differ: failOn: [{ op: "disable"|"enable"|"start", call: <n> }] and onCall: [{ op, call, action: "openClient"|"lockLauncher"|"lockReceipt"|"deleteBackupLauncher"|"writeForeignInhibit"|"editLauncher" }]. This list is a starting point the executor may extend (AC-19).

Stand-in launcher and proxy (fixtures, written from scratch)

Launcher: plain ASCII; parameters RuntimeRoot, NodePath, ProxyPath, Port, ExpectedVersion, ExpectedCommit, ReportOnly with the default-line shapes of AC-2.2; -ReportOnly prints JSON with action (adopt, spawn, waiting, inhibited, blocked, conflict, error) and exits 0, or 2 for error; holds supervisor.lock exclusively and exits 0 if it cannot; does nothing while maintenance.inhibit exists; adopts a healthy matching listener, starts node "<ProxyPath>" … only when no matching proxy exists and nothing listens, otherwise fails closed; writes supervisor-status.json with state, childPid, childCreated, healthMatch, inhibit, supervisorPid, supervisorStarted, heartbeatAt. Its poll and heartbeat intervals are parameters with sub-second defaults. The harness writes it into each sandbox with LF-normalised bytes, so byte comparisons do not depend on checkout line endings.

Proxy: answers /__bili/health with ok, version (from the package.json beside dist\), commit (from dist\build-info.json), pid; a control.json in its release directory can make it report unhealthy or exit at start.

Evidence block (exit 4)

Failing step; health ok/version/commit/pid or the error; listener PIDs; matching supervisor and proxy PIDs with creation times; inhibit present and its content; task enabled; supervisor.lock held; launcher SHA-256 against launcherSha256Before; paths (not contents) of supervisor.log and state\billion-context\bili.log.


Implementation Plan

Task 1: Preconditions and branch

Output: The fork issue that carries this spec already exists (its number is in the handoff message); screen open and closed issues and PRs once more for overlap, then create a branch <yyyy-mm-dd>_live-deploy-scripts (real date) from origin/master.
Verify: origin/master contains the merge of PR #79 and the deploy section has the wording this spec quotes. If #79 is still open or was merged in a different form: stop and report.
Depends on: PR #79 merged by the owner.

Task 2: Sandbox harness and fixtures

Output: Fixtures and a test helper that builds a sandbox (marker file, launcher, both receipts with the fields the scripts read, config file, a healthy previous release with a node_modules junction, a candidate folder), on an allocated port, and tears it down per AC-22.
Verify: Sandbox starts with one supervisor and one healthy proxy and tears down leaving no process; the harness guards of AC-22 each have a failing-case test; LIVE_FILES and shard-weight entries added (AC-23).
Depends on: Task 1.

Task 3: Common file and deploy script through success

Output: AC-1 to AC-9, AC-12, AC-15 to AC-20.
Verify: The matching tests of AC-21.
Depends on: Task 2.

Task 4: Rollback core and failed rollback

Output: AC-10, AC-11.
Verify: The exit 3 and exit 4 scenarios of AC-21.
Depends on: Task 3.

Task 5: Standalone rollback

Output: AC-13, AC-14.
Verify: One test per row of the AC-13 table, plus the exit 0 and exit 4 scenarios; receipts byte-identical to the backup after exit 0.
Depends on: Task 4.

Task 6: Isolation and hygiene tests

Output: AC-18, AC-19, AC-22, AC-25.
Depends on: Task 5.

Task 7: Documentation (English and Chinese together)

Output, in reference/cindy-project.md and reference/cindy-project.zh-CN.md:

  • Steps 5, 6, 7 and 8: replace the sentences that describe the kept script (fixed version and two-line rewrite; the 90-second wait; "then rewrites the receipts and writes Rollback-MasterDeploy.ps1"; "Rollback-MasterDeploy.ps1 in that cutover's backup restores all three files") with what the repository scripts do. Step 5 keeps "the version reported in the step 1 isolated test" as the source of -CandidateVersion.
  • Tooling paragraph: the deploy and rollback scripts are in scripts/live-deploy/; build and isolated-test tooling is still in the kept tooling directory; the kept Deploy-Master.ps1 and Rollback-MasterDeploy.ps1 are superseded history.
  • Closing paragraph: drop "the kept scripts still depart from the steps…".
  • New short subsection after the steps: one full example invocation of each script, the step-coverage table, the exit-code table.
  • reference/testing.md (+ zh, if the sentence exists there) still says to register spawning tests in scripts/run-ci-tests.mjs; the list is in scripts/live-files.mjs. This is a minor non-semantic flaw: fix it in place.
  • Check whether docs/cindy-project.md (+ zh) and the reference/README.md (+ zh) index rows need a matching touch; update them if a statement there becomes false.
    Verify: 10 steps in each language; every changed sentence has its counterpart; CRLF preserved in files that have it; no user-profile path added; node tools/check-architecture-docs.mjs green.
    Depends on: Task 5.

Task 8: PR

Output: A draft PR to the fork with Closes #<issue>, Assisted-by: Cindy (https://cindy.app) on commits and PR body, a reply on the issue linking the PR (first line (ai: <provider>/<model-id>)); marked ready by the GraphQL markPullRequestReadyForReview mutation once CI is green on the pushed head; the automated review read even when green; findings fixed with additional commits.
Verify: AC-24 on the exact head.
Depends on: Tasks 2–7.


Risks & Mitigations

Risk L I Mitigation
A test or a hand run reaches the live proxy on the same machine L H AC-2.3/2.4 bind port and task to the launcher under the given root; AC-15 full-path matching; AC-16 marker and guard; AC-22 harness guards
The stand-in launcher differs from the real one (accepted by the owner) M M Contract above mirrors the real header and status fields; on a real deployment the real launcher is dry-run twice under 5.1; -PreflightOnly exists
The test driver hides real scheduled-task behaviour, e.g. restart-on-failure relaunching a killed supervisor while disabled (accepted, unverified) M M Inhibit is always set first; zero supervisors and a free lock are re-checked immediately before every lift
Process tests are slow or flaky on CI M M Parameterised waits, polling, allocated ports, shard weight entry
Re-applying the hold fails L H Exit 4 regardless, naming the missing part
The session lacks the right to disable or enable the task (cannot be checked read-only) L M Found at the first hold step, before anything else changed: exit 3 hold-failed with supervision running
Machine paths leak into the repository or CI logs L M AC-17, AC-18, AC-25
First real use is still a real deployment (accepted by the owner) — M Owner-approved deployment only, starting with -PreflightOnly

Failure modes

  • Candidate does not come up → rollback → previous healthy → exit 3.
  • Previous does not come up either, or the rollback throws → hold → exit 4 → owner decides.
  • Script killed → exit 1 or none; the record stage plus Rollback-LiveProxy.ps1 -PreflightOnly say what to do.
  • Receipt replace fails after a good cutover → exit 5; finish from the pending files, or roll back.

Open Questions

  • [NON-BLOCKING, owner] deployedConfigSha256 in tuned-deployment.json no longer describes the live config. The scripts ignore it. Whether to correct it, and when, is the owner's call; it needs a write under the live root.
  • [NON-BLOCKING] Whether Windows restarts a force-stopped supervisor through the task's restart-on-failure setting while the task is disabled. Not verifiable without touching the live task; the mitigation above makes the scripts safe either way. Record what is observed at the next owner-approved deployment.
  • [NON-BLOCKING] Whether a sanitised copy of the real launcher should later replace the stand-in (separate issue, owner's call).

Handoff Notes

Prerequisites

  • PR docs: deploy procedure follows the supervisor-start cutover order; tooling tied to steps (#78) #79 merged.
  • Windows, Node 24, PowerShell 7 (pwsh), Windows PowerShell 5.1 (powershell.exe), gh CLI.
  • Read first: AGENTS.md; reference/cindy-project.md (verify and deploy sections); reference/git-pr.md; reference/testing.md; reference/triage.md; scripts/run-ci-tests.mjs and scripts/live-files.mjs (spawn tripwire, zone-slot preload).
  • Hard rules: never merge; never force-push master; never touch port 8787; the live runtime root may be read, never written or executed; use gh … -R anuzzolilli-oss/billion-context for every GitHub call (this fork routing overrides the upstream REST recipe in reference/git-pr.md); %APPDATA%\… paths only in anything posted; one issue = one scope.

Getting Started

git fetch origin
git switch -c <yyyy-mm-dd>_live-deploy-scripts origin/master
npm ci --ignore-scripts --no-fund --no-audit
npm run build
# once the test file exists:
node --import ./tests/preload-test-env.mjs --import ./tests/preload-storage-baseline.mjs --import tsx --test tests/live-deploy-scripts.test.ts

CI adds preload-zone-slot.mjs; take ports from the mechanism reference/testing.md prescribes for LIVE_FILES entries.

First Action for Executor

Confirm Task 1's precondition. Then read the kept Deploy-Master.ps1 and the real launcher's header (both read-only) next to the deploy section, and build the Task 2 harness with its teardown and guard tests.

Key Files

  • reference/cindy-project.md / .zh-CN.md — the procedure.
  • %APPDATA%\CindyGlobal\billion-context-proxy\backups\before-master-20261114-214204\tooling\Deploy-Master.ps1 — behaviour being replaced (read-only).
  • %APPDATA%\CindyGlobal\billion-context-proxy\Start-BillionContextProxy.ps1 — the contract the stand-in imitates (read-only; copy nothing from it).
  • scripts/run-ci-tests.mjs, scripts/live-files.mjs, scripts/ci-shard-weights.json, scripts/test-suite-lock.mjs — test discovery and registration.
  • src/version.ts, src/server/admin.ts — where the health version and commit come from.

Reviewer Roster

Reviewer Status File
Correctness COMPLETE: 5 blocking; round 2 found 3 resolved, 2 partly; addressed in v3 reviews/correctness.md
Completeness COMPLETE: 8 blocking; round 2 found 5 resolved, 3 partly; addressed in v3 reviews/completeness.md
Security COMPLETE: 5 blocking; round 2 found 2 resolved, 3 partly; addressed in v3 reviews/security.md
Ergonomics COMPLETE: 6 blocking; round 2 found 2 resolved, 4 partly; addressed in v3 reviews/ergonomics.md
Second round COMPLETE: verdict on v2 "not ready"; 5 new blocking, 12 new important; addressed in v3, not re-verified reviews/round2.md

Second round (one reviewer over all four reviews): 11 of 24 first-round blocking items confirmed resolved, 13 partly; 5 new blocking and 12 new important findings. All are addressed in v3 in the author's judgement. The review cap was reached, so v3 and v4 were not sent for a third round: the owner's read is the remaining check.

Deferred with reason: a default -ReleaseName (kept explicit so the operator names what is deployed); restricting -ToolingDir to an allowlist of names (top-level files only is enough); ACL notes for the backup folder (inherits, nothing changed).


Revision History

Date Version Author Notes
2026-11-17 v1 Cindy (AI), for the owner Initial draft from the interview
2026-11-17 v2 Cindy (AI), for the owner Resolved all blocking review findings: node_modules link, release-directory lifecycle, receipts contract, supervision proof in the success test, hold window and exit codes, record-driven rollback preconditions, inhibit ownership, candidate validation, sandbox seam, CI registration, task-driver contract, step coverage
2026-11-17 v3 Cindy (AI), for the owner Second-round fixes: holding stage and reachedStage; failed inhibit creation is exit 6; rollback usable under a hold; trap ends when the candidate is healthy; foreign launcher edit never overwritten; hold-failed path; lock probed once; fault-injection vocabulary; guard tested as a pure function; tooling folder allowed from an older backup
2026-11-17 v4 Cindy (AI), for the owner Final read: expected config hash becomes an operator parameter (owner decision; the receipt value is stale); aborted stage for exit 6; status set to ready for review

Activity

  1. anuzzolilli-oss commented on Nov 18, 2026

    @anuzzolilli-oss
    OwnerAuthor

    (ai: anthropic/claude-fable-5-2)

    Implementation is up as draft PR #81: the two scripts under scripts/live-deploy/, a sandbox test (19 tests, Windows only) and the documentation in both languages. Nothing was run against the live proxy. The PR description lists five departures from the spec above; the one that changes an acceptance criterion is that the standalone rollback no longer requires the task to be enabled.

  2. anuzzolilli-oss commented on Nov 18, 2026

    @anuzzolilli-oss
    OwnerAuthor

    (ai: anthropic/claude-fable-5-2)

    Correction to my earlier comment, which is out of date. PR #81 now has 28 sandbox tests, CI is green on head 6ac461eea, and the automated reviewer's fourth pass reports no issues after three rounds of fixes.

    The change to acceptance criterion 13 is narrower than I first wrote: the standalone rollback still refuses a disabled task, except when this deployment's own hold is in place or its record stopped inside the hold window. That exception needs the owner's decision. The PR description lists it together with the other additions that came out of the review.

  3. added a commit that references this issue on Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions