diff --git a/.agents/skills/uzi-lander/references/renovate.md b/.agents/skills/uzi-lander/references/renovate.md index cdfdf54cca..e5fa41fef7 100644 --- a/.agents/skills/uzi-lander/references/renovate.md +++ b/.agents/skills/uzi-lander/references/renovate.md @@ -105,6 +105,13 @@ new version can break uzi or ship capabilities uzi should adopt. lock-only bump. On your own branch (below), take the lock and run `task codex-pins:sync`, which rewrites every registered Codex version and digest pin from it; keep the Codex guardrail/conformance checks green. +- After the Codex adoption read, hand-update `agent/codex/codex-package.adopted`: + exactly `lock-sha256=<64 lowercase hex>` then `runtime-tag=`, each + followed by LF, with no other lines. Hash the raw lock bytes with + `shasum -a 256 agent/codex/codex-package.lock`; take the tag from that lock. Commit both as + regular files. Pin sync and Renovate must never update this assertion. A lock + edit invalidates it; review the edit before recording adoption again. CI validates + the committed assertion, not whether the read occurred (ADR-3512, #4172). - Required PR CI covers the agent suite, the real-binary `test:codex-m4` on amd64 and the worker-uid lane (`test-agent`). The required `codex-packaged-proof-gate` covers the strict packaged proof on amd64 and the arm64 package check (`e2e/codex-proof/package-exec.sh`). The amd64 image builds run but are not required. diff --git a/.agents/skills/uzi-release/SKILL.md b/.agents/skills/uzi-release/SKILL.md index ff8f39bdd8..21c2f9d89c 100644 --- a/.agents/skills/uzi-release/SKILL.md +++ b/.agents/skills/uzi-release/SKILL.md @@ -23,7 +23,7 @@ Do NOT bake step details that live in the repo into your head. Read the source o - A green CI `report` confirms incident delivery only; inspect the failed phase, planned tag and uncited identities, then fix the cause. Prepare/watch failure still makes the run red. - An unconfirmed push receipt stops watch before publication recovery or verification. Inspect remote refs, then run local watch/verify against the same tag/SHA; this branch does not run automatic publication retries. - If promotion succeeds but diagnostics cannot be written, the cutter exits nonzero and retains its local stable tag. CI does not push; inspect local and remote refs before recovery. -- Before local fallback, finish or stop any CI release run, inspect remote main/tags and resolve interrupted publication on the same tag. No local/CI race. A Codex lock change blocks CI planning; perform the adoption read and cut locally. On abandon the range starts at the highest stable, so an already adopted change can still require local fallback. +- Before local fallback, finish or stop any CI release run, inspect remote main/tags and resolve interrupted publication on the same tag. No local/CI race. A Codex lock change without a valid committed adoption record blocks CI planning; perform the adoption read and commit the record, or cut locally. Abandon uses the highest stable boundary and checks the same HEAD record. - Nightly RC attempts are scheduled for 01:47 Europe/Bucharest only when repository variable `RELEASE_NIGHTLY_ENABLED` is exact `true` (default off). Observed scheduler delays can move cuts into daytime (ADR-3512). The shared plan may no-op; scheduled tip waiting is capped at three hours and the original run date's 13:00 pre-push cutoff. Neither applies to manual dispatch or recovery of a pushed tag. Only the user decides activation. Enablement is rechecked at job boundaries; pausing may stop a not-yet-pushed attempt there, possibly red with `nightly-disabled`. Cancel an in-flight run explicitly for an immediate stop; pushed-tag recovery is unaffected. The bounded attempt journal reports release-commit CI without success polling or reruns; tagged-smoke and targeted assertion recovery use the bounded exception above. - NEVER `docker compose -p uzi down -v` and never glob `uzi-` containers (see `CLAUDE.md` "Destructive operations"). This skill touches git, `gh`, and (if you have deploy access) `kubectl` only. - Work on `main` in the repo root worktree; `release-cut.sh` makes its own throwaway worktree for a promote. Never check the root worktree out onto another branch. @@ -36,7 +36,25 @@ Do NOT bake step details that live in the repo into your head. Read the source o **Artifact membership.** Query the public OCI registry anonymously; no Packages PAT is needed. GitHub release and Actions evidence still uses `gh` read access. Pass `--ci` to the cutter in automation (also enabled by `CI=true`): an unknown worker-pin query or checkout identity refuses the cut. Local cuts retain a warning on query errors; a confirmed missing worker image always refuses. -`release-cut.sh --plan [--version X.Y.Z] [--abandon-train]` reports the scheduled RC decision without fetching or changing Git state, CHANGELOG or fragments. Shared validators remove temporary snapshots but may leave an empty `.uzi/scratch` directory. Refresh refs first. Read `OP` (`nextrc`, `new-base`, `noop`, `blocked`), `BASE`, `TAG`, `REASON`, `SHIPPING_COUNT`, `PREVIOUS_TAG`, `PUBLICATION_STATE`, and `HEAD_SHA`; exit 0 includes no-op/blocked plans and some instrument failures (API or fragment reads); exit 2 is reserved for Git/history instruments, exit 3 for usage. An open train continues, otherwise 0.x defaults to the next minor; automatic 1.x needs a policy. A Codex runtime lock change in the plan range blocks with `codex-runtime-needs-local-cut`: perform the adoption read above and cut locally. Failed, pending, missing or unreadable previous publication blocks even a no-op. Only non-metadata shipping changes count, and notes or dependency citations must make them releasable. This recommendation does not remove manual empty-delta cuts or replace exact-tip CI checks. +`release-cut.sh --plan [--version X.Y.Z] [--abandon-train]` reports the scheduled RC decision without fetching or changing Git state, CHANGELOG or fragments. Shared validators remove temporary snapshots but may leave an empty `.uzi/scratch` directory. Refresh refs first. Read `OP` (`nextrc`, `new-base`, `noop`, `blocked`), `BASE`, `TAG`, `REASON`, `SHIPPING_COUNT`, `PREVIOUS_TAG`, `PUBLICATION_STATE`, and `HEAD_SHA`; exit 0 includes no-op/blocked plans and some instrument failures (API or fragment reads); exit 2 is reserved for Git/history instruments, exit 3 for usage. An open train continues, otherwise 0.x defaults to the next minor; automatic 1.x needs a policy. A Codex runtime lock change in the plan range requires the committed adoption record below; a missing or invalid record blocks with `codex-runtime-needs-local-cut`. Perform the adoption read and commit the record, or cut locally. Failed, pending, missing or unreadable previous publication blocks even a no-op. Only non-metadata shipping changes count, and notes or dependency citations must make them releasable. This recommendation does not remove manual empty-delta cuts or replace exact-tip CI checks. + +### Committed Codex adoption (#4172) + +After reading the runtime release notes and reviewing its manifest, hand-update +`agent/codex/codex-package.adopted`. Use exactly two LF-terminated lines, no extras: + +```text +lock-sha256=SHA256_OF_RAW_LOCK_BYTES +runtime-tag=CODEX_TAG_FROM_LOCK +``` + +Use `shasum -a 256 agent/codex/codex-package.lock` for the 64 lowercase hex digest; +copy the lock's `CODEX_TAG` value for the second line. Commit both as mode 100644. +Planning validates both blobs at its pinned HEAD, including the digest and tag; +worktree-only files, symlinks and malformed or stale records do not authorize it. +Pin sync and Renovate never write the record. A later lock edit needs a fresh +adoption assertion. This records the maintainer's review, not a review by CI; +publication, exact-tip coverage and release authorization remain separate gates. ## 2. Local fallback: cut inline with the deterministic scripts — do NOT dispatch a subagent Once step 1 holds (nothing wanted left unmerged, `main` green on the tip), cut the release **inline as the lead**, driving three bundled scripts. Do NOT dispatch the `release` subagent for this: a background subagent spawns a watch and then idles, so every wait becomes a manual re-ping (the "agent keeps dying" ping-pong), while the steps that actually need judgment — flake-vs-defect, whether to rerun, authoring the CHANGELOG — are the lead's regardless. `.claude/agents/release.md` stays the MECHANICS reference (what each step proves, the signing traps, deploy verify), but the driver is the lead + scripts, not that agent. diff --git a/.agents/skills/uzi-release/scripts/release-cut.sh b/.agents/skills/uzi-release/scripts/release-cut.sh index 8c89333313..4f49349407 100755 --- a/.agents/skills/uzi-release/scripts/release-cut.sh +++ b/.agents/skills/uzi-release/scripts/release-cut.sh @@ -332,6 +332,47 @@ EOF # --plan is deliberately before every fetch, index edit and release mutation. # Scratch snapshots use the cutter's existing bounded validators and clean traps. +# The assertion records a maintainer's adoption read, not a review performed by CI. +# Read only regular blobs at the pinned commit; never execute assertion/lock content. +codex_adoption_record_matches() { + python3 -I - "$1" <<'PY' +import hashlib +import re +import subprocess +import sys + + +def blob(path): + entry = subprocess.check_output( + ["git", "ls-tree", "-z", sys.argv[1], "--", path], stderr=subprocess.DEVNULL + ) + metadata, name = entry.removesuffix(b"\0").split(b"\t") + mode, kind, oid = metadata.split(b" ") + if mode != b"100644" or kind != b"blob" or name != path.encode(): + raise ValueError("expected a committed regular file") + return subprocess.check_output( + ["git", "cat-file", "blob", oid.decode("ascii")], stderr=subprocess.DEVNULL + ) + + +try: + lock = blob("agent/codex/codex-package.lock") + record = blob("agent/codex/codex-package.adopted") + fields = re.fullmatch( + rb"lock-sha256=([0-9a-f]{64})\nruntime-tag=(rust-v[0-9]+\.[0-9]+\.[0-9]+)\n", + record, + ) + tags = [line.removeprefix(b"CODEX_TAG=") for line in lock.split(b"\n") + if line.startswith(b"CODEX_TAG=")] + valid = (fields is not None and len(tags) == 1 + and fields[1] == hashlib.sha256(lock).hexdigest().encode("ascii") + and fields[2] == tags[0]) +except (OSError, ValueError, subprocess.SubprocessError): + valid = False +sys.exit(0 if valid else 1) +PY +} + release_plan() ( export GIT_OPTIONAL_LOCKS=0 git() { command git -c core.fsmonitor=false "$@"; } @@ -382,13 +423,13 @@ release_plan() ( fi tag="v$base-rc.1"; op='new-base' fi - # Adoption still needs a local review/cut. Check before the publication refusal, - # which an explicitly authorized supersede may lift in the CI orchestrator. + # Adoption requires a matching committed assertion, or a local cut. Check before + # the publication refusal, which authorized supersede may lift independently. git diff --quiet --no-ext-diff --no-textconv "$previous" "$head" -- agent/codex/codex-package.lock runtime_rc=$? case "$runtime_rc" in 0) : ;; - 1) block_plan codex-runtime-needs-local-cut ;; + 1) codex_adoption_record_matches "$head" || block_plan codex-runtime-needs-local-cut ;; *) return 2 ;; esac # Check the previous publication even for a zero-delta scheduled attempt. diff --git a/.claude/agents/release.md b/.claude/agents/release.md index 7a8e599bf9..1e4b0031b5 100644 --- a/.claude/agents/release.md +++ b/.claude/agents/release.md @@ -60,7 +60,7 @@ NOT modify source code. ## For this repo (uzi) -**Standard releases use `release-auto.yml` for manual dispatch and opt-in nightly RCs** with shared cutter/watch/verify scripts. See [ADR-3512](../../adr/3512-release-automation.md) and `deploy/README.md`. The RC App atomically pushes metadata and an RC tag; stable requires a pinned candidate and human Environment approval. Agents never dispatch `kind=stable` or approve `release-stable`. This agent is a mechanics reference, not a background release driver. Local fallback remains inline with the lead after finishing/stopping CI release work and inspecting remote main/tags. Codex adoption exceptions require the adoption read and a local cut. +**Standard releases use `release-auto.yml` for manual dispatch and opt-in nightly RCs** with shared cutter/watch/verify scripts. See [ADR-3512](../../adr/3512-release-automation.md) and `deploy/README.md`. The RC App atomically pushes metadata and an RC tag; stable requires a pinned candidate and human Environment approval. Agents never dispatch `kind=stable` or approve `release-stable`. This agent is a mechanics reference, not a background release driver. Local fallback remains inline with the lead after finishing/stopping CI release work and inspecting remote main/tags. Codex lock changes require the adoption read and a matching committed adoption record for CI planning; missing or invalid records retain the local-cut refusal (uzi-release, ADR-3512). **RC-first release train (PRD #1265).** A release is cut as a CANDIDATE `vX.Y.Z-rc.1` by default and PROMOTED from that candidate. CI stable promotion stops there; the local --promote verb can couple the next candidate. `release-cut.sh` is a tag-derived state machine: default cuts the next RC (or refuses, exit 3, with the facts when a lower RC is in flight, so the lead asks the user to promote / re-spin / abandon); `--promote` creates the stable `vB` tag LOCALLY from a throwaway `release/B` worktree at the RC commit (metadata-only, a diff allowlist checked before any tag exists) and then cuts the next `vX.Y.Z-rc.1` on main; `--promote-only` (given the in-flight base `B`) runs that promote half alone: `vB` from the RC commit, no next candidate, `main` untouched, one push (the stable tag), for when `main` carries work not wanted in a candidate yet; `--skip-promote` abandons the RC; `--stable` is the old one-step cut, refused while an RC is in flight. An RC tag publishes images + chart at `X.Y.Z-rc.1`, a Release flagged pre-release (never latest), and the opt-in `uzi-cli-rc` Homebrew formula (PRD #1378: `brew.yml` triggers on every `v*` tag and routes an RC to the separate `uzi-cli-rc` formula, so stable users on `uzi-cli` never see a candidate; before #1378 its trigger excluded `v*-*` and cut no RC formula). **Push order on a promote is the STABLE tag, then `main`, then the RC tag** (the RC's `assert-changelog` derives its coverage base from the tags it can see and needs the new stable visible first); `release-cut.sh` prints the three lines in order. A deployment opts into running candidates with the chart range `0.*-0` instead of `0.*` (out of tree; `deploy/README.md`). diff --git a/adr/3512-release-automation.md b/adr/3512-release-automation.md index b682b6b3d3..c938c9087f 100644 --- a/adr/3512-release-automation.md +++ b/adr/3512-release-automation.md @@ -67,10 +67,16 @@ flowchart LR cutter still validates releasability. Supersede lifts only the matched latest failed/missing publication refusal, never pending/unreadable state or another guard. It cannot waive coverage, oracle, metadata, pin or adoption checks. -- Codex lock changes require the adoption read and a local cut. No model or - attestation parsing runs in CI. Abandon uses the highest stable boundary, so - an adopted Codex change in an abandoned train still blocks automatic abandon - until a local cut. This is deliberate fail-closed behavior. +- Codex lock changes require the adoption read. CI accepts a maintainer-reviewed + `agent/codex/codex-package.adopted` assertion binding the raw lock SHA256 and + runtime tag to regular-file blobs at the same pinned HEAD (#4172). Its exact + two-line format is documented in the uzi-release skill. Missing or invalid + assertions retain `codex-runtime-needs-local-cut`; pin sync and Renovate never + certify adoption. This records the maintainer's read, not a read performed by CI. + Abandon still uses the highest stable boundary and validates the same HEAD + assertion. Publication, coverage and authorization gates remain independent. + This supersedes #3524's unconditional local-cut requirement with the + maintainer's acknowledgement; no model, PR-body parsing or new authority is added. - A thin bundle requires its pinned parent to be fetched before verification. Validate digest, exact refs, release SHA, tag object and single parent before minting. No configuration or hooks are imported. A lost push receipt is diff --git a/agent/codex/codex-package.adopted b/agent/codex/codex-package.adopted new file mode 100644 index 0000000000..a17d3033d9 --- /dev/null +++ b/agent/codex/codex-package.adopted @@ -0,0 +1,2 @@ +lock-sha256=0fee0e06f81e2cd2ce19f6f7de121a3a11493ac52655ad60698e05563870a846 +runtime-tag=rust-v0.166.0 diff --git a/agent/test/codex-pins-sync.test.ts b/agent/test/codex-pins-sync.test.ts index ef334078c2..29a10d12da 100644 --- a/agent/test/codex-pins-sync.test.ts +++ b/agent/test/codex-pins-sync.test.ts @@ -1,6 +1,7 @@ import { it } from "node:test"; import assert from "node:assert/strict"; import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; import { copyFileSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, readlinkSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; @@ -172,3 +173,48 @@ it("syncs the small tree and is idempotent", () => { rmSync(dir, { recursive: true, force: true }); } }); + +it("never rewrites a recorded adoption assertion, even malformed text containing a live pin", () => { + const file = "agent/codex/codex-package.adopted"; + // Even an invalid assertion is not sync's to repair or re-certify. + for (const suffix of ["", `binary at ${INSTALL_ROOT}${current.version}/bin/codex\n`]) { + const assertion = `lock-sha256=${"a".repeat(64)}\nruntime-tag=rust-v${current.version}\n${suffix}`; + const dir = smallTree((files) => { files[LOCK] = bumped(realLock); files[file] = assertion; }); + try { + assert.ok(!scannedPinFiles(dir).includes(file), "adoption is excluded from live-pin discovery"); + assert.ok(!syncCodexPins(dir).includes(file)); + assert.equal(readFileSync(join(dir, file), "utf8"), assertion, "sync does not certify adoption"); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + } +}); + +it("a synchronized runtime bump still fails the release adoption guard until hand-recorded", () => { + const file = "agent/codex/codex-package.adopted"; + const assertion = `lock-sha256=${createHash("sha256").update(realLock).digest("hex")}\nruntime-tag=rust-v${current.version}\n`; + const cutter = readFileSync(join(root, ".agents/skills/uzi-release/scripts/release-cut.sh"), "utf8"); + // Exercise the real planner guard without the unrelated release ceremony. + const start = cutter.indexOf("codex_adoption_record_matches() {"); + const end = cutter.indexOf("\n}\n", start); + assert.ok(start >= 0 && end > start); + const guard = `${cutter.slice(start, end + 3)}\ncodex_adoption_record_matches HEAD\n`; + const dir = smallTree((files) => { files[file] = assertion; }); + const commit = () => { git(dir, "add", "-A"); git(dir, "-c", "user.name=Fixture", "-c", "user.email=fixture@example.test", "commit", "-qm", "fixture"); }; + const check = () => spawnSync("bash", ["-s"], { cwd: dir, input: guard, encoding: "utf8" }); + try { + commit(); + assert.equal(check().status, 0, "the original committed assertion authorizes its lock"); + writeFileSync(join(dir, LOCK), bumped(realLock)); + syncCodexPins(dir); + assert.equal(readFileSync(join(dir, file), "utf8"), assertion); + commit(); + assert.equal(check().status, 1, "pin sync cannot authorize the bumped runtime for nightly release"); + const nextLock = readFileSync(join(dir, LOCK)); + writeFileSync(join(dir, file), `lock-sha256=${createHash("sha256").update(nextLock).digest("hex")}\nruntime-tag=rust-v${NEXT.version}\n`); + commit(); + assert.equal(check().status, 0, "the explicit adoption update authorizes the new lock"); + } finally { + rmSync(dir, { recursive: true, force: true }); + } +}); diff --git a/agent/test/codex-pins.ts b/agent/test/codex-pins.ts index 79a021abda..94a3cc9266 100644 --- a/agent/test/codex-pins.ts +++ b/agent/test/codex-pins.ts @@ -22,6 +22,8 @@ const PIN_SCAN_EXCLUDES = [ ":(exclude)e2e/worker-uid/issue-2397-investigation.md", // A recorded frame tape is historical evidence, not a live deployment pin. ":(exclude)agent/test/fixtures/codex-plan-output-schema-0.164.0.json", + // A maintainer's adoption assertion must never be certified by pin sync. + ":(exclude)agent/codex/codex-package.adopted", // Synthetic version output deliberately includes mismatches to test the verifier. ":(exclude)e2e/codex-m4/provision.test.ts", ]; diff --git a/api/internal/provisionsvc/provisionsvc.go b/api/internal/provisionsvc/provisionsvc.go index e1bf1b4320..4448cfa02f 100644 --- a/api/internal/provisionsvc/provisionsvc.go +++ b/api/internal/provisionsvc/provisionsvc.go @@ -211,14 +211,16 @@ func (s *Service) validateConnection(ctx context.Context, forgeType, baseURL, to if err != nil { return validatedConnection{}, &Error{Kind: KindBadRequest, Message: "could not initialize forge client"} } - identity, err := f.VerifyToken(ctx) + // PRD #4173 D3: connection validation uses the code-host half only. + var ch forge.CodeHost = f + identity, err := ch.VerifyToken(ctx) if err != nil { // err is already PAT-redacted by the driver. return validatedConnection{}, &Error{Kind: KindBadGateway, Message: "token verification failed: " + err.Error()} } // Least-privilege gate (PRD #5): token-level violations block the save. - if tr := s.guard.CheckToken(ctx, f, forge.Type(forgeType), identity.IsAdmin); len(tr.Violations) > 0 { + if tr := s.guard.CheckToken(ctx, ch, forge.Type(forgeType), identity.IsAdmin); len(tr.Violations) > 0 { return validatedConnection{}, &Error{ Kind: KindOverPrivileged, Message: "the bot token is over-privileged and was not saved; mint a least-privilege token (see the bot setup doc)", @@ -246,7 +248,9 @@ func (s *Service) DiscoverRepos(ctx context.Context, userID, connID uuid.UUID) ( s.logger().Error("build forge for connection", "error", err) return store.ForgeConnection{}, internal() } - projects, err := f.ListProjects(ctx) + // PRD #4173 D3: discovery lists projects on the code-host half. + var ch forge.CodeHost = f + projects, err := ch.ListProjects(ctx) if err != nil { return store.ForgeConnection{}, &Error{Kind: KindBadGateway, Message: "could not list projects: " + err.Error()} } @@ -306,7 +310,9 @@ func (s *Service) EnableRepo(ctx context.Context, userID, repoID uuid.UUID) (sto s.logger().Error("build forge for connection", "error", err) return store.Repo{}, internal() } - if err := forgesvc.EnsureBoardColumns(ctx, s.q, f, row.ID, row.ForgeProjectID); err != nil { + // PRD #4173 D3: board columns are labels, on the issue-tracker half. + var tr forge.IssueTracker = f + if err := forgesvc.EnsureBoardColumns(ctx, s.q, tr, row.ID, row.ForgeProjectID); err != nil { var seedErr *forgesvc.LabelSeedError if errors.As(err, &seedErr) { return store.Repo{}, &Error{Kind: KindBadGateway, Message: "could not seed board labels on the forge: " + seedErr.Error()} diff --git a/api/internal/seed/seed.go b/api/internal/seed/seed.go index ddf89302e3..354cd39a10 100644 --- a/api/internal/seed/seed.go +++ b/api/internal/seed/seed.go @@ -99,13 +99,17 @@ func ForgeConnection(ctx context.Context, q Store, svc ForgeService, cfg config. slog.Error("seed forge: build forge client failed; skipping seed", "error", err) return nil } - identity, err := f.VerifyToken(ctx) + // PRD #4173 D3: the token and project reads use the code-host half, the + // board columns below the issue-tracker half. + var ch forge.CodeHost = f + var tr forge.IssueTracker = f + identity, err := ch.VerifyToken(ctx) if err != nil { // err is already PAT-redacted by the driver. slog.Error("seed forge: token verification failed; skipping seed (retries next boot)", "error", err) return nil } - projects, err := f.ListProjects(ctx) + projects, err := ch.ListProjects(ctx) if err != nil { slog.Error("seed forge: listing projects failed; skipping seed (retries next boot)", "error", err) return nil @@ -166,7 +170,7 @@ func ForgeConnection(ctx context.Context, q Store, svc ForgeService, cfg config. // and boot continues, like every other seed forge call; the operator // fixes the forge access and reruns /server setup. A database error is // boot-fatal, like every other seed database error. - if err := forgesvc.EnsureBoardColumns(ctx, q, f, repo.ID, repo.ForgeProjectID); err != nil { + if err := forgesvc.EnsureBoardColumns(ctx, q, tr, repo.ID, repo.ForgeProjectID); err != nil { var seedErr *forgesvc.LabelSeedError if !errors.As(err, &seedErr) { return fmt.Errorf("seed forge: board columns for repo %q: %w", path, err) diff --git a/changelog.d/b01bf006-734a-4b2a-9f96-01de70a73083.md b/changelog.d/b01bf006-734a-4b2a-9f96-01de70a73083.md new file mode 100644 index 0000000000..f7f4f32c15 --- /dev/null +++ b/changelog.d/b01bf006-734a-4b2a-9f96-01de70a73083.md @@ -0,0 +1,4 @@ +### Fixed + +- **Nightly Codex runtime adoption (#4172).** + Scheduled RC planning accepts a committed maintainer adoption record matching the reviewed runtime lock, while retaining publication and coverage checks and refusing missing or stale records. diff --git a/prds/4173-external-issue-trackers.md b/prds/4173-external-issue-trackers.md index f0ad3c86ed..b95ec67bd3 100644 --- a/prds/4173-external-issue-trackers.md +++ b/prds/4173-external-issue-trackers.md @@ -1,9 +1,13 @@ # PRD #4173: External issue trackers: `IssueTracker` and `CodeHost` interfaces, with YouTrack as the first external issue tracker -**Issue:** #4173 -**Priority:** Medium -**Status:** In progress, r3 2026-11-11. M1 and M2 delivered. r3 follows the maintainer's scope review: one real issue tracker instead of abstract identity work, and the comment, command and withdrawal behaviours leave this PRD. -**Delivery:** one PR per milestone, each merged before the next starts. No implementation or validation step touches `.github/workflows/**`. +**Issue:** #4173 + +**Priority:** Medium + +**Status:** In progress, r5 2026-11-11. M1 and M2 delivered. r3 followed the maintainer's scope review: one real issue tracker instead of abstract identity work, and the comment, command and withdrawal behaviours leave this PRD. r4 follows the maintainer's review of the M1 PR: the YouTrack id, tag and author decisions are made before the driver, and the issue source of a repo no longer changes. + +**Delivery:** one PR per milestone, each merged before the next starts. No implementation or validation step touches `.github/workflows/**`. + **Builds on:** - **The forge layer** (`api/internal/forge`): three drivers (GitLab, `adr/0065-forgejo-driver.md`, `adr/0238-github-driver.md`) and the shared fake `forgetest.BaseFake`. @@ -46,24 +50,28 @@ These hold in every milestone. A change that weakens one is out of scope for thi - **Jira and Asana.** One PRD each, after YouTrack. - **How the worker clones, pushes and opens the merge request.** The code host is unchanged. -## Current state (anchors from r1, verified on upstream `main` 6a76d09f7; line numbers drift, cite by symbol) - -- **Where the issue side is used today.** Every place in uzi's code, tests excluded, that calls one of the eleven issue-side methods through the `Forge` interface (47 in all; calls a driver makes inside itself are not counted), listed from the type-checked code (`golang.org/x/tools/go/packages`) and cross-checked with `gopls call_hierarchy`: - - *Sync:* `forgesvc` `FullSync`, `IncrementalSync` and `reconcileFindingGroupMarkers` (`ListIssues`, `api/internal/forgesvc/service.go:787`, `:795`, `:816`, `:911`, `:915`, `:926`, `:448`). - - *Board:* `AutoMove` and `SetIssueLabel` (`EnsureLabels`, `UpdateIssueLabels`, `forgesvc/service.go:534-542`, `:600-607`); column setup and label checks in the handlers (`EnsureLabels`, `ListLabels`, `api/internal/handler/board.go:366`, `:432`, `:760`, `handler/labels.go:36`, `:120`); close and reopen (`SetIssueState`, `forgesvc/close_reopen.go:29`, `:62`). - - *Run creation:* the issue's text and comments (`GetIssue`, `ListIssueComments`, `api/internal/issueinput/projection.go:158`, `:169`), reached from `workersvc` `CreateRun`, the scheduler (`api/internal/schedsvc/scheduler.go:594`, `:806`), autopilot (`api/internal/poller/autopilot.go:209`) and the composite flow (`api/internal/workersvc/composite.go:212`); a task continuing an issue (`GetIssue` through the helper `forge.GetIssueBounded`, `forge/response_limit.go:114`, called from `workersvc/task.go:100`); semantic drift (`GetIssue`, `workersvc/semantic_drift.go:75`). - - *The agent:* the worker reads issues through the API, `GET /api/worker/runs/{id}/forge/issues`, `…/issues/{iid}` and `…/issues/{iid}/label-events` (`ListIssues`, `GetIssue`, `ListIssueComments`, `ListIssueLabelEvents`, `handler/worker_forge.go:138`, `:183`, `:265`, `:310`). - - *The issue page:* `GetIssueDetail` (`GetIssue`, `handler/issues.go:216`). - - *Issues uzi creates:* the board's new issue (`handler/issues.go:65`), findings and finding groups, which also ensure their labels first (`EnsureLabels`, `handler/findings_file.go:163`, `:169`, `handler/findings_group_file.go:203`, `:227`), the judge's admin issue (`handler/judge_admin_issue.go:269`), review issues (`handler/review_issue_file.go:159`), the self-improve tracking issue (`schedsvc/self_improve.go:269`) and proposals (`workersvc/composite.go:150`, `workersvc/proposal_filing.go:175`), all through `CreateIssue`; the scheduler's proposal digest and self-improve lookup read issues (`ListIssues`, `schedsvc/scheduler.go:1426`, `schedsvc/self_improve.go:252`). - - *Issue description:* the PRD link patch (`GetIssue`, `UpdateIssueDescription`, `forgesvc/prd_link_patch.go:159`, `:180`). - - *Autopilot:* label events (`ListIssueLabelEvents`, `poller/autopilot.go:140`). - - *Comments out:* see below. -- **The privilege check is code-host only.** `VerifyToken`, `ProjectRole`, `DefaultBranchProtection` (`api/internal/privcheck/checker.go:77`, `:139`, `:147`; `privcheck/service.go:256-283` at claim), fail-closed. -- **Mixed callers.** Comment intake reads issue comments (`ListIssueComments`) and checks each author against the repository (`RepositoryAuthorEligibility`, `api/internal/issueinput/assessment.go:86`); merge request watchers read the MR from the code host (`forgesvc/mr_watch.go:60`) and move the card on the issue side (`AutoMove`, `mr_watch.go:227`). +## Current state (anchors refreshed in M2's PR on upstream `main` f19bc0739 with M2 applied; cited by file and symbol, because line numbers drift) + +- **Where the issue side is used.** Every place in uzi's code, tests excluded, that calls one of the eleven issue-side methods (45 calls, all through `IssueTracker` since M2; calls a driver makes inside itself are not counted), listed from the type-checked code (`golang.org/x/tools/go/packages`). r1 counted 47 through `Forge` on an older `main`. + - *Sync:* `FullSync`, `IncrementalSync` and `reconcileFindingGroupMarkers` (`ListIssues`, seven calls, `api/internal/forgesvc/service.go`). + - *Board:* `AutoMove` and `SetIssueLabel` (`EnsureLabels`, `UpdateIssueLabels`, `forgesvc/service.go`); column setup and label checks in the handlers (`EnsureLabels` in `ConfigureColumns` and `ensureHumanReviewColumn`, `api/internal/handler/board.go`, and in `ensureRepoLabels`, `handler/labels.go`; `ListLabels` in `CheckRepoLabels`, `handler/labels.go`); close and reopen (`SetIssueState` in `CloseIssue` and `ReopenIssue`, `forgesvc/close_reopen.go`). + - *Run creation:* the issue's text and comments (`GetIssue`, `ListIssueComments` in `FetchWithTimeout`, `api/internal/issueinput/projection.go`), reached from `workersvc` `CreateRun`, the scheduler, autopilot and the composite flow; a task continuing an issue (`GetIssue` through the helper `forge.GetIssueBounded`); semantic drift (`GetIssue` in `readCurrentSemanticIssue`, `api/internal/workersvc/semantic_drift.go`). + - *The agent:* the worker reads issues through the API, `GET /api/worker/runs/{id}/forge/issues`, `…/issues/{iid}` and `…/issues/{iid}/label-events` (`ListIssues` in `WorkerForgeListIssues`, `GetIssue` and `ListIssueComments` in `WorkerForgeGetIssue`, `ListIssueLabelEvents` in `WorkerForgeListIssueLabelEvents`, `handler/worker_forge.go`). + - *The issue page:* `GetIssueDetail` (`GetIssue`, `handler/issues.go`). + - *Issues uzi creates* (`CreateIssue`, eight calls): the board's new issue (`CreateIssue`, `handler/issues.go`), findings and finding groups, which also ensure their labels first (`FileFinding`, `handler/findings_file.go`; `FileFindingGroup`, `handler/findings_group_file.go`), the judge's admin issue (`AdminFileJudgeIssue`, `handler/judge_admin_issue.go`), review issues (`FileIssue`, `handler/review_issue_file.go`), the self-improve tracking issue (`ensureTrackingIssue`, `api/internal/schedsvc/self_improve.go`) and proposals (`ConfirmProposalForUser`, `workersvc/composite.go`; `maybeFileProposal`, `workersvc/proposal_filing.go`). The scheduler's proposal digest and the self-improve lookup read issues (`ListIssues` in `proposalDigestSection`, `schedsvc/scheduler.go`, and in `ensureTrackingIssue`). + - *Issue description:* the PRD link patch (`GetIssue`, `UpdateIssueDescription` in `patchOnePRDLink`, `forgesvc/prd_link_patch.go`). + - *Autopilot:* label events (`ListIssueLabelEvents` in `detectOne`, `api/internal/poller/autopilot.go`). + - *Comments out* (`CreateIssueNote`, five calls, all discarding the returned note): autopilot's refusal comment (`recordThenComment`, `poller/autopilot.go`), CI autofix start and halt comments (`detectOne`, `poller/ci_autofix.go`), the merge request rework halt comment (`detectOne`, `poller/mr_review_watch.go`) and the terminal comment (`maybeTerminalComment`, `api/internal/runlifecycle/lifecycle.go`). +- **No call goes through the union.** The same inventory shows 66 code-host calls, all through `CodeHost`, and none of either kind through `forge.Forge`. The connection factories still return `forge.Forge`; each caller narrows the value before its first call. +- **The privilege check is code-host only.** `VerifyToken`, `ProjectRole`, `DefaultBranchProtection` (`Check`, `CheckToken` and `checkRepo`, `api/internal/privcheck/checker.go`, all taking `forge.CodeHost`; `privcheck/service.go` at claim), fail-closed. +- **Mixed callers.** Comment intake reads issue comments (`ListIssueComments`) and checks each author against the repository (`RepositoryAuthorEligibility`, in `Assessment.Author`, `api/internal/issueinput/assessment.go`); merge request watchers read the MR from the code host and move the card on the issue side (`AutoMove`), both in `forgesvc/mr_watch.go`. - **Issue identity is a number.** `issues.forge_issue_iid bigint` (`api/internal/store/migrations/00002_forge.sql:47-50`), `runs.issue_iid`, and one active run per issue on `(repo_id, coalesce(issue_iid, continued_issue_iid))` for issue runs and tasks continuing an issue (`00351_task_continued_issue.sql:11-14`). -- **Board columns are labels.** `board_columns.label_name` (`00002_forge.sql:39`); moving a card swaps labels through `UpdateIssueLabels` (`api/internal/forgesvc/service.go:512-542`), forge-first, with the `move_pending_since` marker (`api/internal/runlifecycle/lifecycle.go:1-17`). -- **Autopilot.** Maps the label adder to a uzi user through `forge_connections.human_username` (`00037_autopilot_mapping.sql:6-20`, `poller/autopilot.go:194`). +- **Board columns are labels.** `board_columns.label_name` (`00002_forge.sql:39`); moving a card swaps labels through `UpdateIssueLabels` (`AutoMove`, `api/internal/forgesvc/service.go`), forge-first, with the `move_pending_since` marker (`api/internal/runlifecycle/lifecycle.go`). +- **Autopilot.** Maps the label adder to a uzi user through `forge_connections.human_username` (`00037_autopilot_mapping.sql:6-20`, `detectOne` in `poller/autopilot.go`). - **Constraints a text key meets.** `issues.forge_issue_iid` is `bigint NOT NULL` with `UNIQUE (repo_id, forge_issue_iid)` (`00002_forge.sql:50`, `:59`), and `runs_kind_shape` requires `issue_iid IS NOT NULL` for `kind = 'issue'` (`00314_cross_check_claude_checker.sql`). An added `issue_key` column alone does not make an issue tracker issue storable. +- **The issue author is checked the same way.** Run creation assesses `Issue.AuthorForgeUserID` through `RepositoryAuthorEligibility` (`Assessment.Author`, `api/internal/issueinput/assessment.go`). An author that cannot be assessed gives the reason `permission_unknown`, and a scheduled start with that reason is not auto-approved (`createRun`, `api/internal/workersvc/service.go`). +- **The forge allowlist is deployment configuration.** `FORGE_ALLOWED_BASE_URLS` is rendered from the Helm value `forge.allowedBaseURLs` (`deploy/chart/values.yaml`, `deploy/chart/templates/api-configmap.yaml`) and passed through `docker-compose.yml`. +- **Disabling a repo drains nothing.** It flips `repos.enabled`; a sync or a run already in flight finishes. - **Comment authors are checked against the repository.** Comment intake keeps a comment only when its author has repository access (`RepositoryAuthorEligibility`, a `CodeHost` method taking a forge user ID). An issue tracker comment's author is an issue tracker account, which that check cannot evaluate. ## Decisions @@ -73,18 +81,44 @@ These hold in every milestone. A change that weakens one is out of scope for thi - **D3. Narrowest interface per caller.** A caller that needs both halves takes two parameters (`ch forge.CodeHost, tr forge.IssueTracker`), never the union. - **D4. What enforces the split.** The interface declarations do: a method belongs to the half that declares it, and passing an `IssueTracker` where a `CodeHost` is required does not compile. The per-driver `var _ IssueTracker` / `var _ CodeHost` lines are readable markers only; implementing `Forge` already implies them. No reflection test in M1 or M2. - **D5. A repo resolves its two sides.** `IssueTrackerForRepo` and `CodeHostForRepo` (`forgesvc`). Through M2 both return the repo's forge driver. From M4, `IssueTrackerForRepo` returns the linked issue tracker's driver. `CodeHostForRepo` always returns the forge driver (S2). -- **D6. Issue tracker drivers live in their own package**, `api/internal/issuetracker`, not in `forge`. A driver is `forge.IssueTracker` plus what a connection needs: verify the token, list projects, list states. A new issue tracker is one kind constant, one registry entry, one driver and its fake. -- **D7. Every driver ships with a fake and contract tests.** The fake follows `forgetest.BaseFake` (loud `ErrNotStubbed` defaults). One contract suite runs against every `IssueTracker` implementation, the three forge drivers' fakes included, so parity is tested, not asserted. +- **D6. Issue tracker drivers live in their own package**, `api/internal/issuetracker`, not in `forge`. A driver is `forge.IssueTracker` plus what a connection needs: verify the token, list projects, list tags and, for a named project, list its states. A new issue tracker is one kind constant, one registry entry, one driver and its fake. +- **D7. Every driver ships with a fake and contract tests, and the contract suite runs against the real drivers.** One contract suite covers every `IssueTracker` implementation. Each one is built by its production constructor and talks HTTP to a fake server: the YouTrack driver to an in-process HTTPS fake, the three forge drivers to `e2e/forge-fake`, as `api/internal/forge/fake_contract_test.go` already does for forge operations. An in-memory fake in the style of `forgetest.BaseFake` (loud `ErrNotStubbed` defaults) also ships, for the tests of callers; passing the suite on it is not evidence of parity. - **D8. Issue tracker connections are separate from forge connections.** Per user, kind and base URL, with the rules in S3. A repo links to one connection and one issue tracker project. NULL means the forge is the issue source. -- **D9. Moving a card is an `IssueTracker` method.** `MoveToColumn`. The forge drivers implement it with one shared helper that does today's label swap in one `UpdateIssueLabels` call. YouTrack maps a column to a value of the project's State field. The column-to-state mapping is a side table, so relinking a repo never has to clear a column other code reads. -- **D10. Issue identity is designed in M4, against YouTrack.** The candidate, taken from a working prototype in our fork: the numeric identity is YouTrack's `numberInProject`, which satisfies the `NOT NULL`, uniqueness and `runs_kind_shape` constraints above with no schema change to them; a nullable `issue_key text` carries the readable id (`ABC-12`) for display and references, NULL for forge issues. An issue moved to another YouTrack project gets a new number, which is treated as the issue leaving the repo. The decision is recorded in the Decision Log in M4, before code. -- **D11. Issue tracker comments are withheld until their author is proven.** An issue tracker comment reaches a run only when its author maps to a uzi user who owns or can access the repo. Until that mapping exists, issue tracker comments are withheld exactly like a comment whose author lookup failed. How an issue tracker account is proven to belong to a uzi user is designed in M4 and recorded in the Decision Log before code; the fail-closed default holds until then. +- **D9. Moving a card is an `IssueTracker` method.** `MoveToColumn`. The forge drivers implement it with one shared helper that does today's label swap in one `UpdateIssueLabels` call. YouTrack maps a column to a value of the project's State field. The column-to-state mapping is a side table, so `board_columns.label_name` keeps its meaning for the code that reads it. +- **D10. YouTrack ids are adapted inside the driver; the neutral types do not change.** `IssueTracker` takes `projectID int64` and `issueIID int64` and returns `IssueNote.ID int64`, while YouTrack project, comment and account ids are strings. Changing the neutral types would cost the three forge drivers and every caller and give the forges nothing, so the driver adapts, and M3 implements exactly this: + - *Project.* A YouTrack driver is built for one project. It holds that project's string id and short name, and the one `projectID int64` value it answers to: the repo's forge project id, which is the number every caller already passes for that repo. An `IssueTracker` method called with a different `projectID` is refused before any request. This catches mismatched forge project numbers; the owner-scoped repo resolver must also bind the driver to the correct repo and issue tracker connection, including when two connections expose the same forge project number. M4 tests that collision. The `IssueTracker` methods of a driver built without a project return an error. + - *Connection operations.* Verify the token, list projects and list tags need no project, so a connection can be set up before one is chosen. Listing states is project-scoped, because states belong to a project: it takes the project it reads, picked from the list of projects, and cannot be called without one. + - *Issue.* `issueIID` is YouTrack's `numberInProject`. It satisfies the `NOT NULL`, uniqueness and `runs_kind_shape` constraints above with no change to them. + - *Comment.* The driver does not derive a number from a YouTrack comment id. YouTrack documents these ids as strings and guarantees no grammar for them. `CreateIssueNote` returns `IssueNote.ID` 0 on YouTrack, and no caller reads that field today: all five call sites discard the returned note. + - *Accounts.* The numeric author id of an issue or comment is 0, which the existing code already reads as unresolved; the login is carried as text. D11 says what an unresolved author may do. + - *Every read and write checks where it landed.* The issue reached must be in the driver's project and carry the requested number; otherwise the driver returns issue-not-found. An issue moved to another YouTrack project gets a new number there, so in the old project it is not found, even where YouTrack still resolves the old readable id. + - *A live run whose issue was moved away.* Its later issue writes (card move, comment, description patch) are skipped and logged. The code-host side of the run (branch, merge request, CI, settle) continues. The next sync removes the issue from the repo's cache. + - *Storage, decided in M4.* A nullable `issue_key text` carries the readable id (`ABC-12`) for display and references, NULL for forge issues. M4 records the column and its constraints in the Decision Log before code. +- **D11. An issue tracker author is unproven until mapped, for comments and for the issue itself.** The forge's `RepositoryAuthorEligibility` is never asked about an issue tracker account. An issue tracker author counts as eligible only when the account maps to a uzi user who owns or can access the repo. Until then: + - *Comments* are withheld from the run, exactly like a comment whose author lookup failed. + - *The issue author* is assessed as `permission_unknown`. A manual start proceeds, because a uzi user chose that issue. A scheduled start is created but never auto-approved: its plan waits for a person. M4 tests the manual and the scheduled start separately. + + How an issue tracker account is proven to belong to a uzi user is designed in M4 and recorded in the Decision Log before code; the fail-closed default holds until then. - **D12. Autopilot is not offered on a repo linked to an issue tracker in this PRD.** It maps the label adder to a uzi user through the forge identity, which an issue tracker account does not have. It stays unchanged on forge-backed repos. +- **D13. A repo's issue source is chosen before the repo is first enabled, and does not change.** Linking is accepted only for a repo that has never been enabled. Such a repo has had no sync and no run, so no stored reference and no operation in flight can meet the change. M4 verifies in code that neither can start on a repo that is not enabled, and refuses the link when anything else already stores an issue number for the repo, such as a schedule pinned to an issue. Relinking and unlinking are out of scope for this PRD, and so is linking a repo that is already in use; moving a repo in use to an issue tracker is left for a later PRD. A repo with no link is unaffected (Goal 5). + - *"Never enabled" is a stored fact, not a reading of `repos.enabled`,* because disabling drains nothing. M4 adds a marker set in the transaction that first enables a repo and never cleared. Its migration marks every repo that exists at that moment as previously enabled, because current rows cannot prove otherwise: a repo that was used, then disabled, can be left with no cached issue, run or schedule. Only a repo created after the migration can be linked. + - *The link and the first enable take the same repo row lock,* so one of them sees the other: either the link lands first and the repo starts on YouTrack, or the enable lands first and the link is refused. + - *An issue tracker connection with a linked repo cannot be deleted.* + - A forge issue 12 and a YouTrack issue 12 can therefore never belong to the same repo. +- **D14. YouTrack tags are not project labels, and the eligibility tag is chosen by id.** A YouTrack tag belongs to the user who created it, is private unless shared, and can share its name with another user's tag. So: + - *Choice.* When a repo is linked (M4), the admin picks the eligibility tag from the tags the connection's token can see. uzi stores the tag's id with the link, and its name for display only. + - *Who must be able to use the tag.* YouTrack separates who can see a tag from who can apply it (`readSharingSettings` and `tagSharingSettings` on the tag). Two parties need the right to apply it: the connection's account, because uzi puts the tag on issues it creates, and the people who start work by tagging an issue. uzi cannot know who those people are, so the admin decides it in YouTrack and confirms it at link time. + - *Verification at link time, by reading only.* uzi reads the tag and its two sharing settings; no issue is modified. The link is refused, with what to change in YouTrack, unless the connection's account can see the tag and is covered by its use settings (it owns the tag, is named in them, or they cover all users), and the use settings cover someone other than the connection's account. The link screen then shows who can see and who can use the tag, and the admin confirms that this is who should start work. + - *Matching.* Intake selects issues by the stored tag id. Another tag with the same name is ignored. + - *Other tags uzi writes* (on issues it creates, such as findings). `EnsureLabels` resolves a name only to a tag owned by the connection's account and creates it there when missing, with the sharing settings of the eligibility tag. It never resolves a name to another user's tag and never creates the eligibility tag. When uzi writes the eligibility label itself, on an issue it creates or moves, that name resolves to the stored tag id, whoever owns the tag, and never to a same-named tag of the connection's account. + - *`ListLabels`* returns the tags the token can see. Board columns are not tags on YouTrack (D9). + - *In the contract suite* the shared rule is "a label is addressed by name within the project the driver serves". The YouTrack subject is seeded with an eligibility tag owned by another user and shared, and a second tag of the same name owned by the connection's account, and the suite shows that reads, `EnsureLabels` and `CreateIssue` only ever use the first. ## Feature parity on a repo linked to an issue tracker -- Bring tagged issues into uzi: the `uzi` tag in the linked project. Milestone M4. +- Bring tagged issues into uzi: the eligibility tag chosen at link time (D14), in the linked project. Milestone M4. - Give the run the issue's title, body and comments: yes, comments subject to D11. M4. +- Auto-approve a scheduled run's plan: only when the issue author is mapped (D11). - Let the agent read an issue through the worker API: yes, same routes. M4. - Show the board and move cards: yes, through the State field (D9). M5. - Close or reopen an issue: yes, through the State field. M5. @@ -93,6 +127,7 @@ These hold in every milestone. A change that weakens one is out of scope for thi - PRD link patch in the issue description: yes. M5. - Eligibility by assigning the issue to the uzi bot: not offered in this PRD; the tag is the only signal. - Autopilot: not offered (D12). +- Change a repo's issue source after it was first enabled: not offered (D13). - Privilege checks, claim and settle, pipeline sync, merge request watchers, clone and push: unchanged, on the forge (S2). ## Milestones @@ -100,11 +135,11 @@ These hold in every milestone. A change that weakens one is out of scope for thi Progress (one PR per milestone, each merged before the next starts): - [x] **M1:** the `IssueTracker` and `CodeHost` interfaces -- [x] **M2:** every caller on its half. Verified in the run: the type-checked inventory shows 0 issue-side calls through `forge.Forge` and 45 through `IssueTracker` (plus `GetIssueBounded` and `EnsureBoardColumns`), and handing `privcheck` an `IssueTracker`-only value fails to compile. `task gate:api` slots green except `deadcode:api` (not run). +- [x] **M2:** every caller on its half. The type-checked inventory shows 45 issue-side calls through `IssueTracker`, 66 code-host calls through `CodeHost` and none through `forge.Forge`, and handing `privcheck` an `IssueTracker`-only value fails to compile. - [ ] **M3:** issue tracker connections and the YouTrack driver - [ ] **M4:** issue identity, the repo link and issue intake from YouTrack - [ ] **M5:** the board and issue writes on YouTrack -- [ ] **M6:** web and CLI surfaces, docs, ADR and live acceptance +- [ ] **M6:** web and CLI surfaces, docs and live acceptance ### M1: the `IssueTracker` and `CodeHost` interfaces @@ -117,21 +152,31 @@ D1, D2 and D4. One file per interface with the types only it uses (`issuetracker D3 and D5. - **Blocked by:** M1. -- **Acceptance:** all 47 issue-side uses go through `IssueTracker`; `privcheck` cannot be handed a value that has only the issue side; existing tests change only in two mechanical ways (a fake's parameter type, and passing the same fake as both halves). +- **Acceptance:** every issue-side use in the refreshed inventory (45 calls, 47 at r1) goes through `IssueTracker`; `privcheck` cannot be handed a value that has only the issue side; existing tests change only in two mechanical ways (a fake's parameter type, and passing the same fake as both halves). ### M3: issue tracker connections and the YouTrack driver -D6, D7, D8 and S3. The `api/internal/issuetracker` package, the YouTrack driver for every `IssueTracker` method plus verify, list projects and list states, its fake, the contract suite, the connection table and the allowlist. No repo can link to an issue tracker yet, so no behaviour changes. +D6, D7, D8, D10 (the id adapter), D14 (the tag rules inside the driver) and S3. The `api/internal/issuetracker` package, the YouTrack driver for every `IssueTracker` method plus verify, list projects, list tags and list a project's states, its fakes, the contract suite, the connection table and the allowlist. No repo can link to an issue tracker yet, so no behaviour changes. +- **Deployment wiring for S3.** The issue tracker allowlist gets a Helm value rendered into the API ConfigMap, with the same https-only validation as `forge.allowedBaseURLs`, and a pass-through in `docker-compose.yml`. It is empty by default in both. It does not feed the worker egress list, because the worker never calls an issue tracker (S5). +- **ADR.** The ADR for the issue tracker seam ships with this milestone, where the package (D6) and the connections (D8) become real: adr/4173-issue-tracker-seam.md. - **Blocked by:** M2. -- **Acceptance:** the contract suite passes on the YouTrack fake and on the three forge fakes; with the allowlist empty no connection can be created and uzi still boots; a redirect, an oversized body and an error carrying the token are each refused or scrubbed in a test. +- **Acceptance:** the contract suite passes on the real YouTrack driver against its HTTPS fake and on the three real forge drivers against `e2e/forge-fake`; the same-name tag case of D14 passes; a call with a `projectID` other than the driver's and an issue found in another project are each an error; the tag's sharing settings are parsed from a response recorded from a real YouTrack; with the allowlist empty no connection can be created and uzi still boots; the chart renders the allowlist and refuses a non-https entry; a redirect, an oversized body and an error carrying the token are each refused or scrubbed in a test. ### M4: issue identity, the repo link and issue intake from YouTrack -D5, D10, D11 and S2. The identity decision (recorded in the Decision Log before code), the repo link to a connection and project, `IssueTrackerForRepo` returning the issue tracker, sync on the poll cadence, run creation from an issue tracker issue. +D5, D10 (storage), D11, D13, D14 (choice and verification of the tag) and S2. The storage decision and the author mapping (both recorded in the Decision Log before code), the never-enabled marker, the repo link to a connection, a project and an eligibility tag, `IssueTrackerForRepo` returning the issue tracker, sync on the poll cadence, run creation from an issue tracker issue. - **Blocked by:** M3. -- **Acceptance:** a tagged YouTrack issue appears on the board and starts a run that receives its title and body; an issue tracker comment from an unmapped author is withheld; the privilege check and claim still run against the forge and fail closed with the issue tracker unreachable; a repo with no link is byte-for-byte unchanged in its forge calls (recorded-call test). +- **Acceptance:** + - a YouTrack issue carrying the eligibility tag appears on the board and starts a run that receives its title and body, and an issue carrying only a same-named tag of another user does not; + - an issue tracker comment from an unmapped author is withheld; + - with an unmapped issue author, a manual start proceeds and a scheduled start is created without auto-approval, each in its own test; + - two repos on different forge connections that expose the same forge project number, one linked to YouTrack and one not, each resolve to their own issue source: a call for one never reaches the other's driver (D10); + - the privilege check and claim still run against the forge and fail closed with the issue tracker unreachable; + - a repo with no link is byte-for-byte unchanged in its forge calls (recorded-call test); + - linking is refused for every repo that existed before the migration and for a repo enabled since, including one that is disabled now; it is refused for a tag the connection's account cannot apply or that nobody else can apply, and the check modifies no issue; a link racing the first enable ends in exactly one of the two outcomes of D13; a connection with a linked repo cannot be deleted; + - when a run's issue is moved to another YouTrack project, its issue writes are skipped and its merge request still completes (D10). ### M5: the board and issue writes on YouTrack @@ -140,24 +185,26 @@ D9. `MoveToColumn` on all four drivers, the column-to-state mapping, close and r - **Blocked by:** M4. - **Acceptance:** the recorded-call test shows identical label calls on the three forges; a card move changes the YouTrack State and a failed move snaps back; closing never lands on a state that means "won't fix". -### M6: web and CLI surfaces, docs, ADR and live acceptance +### M6: web and CLI surfaces, docs and live acceptance -Settings → Issue trackers, the CLI commands, a setup page under `docs/`, an ADR for the issue tracker seam. +Settings → Issue trackers, the CLI commands, a setup page under `docs/`. - **Blocked by:** M5. -- **Acceptance (maintainer-owned, live):** connect, link, tag, run, board move and close verified on a real YouTrack, and a regression pass on GitHub, GitLab and Forgejo. +- **Acceptance (maintainer-owned, live):** connect, link, tag, run, board move and close verified on a real YouTrack from a hosted Kubernetes install, with the allowlist set through the chart, and a regression pass on GitHub, GitLab and Forgejo. ## Testing decisions -- **Every milestone's gate:** `task gate:api`, plus `task gate:web` when web changes, `check:changelog-entry`, and `docs:sync` with `check-docs:web` when docs change. Token-shaped test strings are assembled at runtime. +- **In a uzi run, the in-run boundary of `AGENTS.md` "Run economy" applies:** `task check:api` plus `go test -race -count=1` on the affected packages, with each focused regression test watched red then green, `task check:web` plus the touched tests when web changes, and the companion checks for the files changed (`check:changelog-entry`, the migration checks, `docs:sync` with `check-docs:web`, `check:token-literals`). M3 to M5 touch credentials and untrusted input, so `task scan:secrets` also runs in the run and its canaries-detected line is part of the evidence. Token-shaped test strings are assembled at runtime. The full `gate:*` targets are the PR's CI on the exact head. +- **The E2E harness follows every changed issue or forge call in the same PR** (`AGENTS.md`, "A product contract change updates the E2E harness in the same PR"): the old identifiers are searched across `e2e/phases/`, `e2e/lib.sh`, `e2e/lib.test.sh` and `e2e/forge-fake/`, stale expectations and fake routes are updated, `task test:e2e-lib` and `task test:forge-fake` run, and the PR names the affected phases left unrun. - **No change in behaviour on forge-backed repos:** the existing API, web, agent and E2E suites pass; M2's two mechanical test edits are the only ones allowed without a finding. -- **Parity is the contract suite (D7)**, not a per-driver checklist. +- **Parity is the contract suite on the real drivers (D7)**, not a per-driver checklist and not a run against in-memory fakes. +- **Hosted Kubernetes is verified, not only compose:** the chart test in M3 and the live acceptance in M6. ## Risks -- **Size of M2.** 57 production and 31 test files, mechanical. A type-checked inventory of forge calls was the guard. +- **Size of M2.** 54 production and 31 test files, mechanical. A type-checked inventory of forge calls was the guard. - **Drift from `main` between milestones.** Each milestone's branch is rebased on `main` before its PR. -- **YouTrack specifics leak into the seam.** The contract suite runs on all four implementations, and the second issue tracker (Asana) is the real test of the seam. +- **YouTrack specifics leak into the seam.** The id adapter (D10) and the tag rules (D14) stay inside the driver, the contract suite runs on all four implementations, and the second issue tracker (Asana) is the real test of the seam. ## Decision Log @@ -169,3 +216,10 @@ Settings → Issue trackers, the CLI commands, a setup page under `docs/`, an AD - **2026-11-10: the PRD link patch takes both halves.** D3 placed it on `IssueTracker`, but it also reads the merge request (`GetMergeRequest`), so `SyncPRDLinkPatches` takes `(ch forge.CodeHost, tr forge.IssueTracker)`. Comment intake (`issueinput.Fetch`), the autopilot, CI-autofix and MR-rework detectors, the MR state watcher, self-improve and the worker get-issue route are mixed the same way. - **2026-11-10: "`privcheck` compiles against `CodeHost` only" means every privcheck parameter, local and forge call is `CodeHost`.** Its `ForgeBuilder` factory interface still returns `forge.Forge`, converted to `CodeHost` straight after the build; handing privcheck an `IssueTracker`-only value is a compile error. - **2026-11-10: scope after the maintainer's review.** YouTrack becomes part of this PRD and drives the identity design. The living comment, the outbox, withdrawal, comments to the agent and `/uzi` commands move out. Asana follows in its own PRD. +- **2026-11-11: the issue source is fixed before first enable (D13).** Review of the M1 PR found that project-local numbers let an old run or pinned schedule resolve to an unrelated issue after a repo changes its issue source. The first answer was a per-repo fence over run creation, sync and every issue write. The maintainer proposed the simpler policy: choose the source before the repo is first enabled, and leave relinking out. It holds, because a never-enabled repo has had no sync and no run, provided "never enabled" is stored rather than read from `repos.enabled` and the link and the first enable share the repo row lock. The fence is dropped. +- **2026-11-11: id adapter instead of new neutral types (D10).** The driver is built for one project, `issueIID` is `numberInProject`, the driver answers only to the repo's forge project id, `IssueNote.ID` is not derived and account ids are 0. Decided before the M3 driver; only the `issue_key` column waits for M4. +- **2026-11-11: the eligibility tag is chosen by id (D14)**, because YouTrack tags are user-owned, private by default and not unique by name. +- **2026-11-11: an unmapped issue author is `permission_unknown` (D11).** Manual start proceeds, scheduled start is never auto-approved. +- **2026-11-11: the contract suite runs on the real drivers (D7), the allowlist is wired through the chart and compose in M3, and the ADR moves from M6 to M3.** +- **2026-11-11: second review pass.** Every repo that exists at the M4 migration counts as previously enabled (D13). The driver refuses a `projectID` other than its own, and listing states is project-scoped (D10). The eligibility tag is checked through its see and use settings, read-only (D14). No number is derived from a comment id (D10). +- **2026-11-11: the `projectID` guard is not the whole binding (D10).** `repos` is unique on `(connection_id, forge_project_id)`, so two connections can expose the same forge project number. The guard catches a mismatched number; the owner-scoped repo resolver binds the driver to the right repo and issue tracker connection, and M4 tests the collision. diff --git a/scripts/lib/shipping-paths.sh b/scripts/lib/shipping-paths.sh index 1505ba9708..ab917e48fc 100644 --- a/scripts/lib/shipping-paths.sh +++ b/scripts/lib/shipping-paths.sh @@ -5,6 +5,7 @@ # AGENT_PATHS is the space-separated runtime surface used by autobump's raw Git # pathspec diffs: source, dependency manifests, TypeScript config, binaries, # templates, global devbox toolchain, baked sqlc archive and Codex packaging/supervisor. +# The exact Codex adoption assertion is release review metadata, not execution input. # is_shipping returns 0 for that set (exact entries or slash descendants), # api, controller, web app, chart and docs; 1 for tests, e2e, fixtures and everything # else (including skill/PRD-only changes). Tests are excluded before runtime matching. @@ -16,10 +17,11 @@ # next candidate; skill/PRD-only work with empty [Unreleased] is nonshipping. # - scripts/worker-tag-autobump.sh: which runtime paths warrant rolling workers. # - scripts/check-changelog-entry.sh: which branch changes need a changelog entry. -AGENT_PATHS="agent/src agent/package.json agent/package-lock.json agent/tsconfig.json agent/bin agent/templates agent/devbox-global agent/codex scripts/sqlc.sh" +AGENT_PATHS="agent/src agent/package.json agent/package-lock.json agent/tsconfig.json agent/bin agent/templates agent/devbox-global agent/codex scripts/sqlc.sh :(exclude)agent/codex/codex-package.adopted" is_shipping() { case "$1" in + agent/codex/codex-package.adopted) return 1 ;; *_test.go|*.test.ts|*.test.tsx|*/testdata/*|*/test/*|e2e/*|fixtures/*) return 1 ;; api/*|controller/*|web/src/*|deploy/chart/*|docs/*) return 0 ;; esac diff --git a/scripts/release-scripts-test.sh b/scripts/release-scripts-test.sh index 559f68346d..291a1f6917 100755 --- a/scripts/release-scripts-test.sh +++ b/scripts/release-scripts-test.sh @@ -1072,6 +1072,9 @@ fi if [ "${PLAN_HISTORY_FAILURE:-0}" = 1 ]; then case " $* " in *' log --first-parent '*) exit 74 ;; esac fi +if [ "${PLAN_BLOB_FAILURE:-0}" = 1 ]; then + case " $* " in *' cat-file '*) exit 75 ;; esac +fi exec "$PLAN_REAL_GIT" "$@" GIT chmod +x "$d/plan-bin/git" @@ -1183,11 +1186,62 @@ GIT gcommit "$d" 'docs: isolated fragment' plan_expect 'only committed fragment makes range releasable' $'OP=nextrc\nTAG=v0.2.0-rc.12\nSHIPPING_COUNT=1' mkdir -p "$d/agent/codex" - echo 'runtime pin fixture' > "$d/agent/codex/codex-package.lock" + printf 'CODEX_VERSION=2.0.0\nCODEX_TAG=rust-v2.0.0\n' > "$d/agent/codex/codex-package.lock" git -C "$d" add agent/codex/codex-package.lock gcommit "$d" 'chore: change Codex runtime' plan_expect 'Codex adoption requires a local cut' $'OP=blocked\nREASON=codex-runtime-needs-local-cut' PLAN_PUBLICATION=failed plan_expect 'Codex block precedes liftable publication block' $'OP=blocked\nREASON=codex-runtime-needs-local-cut' + # #4172: an adoption assertion belongs to the committed lock, never a worktree file. + local adoption digest valid_record malformed publication_reason + adoption=agent/codex/codex-package.adopted + digest="$(shasum -a 256 "$d/agent/codex/codex-package.lock" | awk '{print $1}')" + valid_record="$(printf 'lock-sha256=%s\nruntime-tag=rust-v2.0.0' "$digest")" + printf '%s\n' "$valid_record" > "$d/$adoption" + plan_expect 'Codex untracked adoption cannot authorize release' $'OP=blocked\nREASON=codex-runtime-needs-local-cut' + git -C "$d" add "$adoption" + gcommit "$d" 'docs: record Codex adoption' + plan_expect 'Codex matching committed adoption' $'OP=nextrc\nTAG=v0.2.0-rc.12' + PLAN_BLOB_FAILURE=1 plan_expect 'Codex blob read failure stays closed' $'OP=blocked\nREASON=codex-runtime-needs-local-cut' + for PLAN_PUBLICATION in failed pending missing error transient; do + export PLAN_PUBLICATION + publication_reason=previous-publication-not-successful + case "$PLAN_PUBLICATION" in error|transient) publication_reason=previous-publication-unreadable ;; esac + plan_expect "Codex adopted publication $PLAN_PUBLICATION" "$(printf 'OP=blocked\nREASON=%s' "$publication_reason")" + done + export PLAN_PUBLICATION=success + PLAN_PREVIOUS=v0.1.0 plan_expect 'Codex adopted abandon uses stable boundary' $'OP=new-base\nPREVIOUS_TAG=v0.1.0' --version 0.3.0 --abandon-train + for malformed in stale tag duplicate unknown extra uppercase short garbage missing newline crlf; do + case "$malformed" in + stale) printf 'lock-sha256=%064d\nruntime-tag=rust-v2.0.0\n' 0 ;; + tag) printf 'lock-sha256=%s\nruntime-tag=rust-v9.9.9\n' "$digest" ;; + duplicate) printf '%s\nlock-sha256=%s\n' "$valid_record" "$digest" ;; + unknown) printf '%s\nunknown=true\n' "$valid_record" ;; + extra) printf '%s\n\n' "$valid_record" ;; + uppercase) printf 'lock-sha256=%s\nruntime-tag=rust-v2.0.0\n' "$(printf '%s' "$digest" | tr '[:lower:]' '[:upper:]')" ;; + short) printf 'lock-sha256=abc\nruntime-tag=rust-v2.0.0\n' ;; + garbage) printf '$(touch adoption-executed)\n' ;; + missing) printf 'lock-sha256=%s\n' "$digest" ;; + newline) printf '%s' "$valid_record" ;; + crlf) printf 'lock-sha256=%s\r\nruntime-tag=rust-v2.0.0\r\n' "$digest" ;; + esac > "$d/$adoption" + git -C "$d" add "$adoption" + gcommit "$d" "test: Codex adoption $malformed" + plan_expect "Codex $malformed adoption refuses" $'OP=blocked\nREASON=codex-runtime-needs-local-cut' + done + assert_eq 'Codex adoption data never executed' no "$(if [ -e "$d/adoption-executed" ]; then echo yes; else echo no; fi)" + PLAN_PREVIOUS=v0.1.0 plan_expect 'Codex invalid adoption blocks abandon' $'OP=blocked\nREASON=codex-runtime-needs-local-cut' --version 0.3.0 --abandon-train + # A matching worktree target cannot turn a committed symlink into an assertion. + printf '%s\n' "$valid_record" > "$d/adoption-target" + git -C "$d" rm -q "$adoption" + ln -s ../../adoption-target "$d/$adoption" + git -C "$d" add "$adoption" + gcommit "$d" 'test: Codex adoption symlink' + plan_expect 'Codex symlink adoption refuses' $'OP=blocked\nREASON=codex-runtime-needs-local-cut' + git -C "$d" rm -q "$adoption" + printf '%s\n' "$valid_record" > "$d/$adoption" + git -C "$d" add "$adoption" + gcommit "$d" 'docs: restore valid adoption' + plan_expect 'Codex restored committed adoption' $'OP=nextrc\nTAG=v0.2.0-rc.12' # A human's local cut establishes a new boundary after the adoption read. git -C "$d" tag v0.2.0-rc.12 export PLAN_PREVIOUS=v0.2.0-rc.12 @@ -1285,7 +1339,7 @@ shipping_path_regressions() { # shellcheck source=scripts/lib/shipping-paths.sh . "$SCRIPTS_DIR/lib/shipping-paths.sh" local expected="agent/src agent/package.json agent/package-lock.json agent/tsconfig.json agent/bin agent/templates agent/devbox-global agent/codex scripts/sqlc.sh" - assert_eq "shared runtime set stays exact" "$expected" "$AGENT_PATHS" + assert_eq "shared runtime set stays exact" "$expected :(exclude)agent/codex/codex-package.adopted" "$AGENT_PATHS" local p d out status subject for p in $expected; do if is_shipping "$p"; then pass "runtime entry $p ships"; else fail "runtime entry $p ships"; fi @@ -1295,7 +1349,7 @@ shipping_path_regressions() { for p in api/x.go controller/x.go web/src/x.ts deploy/chart/values.yaml docs/cli.md; do if is_shipping "$p"; then pass "existing shipping $p"; else fail "existing shipping $p"; fi done - for p in api/x_test.go agent/src/foo.test.ts web/src/foo.test.tsx api/testdata/x agent/test/x.ts agent/templates/test/x agent/codex/x_test.go e2e/x fixtures/x .agents/skills/x prds/x.md agent/README.md; do + for p in api/x_test.go agent/src/foo.test.ts web/src/foo.test.tsx api/testdata/x agent/test/x.ts agent/templates/test/x agent/codex/x_test.go agent/codex/codex-package.adopted e2e/x fixtures/x .agents/skills/x prds/x.md agent/README.md; do if is_shipping "$p"; then fail "excluded $p"; else pass "excluded $p"; fi done @@ -1374,6 +1428,42 @@ MD pass "autobump has no private AGENT_PATHS assignment" fi assert_contains "autobump sources shipping library" '/lib/shipping-paths.sh"' "$(cat "$SCRIPTS_DIR/worker-tag-autobump.sh")" + # #4172: the release adoption assertion is not worker execution input. Its + # addition AND later record-only edits must preserve the published worker pin. + d="$(mktemp -d)"; seed_repo "$d" + mkdir -p "$d/agent/codex" + for subject in seed amend; do + printf 'adoption assertion %s\n' "$subject" > "$d/agent/codex/codex-package.adopted" + git -C "$d" add agent/codex/codex-package.adopted + gcommit "$d" "docs: $subject adoption assertion" + status=0 + out="$(cd "$d" && bash scripts/worker-tag-autobump.sh --check 0.2.0 2>&1)" || status=$? + assert_eq "$subject adoption-only check needs no worker bump" 0 "$status" + status=0 + out="$(cd "$d" && bash scripts/worker-tag-autobump.sh 0.2.0 2>&1)" || status=$? + assert_eq "$subject adoption-only edit succeeds" 0 "$status" + assert_eq "$subject adoption-only keeps worker pin" 0.1.0 "$(pin_tag "$d")" + assert_eq "$subject adoption-only leaves tracked inputs unchanged" '' "$(git -C "$d" status --porcelain --untracked-files=no)" + # Keep the next fixture discriminating even against the unmodified autobump. + git -C "$d" checkout -- deploy/chart/values.yaml scripts/assert-worker-tag-decoupled.sh + done + # The exception is exactly one path; neither its neighbor nor the lock is exempt. + for p in agent/codex/codex-package.adopted.extra agent/codex/codex-package.lock; do + printf 'runtime change\n' > "$d/$p" + git -C "$d" add "$p"; gcommit "$d" "chore: change $p" + status=0 + out="$(cd "$d" && bash scripts/worker-tag-autobump.sh --check 0.2.0 2>&1)" || status=$? + assert_eq "$p still requires worker bump" 1 "$status" + assert_contains "$p appears in changed runtime paths" "$p" "$out" + git -C "$d" rm -q "$p"; gcommit "$d" "test: remove $p" + done + printf 'runtime change\n' > "$d/agent/codex/codex-package.lock" + git -C "$d" add agent/codex/codex-package.lock; gcommit "$d" 'chore: change actual runtime lock' + status=0 + out="$(cd "$d" && bash scripts/worker-tag-autobump.sh 0.2.0 2>&1)" || status=$? + assert_eq 'runtime lock edit still succeeds' 0 "$status" + assert_eq 'runtime lock still advances worker pin' 0.2.0 "$(pin_tag "$d")" + rm -rf "$d" # Override only the fixture-local library. A non-agent sentinel proves both actual # Git diff selection and changed-path reporting consume that library's pathspecs. d="$(mktemp -d)"; seed_repo "$d" diff --git a/scripts/worker-tag-autobump.sh b/scripts/worker-tag-autobump.sh index ada2b385a3..405645f7d4 100755 --- a/scripts/worker-tag-autobump.sh +++ b/scripts/worker-tag-autobump.sh @@ -27,6 +27,8 @@ # agent/bin agent/templates agent/devbox-global agent/codex scripts/sqlc.sh # (agent/codex -- the pinned Codex package lock, its install/assert scripts and the Go # supervisor the runtime stage builds -- was missing until #1720.) +# The exact codex-package.adopted assertion is excluded: it records release review +# and changes no worker execution input (#4172). # (A self-improve run DOES read /opt/uzi-src, but it is admin-only / off by default; if # that path ever needs a fresh per-release snapshot, widen AGENT_PATHS deliberately -- # do not silently fold in the whole tree, which defeats the decoupling.)