From c9719b192e07805fa4b9cd50d5f115ce169f831d Mon Sep 17 00:00:00 2001 From: neal Date: Thu, 19 Nov 2026 10:52:52 -0500 Subject: [PATCH 1/2] feat(repo-builder): three-way merge modified managed paths on update A template update stopped for a hand merge whenever a managed file carried both the template's change and the destination's own edits, though the two usually touched different lines. apply now runs git merge-file over the old payload, the destination's file and the new payload. A clean merge lands staged at the payload's mode as action merge, overlapping edits stay left as conflicted, and a merge that cannot run (binary) stays left as modified. pr-body Decisions and the report's Preserved/Merged/Conflicted line name merged paths from the records instead of leaving a slot. Generated-By: claude-sonnet-5-5 --- apps/repo-builder/CLAUDE.md | 2 +- apps/repo-builder/src/references/reporting.md | 2 +- apps/repo-builder/src/references/update.md | 8 +- .../src/scripts/tests/test_update.py | 133 +++++++++++++++++- apps/repo-builder/src/scripts/update.py | 81 +++++++++-- 5 files changed, 203 insertions(+), 23 deletions(-) diff --git a/apps/repo-builder/CLAUDE.md b/apps/repo-builder/CLAUDE.md index ff71b3fd..6ea07134 100644 --- a/apps/repo-builder/CLAUDE.md +++ b/apps/repo-builder/CLAUDE.md @@ -10,7 +10,7 @@ Develops the `/repo-builder` skill, which builds a repository from the payload a - `src/references/` — the flows the entry point routes to: `lifecycle.md` (read whole only by an adopt), `generate.md` with `generate-into-content.md`, `update.md`, `adopt.md`, `retrofit.md`, `wayfinding.md`, `addon-adoption.md`, and `reporting.md` at the end of any of them. `retrofit.md` is an index: it states the `lifecycle.md` rules a retrofit needs throughout inline, each with an anchor link for the reasoning, so a retrofit never opens the whole file, and names the step that reads each of the rest, states the records convention once, and links one file per step, `retrofit-step-1-preflight.md` through `retrofit-step-9-publish.md`, each read when the flow reaches it. `retrofit-unmet-bar.md` is read from step 7 only when the bar is unmet, and `retrofit-report.md` once the flow ends or stops. The reasoning behind the flow's shape is `docs/adrs/0005-stage-a-retrofit-locally-and-write-the-host-last.md`, which step files link rather than restate. `update.md` is inlined the same way: it opens with the lifecycle rules every update needs, each with an anchor link, puts each conditional rule at its triggering step, and leaves out the sections an update never reads (the first manifest's tools, hooks after a merge, ruleset satisfiability, architecture records). `generate.md` is inlined the same way, with the eight lifecycle rules a generate needs and a link to `### Working hooks in a candidate` at step 9; the rules for a destination that already has content are in `generate-into-content.md`, read from step 1 only then, and the reasoning behind the fixed steps is `docs/adrs/0010-run-generates-fixed-steps-as-commands.md`, which steps link rather than restate. A lifecycle rule a generate, a retrofit or an update newly depends on goes on that flow's list or behind an anchor link at the step that needs it, or the flow never reads it. `choosing_a_language.md` is the method wayfinding is a short path through. - `src/scripts/preflight.py` — validates and describes a flow's inputs before it acts. Retrofit shares the git helpers and the addon manifest reader, and none of the record ones: every check in `load_provenance` reads a `.repo-template.json` a retrofit has not written yet. - `src/scripts/retrofit.py` — runs a retrofit's fixed steps and reports each as JSON: `proofs`, `normalize`, `facts`, `references` (tracked lines naming a moved path or a script that now has a console script, read from the proofs record and each unit's `[project.scripts]`; it skips ADRs and plans at any depth, changelogs, and the proofs record's `copy.identical_paths`, and marks a basename or console-script stem several moves or units share as ambiguous with its candidates; its record keeps the `moves` it searched so a rerun after repairs still searches a move the new proofs lost, and a mention resolving to a tracked file from its own directory or unit root is no hit unless that file sits at a move's old path), `autofix`, `hooks`, `sync` (fast-forwards the operator's clone onto a taken merge where it is clean on the default branch, and deletes each `emptied` directory the pulled tree leaves untracked unless it holds a file the proofs record's residue did not name), `sweep`, `hosted read`, `hosted apply` and `resume` (merges the default branch into a standing candidate, taking the default branch's side of a conflicted lockfile), plus `report` (its Orca line read from the record `orca_icon.py` writes) and `pr-body`, which render the final report and the pull-request body from the records the others wrote, and `decide`, which records the judgements both leave as slots. Every slot is keyed in `SLOTS`, so a judgement both renderers show is decided once; `Report` takes the slot table as an argument so `generate.py` renders with its own keys; a line a record can settle renders from that record rather than becoming a slot. It writes where `preflight.py` is read-only by contract, which is why it is a second file rather than more subcommands there; it imports `preflight.py`'s git helpers rather than copying them. -- `src/scripts/update.py` — runs an update's mechanical steps and reports each as JSON: `apply` runs `update_preflight` and lands every delta path that needs no judgment, staged, at the payload's mode read from the target commit (an unmodified path, a `current` path the destination already holds at the target, an absent add, an expired override, a deletion the destination already made, and every `.github/` path in the delta, plus `codeql.yml` unless the record omitted it by choice; `dependabot.yml` lands as the payload's file with the ecosystem entries derived from the destination's tracked manifests appended, staged), skips overridden paths except under `.github/`, whose entry it lands over, counts as discarded, and reports under `refused_overrides` with the path the entry records (a `.github/` entry outside the delta is reported there too, not under `unmatched_overrides`), and lists the rest under `left` with a reason (`modified`, `absent`, `product-owned` including a rename whose old path is product-owned, `codeql-omitted-by-choice`) for the flow to reconcile by hand; a path the ignore file excludes is written to disk and not staged. Its output is `applied` as counts by action (`write`, `already-current`, `move`, `delete`, `already-absent`), `applied_paths` (the same actions mapped to their paths, for `report`), `removed` (each `delete`, `move` with its `old_path`, and `already-absent`, for step 6's grep), `discarded` (each `.github/` path it overwrote, deleted, or moved over destination edits or a refused override, for the loss report), `left`, `dependabot` (null unless the delta holds `dependabot.yml`; `watched` the `[ecosystem, directory]` pairs parsed from the written file, `dropped` every pair the old file watched that it lacks), `pruned` (each `apps//src/.gitkeep` or `apps///src/.gitkeep` it deleted and staged because that `src/` tracks another file, a domain being any folder but `src/`, a scoped folder, or a toolchain folder, which `NOT_DOMAINS` lists and `test_update.py` holds to `scripts/structure`; `pr-body` lists them in the blast radius and `report` under Renamed/deleted), `overridden` and `refused_overrides` as `{path, reason}`, and `unmatched_overrides`. The derivation reads the mapping and the YAML reader from the payload's `scripts/libs/dependabot.sh` at the target commit through bash and keeps no copy in Python; it reads workspace members itself, since the library has no reader for them, and a payload without the library fails the run. `proof` is the copy proof of `references/reporting.md`'s "Reviewing the pull request", stateless and run before `advance`: `identical`, `total`, and the `authored` surface, with `.repo-template.json` always authored; ignored payload paths count only where they are in the delta from the recorded commit to the target; the split itself is `copy_proof`, which takes the ignored paths from its caller so `generate.py filelist` shares it. `pr-body` renders the pull-request body from four records the flow saves beside the clone, named by appending to the destination path (`.apply.json`, `.proof.json`, `.check.txt` for the `scripts/check` table, `.settings.txt` for `scripts/repo-settings check`), and writes `.pr-body.md`: one section per `## ` heading of the destination's `.github/PULL_REQUEST_TEMPLATE.md` (its blast radius the delta without renames folded and without overridden paths), rendered from the records where they settle it, else a `[[FILL: ]]` slot (Decisions is one when `left` was non-empty), with the slot names under `slots`; a missing record fails naming it. `advance` rewrites `template.commit` textually so the file's formatting survives, refuses unless the `"commit": ""` pair occurs exactly once, revalidates the manifest, and stages it. `eligible --owner --template-repo --target [--clone ]` is the several-destinations start: it lists the owner's non-archived repositories, reads each `.repo-template.json` through `gh api` (a repository without one, a 404, is omitted; any other read failure stops it), and reports `{repository, commit, verdict}` with `current`, `eligible` (a strict ancestor of the target) or `not-ancestor`, and with `--clone` writes an empty `/pytest.ini` and clones each eligible one to `/` unless that exists, adding `cloned`. `run` is the several-destinations path: for each clone passed to `--destination`, in order, it applies (record `.apply.json`; a non-empty `left` stops it as `reconcile`), runs `npm ci` for a root `package-lock.json` (`install-failed`), installs hooks through `retrofit.py`'s `hook_types`, `prior_hooks` and `install_hooks` where `.pre-commit-config.yaml` exists, proves, advances, runs `scripts/check` (on failure, or when it cannot run, the prior manifest text is restored and staged, `check-failed` or `stopped`), runs `scripts/repo-settings check` (its `FAIL` rows are `drift` and its `unavailable` rows, or why it could not run, `settings_unavailable`, never a stop; that reason is also written to `.settings.txt` as a `settings unavailable ` row so `report` reads it), branches `repo-builder/`, commits with `--generated-by` as the `Generated-By` trailer, and renders the body, reporting `{destination, status, detail, drift, settings_unavailable}` per clone with `ready` for a finished one; a `PreflightError` or failed subprocess is `stopped`, and the next destination still runs. `publish` pushes each destination's branch and opens its pull request with `.pr-body.md`, refusing one off a `repo-builder/` branch, with nothing ahead of the origin default branch, or whose body holds a `[[FILL:` slot; it is separate from `run` so the remote gate stays the parent's, and records each result as `.publish.json`; a destination whose record already says `published` keeps it and is not pushed again. `verify` then reads each published pull request back, as step 9 of `references/update.md` describes: it waits on `gh pr checks`, and records `{destination, status, pull_request, base, head, files, template_commit, checks}` in `.verify.json`, with `not-published` for a destination whose `.publish.json` is missing or not `published`. `report` renders the Final report for several destinations from the six records beside each clone (`.apply.json`, `.proof.json`, `.check.txt`, `.settings.txt`, `.publish.json`, `.verify.json`; a missing one fails naming it) into `/report.md`, aggregating reconciliation and the authored surface by path with the destinations each applies to, and returns `{operation, path, slots}`; a line the records cannot settle (status when a destination is unverified, has no checks, or has a failing check conclusion, `left`, cleanup for an unpublished destination, pending action) is a `[[FILL: ]]` slot. It writes where `preflight.py` is read-only by contract, which is why it is a separate file; it imports `preflight.py`'s helpers rather than copying them. +- `src/scripts/update.py` — runs an update's mechanical steps and reports each as JSON: `apply` runs `update_preflight` and lands every delta path that needs no judgment, staged, at the payload's mode read from the target commit (an unmodified path, a `current` path the destination already holds at the target, an absent add, an expired override, a deletion the destination already made, a `modified` path whose status is a modification and whose three-way merge is clean, written as action `merge`, and every `.github/` path in the delta, plus `codeql.yml` unless the record omitted it by choice; `dependabot.yml` lands as the payload's file with the ecosystem entries derived from the destination's tracked manifests appended, staged), skips overridden paths except under `.github/`, whose entry it lands over, counts as discarded, and reports under `refused_overrides` with the path the entry records (a `.github/` entry outside the delta is reported there too, not under `unmatched_overrides`), and lists the rest under `left` with a reason (`conflicted` for a merge with overlapping edits, `modified` for a deletion, rename or a file `git merge-file` cannot run, `absent`, `product-owned` including a rename whose old path is product-owned, `codeql-omitted-by-choice`) for the flow to reconcile by hand; a path the ignore file excludes is written to disk and not staged. Its output is `applied` as counts by action (`write`, `merge`, `already-current`, `move`, `delete`, `already-absent`), `applied_paths` (the same actions mapped to their paths, for `report`), `removed` (each `delete`, `move` with its `old_path`, and `already-absent`, for step 6's grep), `discarded` (each `.github/` path it overwrote, deleted, or moved over destination edits or a refused override, for the loss report), `left`, `dependabot` (null unless the delta holds `dependabot.yml`; `watched` the `[ecosystem, directory]` pairs parsed from the written file, `dropped` every pair the old file watched that it lacks), `pruned` (each `apps//src/.gitkeep` or `apps///src/.gitkeep` it deleted and staged because that `src/` tracks another file, a domain being any folder but `src/`, a scoped folder, or a toolchain folder, which `NOT_DOMAINS` lists and `test_update.py` holds to `scripts/structure`; `pr-body` lists them in the blast radius and `report` under Renamed/deleted), `overridden` and `refused_overrides` as `{path, reason}`, and `unmatched_overrides`. The derivation reads the mapping and the YAML reader from the payload's `scripts/libs/dependabot.sh` at the target commit through bash and keeps no copy in Python; it reads workspace members itself, since the library has no reader for them, and a payload without the library fails the run. `proof` is the copy proof of `references/reporting.md`'s "Reviewing the pull request", stateless and run before `advance`: `identical`, `total`, and the `authored` surface, with `.repo-template.json` always authored; ignored payload paths count only where they are in the delta from the recorded commit to the target; the split itself is `copy_proof`, which takes the ignored paths from its caller so `generate.py filelist` shares it. `pr-body` renders the pull-request body from four records the flow saves beside the clone, named by appending to the destination path (`.apply.json`, `.proof.json`, `.check.txt` for the `scripts/check` table, `.settings.txt` for `scripts/repo-settings check`), and writes `.pr-body.md`: one section per `## ` heading of the destination's `.github/PULL_REQUEST_TEMPLATE.md` (its blast radius the delta without renames folded and without overridden paths), rendered from the records where they settle it, else a `[[FILL: ]]` slot (Decisions names each merged path when `left` is empty and is one otherwise), with the slot names under `slots`; a missing record fails naming it. `advance` rewrites `template.commit` textually so the file's formatting survives, refuses unless the `"commit": ""` pair occurs exactly once, revalidates the manifest, and stages it. `eligible --owner --template-repo --target [--clone ]` is the several-destinations start: it lists the owner's non-archived repositories, reads each `.repo-template.json` through `gh api` (a repository without one, a 404, is omitted; any other read failure stops it), and reports `{repository, commit, verdict}` with `current`, `eligible` (a strict ancestor of the target) or `not-ancestor`, and with `--clone` writes an empty `/pytest.ini` and clones each eligible one to `/` unless that exists, adding `cloned`. `run` is the several-destinations path: for each clone passed to `--destination`, in order, it applies (record `.apply.json`; a non-empty `left` stops it as `reconcile`), runs `npm ci` for a root `package-lock.json` (`install-failed`), installs hooks through `retrofit.py`'s `hook_types`, `prior_hooks` and `install_hooks` where `.pre-commit-config.yaml` exists, proves, advances, runs `scripts/check` (on failure, or when it cannot run, the prior manifest text is restored and staged, `check-failed` or `stopped`), runs `scripts/repo-settings check` (its `FAIL` rows are `drift` and its `unavailable` rows, or why it could not run, `settings_unavailable`, never a stop; that reason is also written to `.settings.txt` as a `settings unavailable ` row so `report` reads it), branches `repo-builder/`, commits with `--generated-by` as the `Generated-By` trailer, and renders the body, reporting `{destination, status, detail, drift, settings_unavailable}` per clone with `ready` for a finished one; a `PreflightError` or failed subprocess is `stopped`, and the next destination still runs. `publish` pushes each destination's branch and opens its pull request with `.pr-body.md`, refusing one off a `repo-builder/` branch, with nothing ahead of the origin default branch, or whose body holds a `[[FILL:` slot; it is separate from `run` so the remote gate stays the parent's, and records each result as `.publish.json`; a destination whose record already says `published` keeps it and is not pushed again. `verify` then reads each published pull request back, as step 9 of `references/update.md` describes: it waits on `gh pr checks`, and records `{destination, status, pull_request, base, head, files, template_commit, checks}` in `.verify.json`, with `not-published` for a destination whose `.publish.json` is missing or not `published`. `report` renders the Final report for several destinations from the six records beside each clone (`.apply.json`, `.proof.json`, `.check.txt`, `.settings.txt`, `.publish.json`, `.verify.json`; a missing one fails naming it) into `/report.md`, aggregating reconciliation and the authored surface by path with the destinations each applies to, and returns `{operation, path, slots}`; Preserved/Merged/Conflicted aggregates the merged paths, `none` when there are none; a line the records cannot settle (status when a destination is unverified, has no checks, or has a failing check conclusion, `left`, cleanup for an unpublished destination, pending action) is a `[[FILL: ]]` slot. It writes where `preflight.py` is read-only by contract, which is why it is a separate file; it imports `preflight.py`'s helpers rather than copying them. - `src/scripts/generate.py` — runs a generate's mechanical steps and reports each as JSON under the retrofit records convention. `materialize` archives the payload subtree at the exact source commit into `tmp//` under the template checkout (`--template-repo`, default the working directory), runs `git init -b ` there with no remote, and records `candidate`, `commit`, `subtree`, `head` and `payload_paths`; it refuses a candidate directory that already holds files, and fails when the archive leaves out a payload path, as an `export-ignore` attribute would. `settings` reads the candidate from `.materialize.json` and sends step 5's writes through `retrofit.py`'s writers and `WriteLog`: the merge settings, Dependabot alerts and security updates, push protection, and the ruleset body `ruleset.json` beside the payload subtree, read at the materialize record's commit (`--template-repo`), recording each outcome, a 403 or 422 on push protection or the ruleset as `not offered`; only where the ruleset was created, by this run or one the write log records, does it probe, pushing an empty `commit-tree` commit on the remote tip straight to the default branch and requiring `GH013`, and an accepted push exits non-zero naming the commit. `test_generate.py` holds the ruleset's required contexts to the payload workflows' job `name:` values. `bootstrap` reads the candidate from `.materialize.json`, pushes a parentless empty-tree commit built with `commit-tree` to `origin`'s default branch, and records it as `root`; it refuses a default branch already on `origin` or one in the candidate that is not an empty root, and pushes an empty root already there, which is a push that failed. `labels` creates the sixteen triage roles (respelled by the candidate manifest's `generation.features.triage_labels`, which also turns case-variant renames off) and the five `wayfinder:` labels through `retrofit.py`'s `write_labels` and `WriteLog`, recording `created`, `renamed`, `existing` and `kept`, and fails on a refused write. `hooks --tree ` installs the candidate's hooks through `retrofit.py`'s `hook_types`, `prior_hooks` and `install_hooks` into its own `.git/hooks` at local scope, records `not-applicable` without a `.pre-commit-config.yaml`, and exits 1 unless every configured type resolves where git will consult it. `filelist --tree --commit ` reads the subtree and `generation.applications` from the candidate's `.repo-template.json`, maps the payload's `apps/app-name/` onto the first application, and records `missing`, `additions` and `strays` (the allowed candidate-only set is `.repo-template.json`, `docs/adrs/NNNN-*.md`, `docs/adrs/index.md`, every later application, and `.scratch/` where `generation.features.issue_tracker` is recorded), `ignored_present`, `ignored_tracked` (from `git check-ignore --no-index`), and `copy` from `update.py`'s `copy_proof`; it exits 1 on a missing path, a stray or an ignored-but-tracked path. A step that ran and found defects writes its record with `ok` false and exits 1; a refusal writes none and exits 2. `manifest` writes the candidate's `.repo-template.json` from `apps/github-repository-template/src/ownership.json` at the source commit, `--application` and `--feature` (`triage_labels.=