diff --git a/DOCS.md b/DOCS.md index 6937f1457..debc3a037 100644 --- a/DOCS.md +++ b/DOCS.md @@ -247,8 +247,10 @@ resolved Git common directory and primary path, so separate clones and primary folders do not share a location. The first allocation pins the workspace location in -`/gsd-path/workspaces/.json`. Later environment -changes do not move it. Existing sibling worktrees remain at their original +`/gsd-path/workspaces/.json`. Environment +changes do not move it while a linked worktree remains. When a sidecar is +retired and no linked worktree remains, Path removes the pin, and the next +allocation reads `GSD_PATH_WORKTREE_ROOT` again. Existing sibling worktrees remain at their original paths and retain the same recovery and cleanup checks. These folders contain active work and evidence; they are not a disposable cache. Relocation needs a separate migration; do not edit placement records or move folders manually. @@ -397,6 +399,14 @@ the supplied package's runtime and its compatible status launcher. Review those configuration changes before committing them. Existing versions remain available for other projects and branches. +To record where a patched runtime came from, add `--runtime-provenance-source`, +`--runtime-provenance-patch`, or `--runtime-provenance-note` to +`--runtime-upgrade`. The values are written to the `provenance` object +(`source`, `patch_ref`, `note`) in `.gsd-path/runtime.json`. Without a patch +value, a source root that is a Git checkout records its `HEAD` commit. A later +upgrade without these flags keeps the object only when the digest is unchanged. +Provenance is a record only; it does not change the digest or runtime validation. + A new machine needs the selected runtime installed before status and guards can run. `gsd-path --runtime-restore --project /absolute/project --source-root /path/to/matching/package` restores that exact version without changing project diff --git a/HOOKS.md b/HOOKS.md index 49e1661d8..1dfb83b62 100644 --- a/HOOKS.md +++ b/HOOKS.md @@ -146,7 +146,8 @@ See [UPDATE.md](UPDATE.md). source trees containing directory symlinks are denied; use literal directory paths and copy directory links separately - shell commands that reference the archive unless the whole command is a - recognized standalone read or a single-command invocation of the bundled + recognized standalone read, a pipeline in which each segment is a recognized + read, or a single-command invocation of the bundled `pipeline_state.py` / `archive_milestone.py` helper, resolved to a regular file inside the verified runtime selected by `.gsd-path/runtime.json` (legacy projects use `.gsd-path/runtime/` beside `guard_hook.py` in the repository layout), using exactly `python` or `python3` with optional @@ -162,11 +163,93 @@ See [UPDATE.md](UPDATE.md). - deletion or move commands outside a single simple segment, including command chains, pipes, newlines, grouping, directory changes, and command substitution - deletion or move commands with any argument outside the literal-path character - set: ASCII letters, digits, `.`, `_`, `-`, and `/` (plus a drive prefix and - backslashes on Windows). One matching pair of surrounding quotes is allowed; - spaces, parameters, wildcards, braces, and tilde paths are denied even when - quoted. This also applies to `find -delete` and supported destructive aliases + set: ASCII letters, digits, `.`, `_`, `-`, `/`, and ASCII spaces inside one + shell unit (plus a drive prefix and backslashes on Windows). One matching + pair of surrounding quotes is allowed; parameters, wildcards, braces, and + tilde paths are denied even when quoted. This also applies to + `find -delete` and supported destructive aliases - destructive Git commands nested in supported shell and command wrappers +- Git arguments that contain a shell parameter. Only plain `$name` and + `${name}` expansions pass, and only inside plain double-quoted spans of + read-only subcommands (`status`, `log`, `diff`, `show`, and the other + inspection commands) that the guard parses directly, such as + `git log --format="$t %H"` or `git show "HEAD:$f"`, because only double + quoting prevents word-splitting into options. A parameter outside double + quotes (`$x`, `HEAD:$f`, `--format=$x`, `"a b"$x`) is denied because its + expansion can split into a write option such as `--output`. Every other + parameter form (`${=x}`, `${(z)x}`, `${x:-y}`, `$x[@]`, `$1`, `$@`, `$*`, + `$#`, and the unbraced zsh flags `$=x`, `$~x`, `$^x`, `$+x`) is denied even + inside double quotes, because it can expand to more than one word; a + backslash-escaped marker (`"\$t"`) is likewise denied, because the shell + executes the literal text while the argument still shows the bare marker. + The same four zsh flags are denied in a `git -C` directory. A literal `$` + that starts no expansion (a regex anchor such as `--grep='fix$'`) passes. + A `-c`/`--config` key whose value git executes as a program — + `core.fsmonitor`, editors and pagers, `core.sshCommand`, + `core.askPass`, `filter.*` clean/smudge/process, `diff.*` + textconv/external/command, `merge.*` drivers, `core.gitProxy`, + `interactive.diffFilter`, + `remote.*` uploadpack/receivepack, `gpg.program` and `gpg.*.program`, `credential.helper` and + `credential.*.helper`, each `pager.` key — is denied with a literal + value as with a parameter, because git runs it during an otherwise + read-only subcommand. A disabling form of such a key passes: the bare key, + an empty value, or `false`. A `--config-env` form is always denied, because + its value comes from the environment. A key through which git loads code — + `core.hooksPath`, `include.path`, `includeIf.*.path`, `url.*.insteadOf`, + `url.*.pushInsteadOf` — is denied with each value. `git config` is denied + when it sets one of these keys; to read or unset the key stays allowed. + A redirection at the end of a read (`git config 2>/dev/null`) is not + a value. A separate digit before a redirection (`5 >/dev/null`), a word of two + or more digits (`12>/dev/null`) and a quoted operator are values, so they + make a set. + Each `git config` argument must be a plain literal word, bare or in one + pair of quotes: a backslash, mixed quotes, an unquoted glob (`*`, `?`, `[`), + an unquoted brace list or range (`{a,b}`, `{1..3}`) or a line continuation + in the word or one that joins it to the next word is denied, because the shell can make a different + argument list from it. + An unquoted word that starts with `#` among the `git config` arguments + is denied, because the shell reads it and the words after it as a comment. + The git subcommand word must be a plain literal word in each git command: + `git {config,...}` and a subcommand split by a line continuation are denied. + A program that runs the command in its arguments (`nice`, `nohup`, + `timeout`, `sudo`, `watch`, `time`, `env`, `stdbuf`, `setsid`) is + unwrapped: the guard skips the options of the program and checks the + command that it runs, so `nice git -c core.fsmonitor=/abs/h.sh status` is + denied by the git rule and `sudo apt install git` passes. A runner + option that writes, edits or enters a file or directory by itself is + denied (`time -o`, `sudo -e`, `sudo -D`, `sudo -R`, `watch -s`). The + `timeout` duration and the `watch` interval must be literal numbers, and + each runner option word must be a literal word; only a separate option + value can be a parameter in double quotes (`sudo -u "$USER" ls`). A short + option word is read as getopt reads it, so the attached value in + `sudo -upostgres psql` is not taken as more options. With an archive working directory a command runner is not + unwrapped and is denied as an unknown program. Other command + runners (`doas`, `ionice`, `noglob`) are not unwrapped, and `watch` with + the command in one quoted word is not checked. + Each git global option word, and the value word of `-c`, `-C`, + `--git-dir`, `--work-tree`, `--namespace` and `--config-env`, must resolve + to one word: a backslash, an unquoted glob or an unquoted brace list or + range in it is denied (`git -c {core.fsmonitor=/abs/h.sh,status}`, + `git -{p,c} core.fsmonitor=/abs/h.sh status`). A parameter in such a value + is denied also when no subcommand follows (`git -c $x`). Plain quoted parts + pass (`git -c user.name='A B' log`). + `git config` section renames and removals (`--rename-section`, + `--remove-section`) and `--edit` are denied, because a rename can move a + key into an executed section and `--edit` starts an editor. + Command substitution output, a parameter that starts an + argument, and a parameter in the value of an option that git writes to or + runs as a command (`--output`, `--upload-pack`, `--receive-pack`, `--exec`, + `-c`, `--config-env`, or a `::
` remote) are denied in + every form. These option names match by prefix (`--upload-pa`, `--exe`), + because git accepts abbreviated long options; in archive context the same + prefix match applies to `--output`, `--ext-diff`, and `--textconv`. The + exact read-only option `--text` is not an abbreviation and stays allowed. Any + other quoting construct in the command (`$'...'`, `$"..."`, quotes inside `${...}`, quotes or + backslashes in a comment, here-documents and here-strings) disables the + exception, so every parameter in a git argument of that command is denied. + The exception applies only to commands the guard parses directly, never + inside wrapped shell strings (`eval`, `bash -c`, `sh -c`, PowerShell or + `cmd` command strings), where the outer shell expands the parameter first - archive glob/brace expansions and execution-capable read options such as `rg --pre` - direct write, edit, and patch tool calls targeting `.project/STATE.md`, diff --git a/daemon/gsd_daemon/probe.py b/daemon/gsd_daemon/probe.py index 2c80a95eb..23c9ba34b 100644 --- a/daemon/gsd_daemon/probe.py +++ b/daemon/gsd_daemon/probe.py @@ -623,15 +623,12 @@ def tree_mtime(path: Union[str, Path]) -> Optional[float]: return latest -def _runtime_status(root: str) -> Optional[dict]: - runtime = Path(root) / ".gsd-path" / "runtime" / "pipeline_state.py" - if not runtime.is_file(): - return None +def _run_status_command(command: List[str]) -> Optional[dict]: try: # The daemon's own interpreter: python3 is usually absent on Windows, # where process start and git are also slow enough to need the margin. result = subprocess_platform.run( - [sys.executable, "-B", str(runtime), "status", "--repo", root], + command, capture_output=True, encoding="utf-8", errors="replace", @@ -651,6 +648,19 @@ def _runtime_status(root: str) -> Optional[dict]: return payload +def _runtime_status(root: str) -> Optional[dict]: + # Current installs pin a shared runtime and proxy status through the + # project launcher, which resolves the declaration exactly like the + # status command does. Unmigrated projects keep the per-project runtime. + launcher = Path(root) / ".gsd-path" / "status_runtime.py" + if launcher.is_file(): + return _run_status_command([sys.executable, "-B", str(launcher), "--repo", root]) + runtime = Path(root) / ".gsd-path" / "runtime" / "pipeline_state.py" + if not runtime.is_file(): + return None + return _run_status_command([sys.executable, "-B", str(runtime), "status", "--repo", root]) + + def _activity_time(status: ProjectStatus) -> Optional[datetime]: parsed = _parse_iso(status.last_activity_iso) if parsed is not None: diff --git a/scripts/check_handoffs.py b/scripts/check_handoffs.py index f86fa4306..743f7cdf9 100644 --- a/scripts/check_handoffs.py +++ b/scripts/check_handoffs.py @@ -1386,6 +1386,20 @@ def _task_deps(task_text: str, task_id: str) -> List[str]: return _inline_ids(rendered, "T", f"{task_id} deps") +def _dependency_reachable(graph: Dict[str, List[str]], start: str, target: str) -> bool: + seen: Set[str] = set() + pending = list(graph[start]) + while pending: + task_id = pending.pop() + if task_id in seen: + continue + if task_id == target: + return True + seen.add(task_id) + pending.extend(graph[task_id]) + return False + + def _canonical_repo_path(value: str, label: str) -> str: cleaned = value.strip().strip("`\"'") path = PurePosixPath(cleaned) @@ -1509,6 +1523,12 @@ def _validate_task_graph( and _task_scalar(tasks[right], right, "status") == "done" ): continue + # Dependency-ordered tasks serialize safely; only unordered + # same-wave overlap races on a shared file. + if _dependency_reachable(graph, left, right) or _dependency_reachable( + graph, right, left + ): + continue raise HandoffError( f"same-wave file overlap between {left} and {right}: {', '.join(overlap)}" ) diff --git a/scripts/dispatch_driver.py b/scripts/dispatch_driver.py index 010358d9d..c18b1e008 100644 --- a/scripts/dispatch_driver.py +++ b/scripts/dispatch_driver.py @@ -702,14 +702,14 @@ def finish_task(primary: Path, state: Dict[str, object]) -> Dict[str, object]: if not command: raise DriverStop("task has no Verify command", task=task_id) if state["mode"] == "serial": - # ponytail: sidecar reproduction is git plumbing that belongs in isolation.py as the - # inverse of clean_verify; lift it there when a second caller appears. sidecar = state["sidecar"] - sidecar_path = Path(str(sidecar["worktree"])) + ensured = isolation.ensure_verify_sidecar(primary, base, str(sidecar["branch"])) + sidecar_path = Path(str(ensured["worktree"])) tree = snapshot_tree(primary) - isolation.git_output(sidecar_path, "read-tree", "-u", "--reset", tree) - if isolation.git_output(sidecar_path, "write-tree") != tree: - raise DriverStop("sidecar reproduction differs from the primary task diff", task=task_id) + try: + isolation.reproduce_verify_sidecar(sidecar_path, tree) + except isolation.IsolationError as error: + raise DriverStop(str(error), task=task_id) from error execution = run_verify(command, sidecar_path) location = f"sidecar {sidecar['branch']}" isolation.clean_verify(primary, sidecar_path, base, str(sidecar["branch"])) @@ -2036,7 +2036,7 @@ def fix_tasks(primary: Path, options: argparse.Namespace) -> Dict[str, object]: texts = contracts._task_texts(primary, options.project_dir) groups = {group["locator"]: group for group in findings["groups"]} today = now()[:10] - # Same-wave file overlap is forbidden, so repairs land in a newly appended wave. + # Unordered same-wave file overlap is rejected, so repairs land in a newly appended wave. plan_path = project / "plan/PLAN.md" plan_text = plan_path.read_text(encoding="utf-8") depths, waves = contracts._plan_waves(plan_text) diff --git a/scripts/guard_hook.py b/scripts/guard_hook.py index 286afe31a..2dabda185 100644 --- a/scripts/guard_hook.py +++ b/scripts/guard_hook.py @@ -121,6 +121,10 @@ "committed GSD Path archives under .project/archive/ are read-only; " "only the bundled pipeline helpers may write there during a ship transaction" ) +ARCHIVE_UNPROVEN_READ_REASON = ( + "commands run in archive context must be provably read-only; only " + "recognized read commands and the bundled pipeline helpers may run there" +) DESTRUCTIVE_SHAPE_REASON = ( "destructive commands must run alone with literal paths; split compound, " "substituted, or expanded destructive commands into single commands" @@ -234,6 +238,147 @@ "symbolic-ref": frozenset({"-q", "--quiet", "--short"}), } GIT_READ_WRITE_OPTIONS = frozenset({"--output", "--ext-diff", "--textconv"}) +# Options whose value git writes to or runs as a command: one unknown word is +# already enough, so a parameter in or after them is never accepted. +GIT_PARAMETER_DENIED_OPTIONS = GIT_READ_WRITE_OPTIONS | { + "--upload-pack", "--receive-pack", "--exec", "-c", "--config-env", +} + + +# -c/--config keys whose value git executes as a program. A literal value is +# as dangerous as a parameter here: git -c core.fsmonitor= status runs +# the configured program while the subcommand stays nominally read-only. +GIT_EXECUTED_CONFIG_KEYS = frozenset({ + "core.fsmonitor", "core.editor", "sequence.editor", "core.sshcommand", + "core.askpass", "core.pager", "core.gitproxy", "interactive.difffilter", +}) +GIT_EXECUTED_CONFIG_SECTIONS = { + "filter": (".clean", ".smudge", ".process", ".command"), + "diff": (".command", ".textconv", ".external"), + "merge": (".driver",), + "remote": (".uploadpack", ".receivepack"), + "gpg": (".program",), + "credential": (".helper",), + "pager": ("",), +} +# Keys whose value is a path or URL git loads code through: an included file +# can set every key above, and a rewritten URL can retarget the transport. +# No value of these is a disabling form. +GIT_LOADED_CONFIG_KEYS = frozenset({"core.hookspath", "include.path"}) +GIT_LOADED_CONFIG_SECTIONS = { + "includeif": (".path",), + "url": (".insteadof", ".pushinsteadof"), +} +GIT_CONFIG_DISABLING_VALUES = frozenset({"", "false"}) + + +def config_key_in(key, keys, sections): + if key in keys: + return True + section, _, remainder = key.partition(".") + return bool(remainder) and key.endswith(tuple(sections.get(section, ()))) + + +def loaded_config_key(key): + """Whether a config key's value is a path or URL git loads code through.""" + return config_key_in(key, GIT_LOADED_CONFIG_KEYS, GIT_LOADED_CONFIG_SECTIONS) + + +def executed_config_key(key): + """Whether a config key's value is a program git runs or code it loads.""" + return loaded_config_key(key) or config_key_in( + key, GIT_EXECUTED_CONFIG_KEYS, GIT_EXECUTED_CONFIG_SECTIONS + ) + + +# Real read-only git options that are a strict prefix of a denied option name. +GIT_SAFE_PREFIX_OPTIONS = frozenset({"--text"}) + + +def git_option_in(token, options): + """Whether a token names one of the options, by full name or any long-option prefix. + + git accepts each unique prefix of a long option; matching every prefix is + stricter than git and so only fails closed. A prefix that is itself the + exact name of a read-only option is that option, not an abbreviation. + """ + name = token.split("=", 1)[0] + if name in GIT_SAFE_PREFIX_OPTIONS and name not in options: + return False + if name.startswith("--") and len(name) > 2: + return any(option.startswith(name) for option in options) + return name in options +GIT_CONFIG_READ_MODES = frozenset({ + "--get", "--get-all", "--get-regexp", "--get-urlmatch", "--get-color", + "--get-colorbool", "-l", "--list", "--unset", "--unset-all", +}) +GIT_CONFIG_READ_SUBCOMMANDS = frozenset({"get", "unset", "list"}) +GIT_CONFIG_DENIED_MODES = frozenset({"--rename-section", "--remove-section", "--edit"}) +GIT_CONFIG_DENIED_SUBCOMMANDS = frozenset({"rename-section", "remove-section", "edit"}) +GIT_CONFIG_VALUE_OPTIONS = frozenset({ + "--file", "--blob", "--type", "--default", "--comment", "--value", "--url", +}) +GIT_CONFIG_EDIT_SHORT_OPTION = re.compile(r"-[A-Za-z]*e[A-Za-z]*") +# An unquoted word the shell can turn into several arguments: a glob, or a +# brace list or range. +PLAIN_QUOTED_SEGMENT = re.compile(r"'[^']*'|\"[^\"]*\"") +ANSI_QUOTE_START = re.compile(r"\$['\"]") +GIT_CONFIG_EXPANDING_WORD = re.compile(r"[*?\[]|\{.*(?:,|\.\.)") + + +def denied_git_config_argument(arguments): + """The git config argument that persists or runs a program git executes. + + A section rename can carry a key into an executed section and --edit runs + the editor, so both are denied outright. Otherwise only a set denies: an + executed key with a value operand after it and no read or unset mode. + Only a trailing redirection is dropped: the operator, its target and a + file descriptor digit written against it. A separate digit (5 >file) and a + quoted operator are operands. + """ + arguments = list(arguments) + while ( + len(arguments) >= 2 + and getattr(arguments[-2], "redirection", False) + and not arguments[-2].isdigit() + ): + del arguments[-2:] + if ( + arguments + and getattr(arguments[-1], "redirection", False) + and arguments[-1].isdigit() + ): + del arguments[-1] + separator = arguments.index("--") if "--" in arguments else len(arguments) + options = arguments[:separator] + for token in options: + if ( + git_option_in(token, GIT_CONFIG_DENIED_MODES) + or GIT_CONFIG_EDIT_SHORT_OPTION.fullmatch(token) + or token.casefold() in GIT_CONFIG_DENIED_SUBCOMMANDS + ): + return token + if options[:1] and options[0].casefold() in GIT_CONFIG_READ_SUBCOMMANDS: + return None + for position, token in enumerate(options): + # A mode word that is the value of the option before it is no mode. + if token in GIT_CONFIG_READ_MODES and not ( + position + and ( + options[position - 1] == "-f" + or git_option_in(options[position - 1], GIT_CONFIG_VALUE_OPTIONS) + ) + ): + return None + operands = [ + token for token in options if not token.startswith("-") + ] + arguments[separator + 1:] + for token in operands[:-1]: + if executed_config_key(token.casefold()): + return token + return None + + ARCHIVE_READ_EXECUTION_OPTIONS = {"rg": frozenset({"--pre"})} AMBIGUOUS_SHELL_SYNTAX = re.compile(r"[\r\n|;&<>`]|\$\(|@\(") SUBSTITUTION_PLACEHOLDER = "COMMAND_SUBSTITUTION_" @@ -248,9 +393,12 @@ NAMED_SHELL_PARAMETER_SYNTAX = re.compile( r"\$(?:([A-Za-z_][A-Za-z0-9_]*)|\{([A-Za-z_][A-Za-z0-9_]*)\})" ) +ZSH_EXPANSION_FLAG_SYNTAX = re.compile(r"\$[=~^+]") CMD_PARAMETER_SYNTAX = re.compile( r"%([A-Za-z_][A-Za-z0-9_]*)%|!([A-Za-z_][A-Za-z0-9_]*)!" ) +# The shell reassigns these on every cd, so a tracked value is stale. +SHELL_DIRECTORY_VARIABLES = frozenset({"PWD", "OLDPWD"}) DIRECTORY_CHANGE_COMMANDS = frozenset( {"cd", "chdir", "pushd", "set-location", "sl"} ) @@ -390,6 +538,43 @@ } ) FIND_EXECUTION_ACTIONS = frozenset({"-exec", "-execdir", "-ok", "-okdir"}) +# Programs that run the command in their arguments, with the options of each +# that take a separate value. The guard checks the command they run. +EXEC_PREFIX_VALUE_OPTIONS = { + "nice": frozenset({"-n", "--adjustment"}), + "nohup": frozenset(), + "setsid": frozenset(), + "stdbuf": frozenset({"-i", "-o", "-e", "--input", "--output", "--error"}), + "sudo": frozenset({ + "-u", "-g", "-h", "-p", "-C", "-T", "-U", "-r", "-t", + "--user", "--group", "--host", "--prompt", "--close-from", + "--command-timeout", "--other-user", "--role", "--type", + }), + "time": frozenset({"-f", "--format"}), + "timeout": frozenset({"-s", "-k", "--signal", "--kill-after"}), + "watch": frozenset({"-n", "-q", "--interval", "--equexit"}), +} +# Runner options that write, edit or enter a file or directory by themselves. +EXEC_PREFIX_FILE_OPTIONS = { + "sudo": frozenset({"-e", "-D", "-R", "--edit", "--chdir", "--chroot"}), + "time": frozenset({"-o", "--output"}), + "watch": frozenset({"-s", "--shotsdir"}), +} +# The short option letters of each runner that take no value. +EXEC_PREFIX_FLAG_LETTERS = { + "nice": "", + "nohup": "", + "setsid": "cfw", + "stdbuf": "", + "sudo": "AbBEHiKklnNPSsVv", + "time": "ahlpqvV", + "timeout": "fpv", + "watch": "bcCdegprtwx", +} +EXEC_PREFIX_INTERVAL_OPTIONS = frozenset({"-n", "--interval"}) +RUNNER_DURATION = re.compile( + r"(?:(?:\d+(?:\.\d*)?|\.\d+)(?:e[+-]?\d+)?|inf(?:inity)?)[smhd]?", re.IGNORECASE +) SHELL_CONTROL_WORDS = frozenset( { "!", @@ -1176,10 +1361,10 @@ def directory_change_target(arguments): return operands[-1] -def segment_directories(tokens, working_directories): +def segment_directories(tokens, working_directories, contexts=None): """Yield each command segment with the working directories in effect for it.""" current_directories = list(working_directories) - for segment in command_segments(tokens): + for position, segment in enumerate(command_segments(tokens)): yield segment, current_directories invocation = command_invocation(segment) if invocation is None: @@ -1192,16 +1377,31 @@ def segment_directories(tokens, working_directories): if command not in DIRECTORY_CHANGE_COMMANDS: continue target = directory_change_target(arguments) - if SHELL_PARAMETER_SYNTAX.search(target): + tracked = {} + if contexts is not None and position < len(contexts): + tracked = { + name: value + for name, value in contexts[position].items() + if name not in SHELL_DIRECTORY_VARIABLES + } + expanded = expand_environment_parameters(target, tracked, tracked_only=True) + if SHELL_PARAMETER_SYNTAX.search(expanded) or CMD_PARAMETER_SYNTAX.search(expanded): raise ValueError( f"cd target {target} cannot be resolved by the guard; " "run that command first and cd to the literal result" ) - if is_absolute_path(target): - current_directories = [target] + if ( + (SHELL_PARAMETER_SYNTAX.search(target) or CMD_PARAMETER_SYNTAX.search(target)) + and any(character.isspace() or character in "*?[]" for character in expanded) + ): + raise ValueError( + f"cd target {target} may split or glob; pass a literal path" + ) + if is_absolute_path(expanded): + current_directories = [expanded] else: bases = current_directories or ["."] - current_directories = [f"{base}/{target}" for base in bases] + current_directories = [f"{base}/{expanded}" for base in bases] def command_can_destroy_files(invocation): @@ -1217,9 +1417,11 @@ def command_can_destroy_files(invocation): def literal_path(operand): if len(operand) >= 2 and operand[0] in "\"'" and operand[-1] == operand[0]: operand = operand[1:-1] - pattern = r"[A-Za-z0-9._/-]+" + # A space is safe here: the operand is one shlex unit, so it cannot carry a + # second word; injection-relevant characters stay excluded. + pattern = r"[A-Za-z0-9._/ -]+" if os.name == "nt": - pattern = r"(?:[A-Za-z]:(?=[/\\]))?[A-Za-z0-9._/\\-]+" + pattern = r"(?:[A-Za-z]:(?=[/\\]))?[A-Za-z0-9._/\\ -]+" return operand if re.fullmatch(pattern, operand) is not None else None @@ -1235,13 +1437,24 @@ def literal_destructive_operand(operand): def segment_references_archive(segment, directories, assignments=None): invocation = command_invocation(segment) ancestors = command_can_destroy_files(invocation) - operands = [expand_environment_parameters(token, assignments) for token in segment] + # A here-document body survives tokenization only inside a wrapped payload + # (bash -c '...'), where it is data, not commands; keep its text from + # flipping the segment into archive context. + def searchable(token): + return strip_heredoc_bodies(token) if "\n" in token else token + + operands = [ + expand_environment_parameters(searchable(token), assignments) + for token in segment + ] if ancestors: resolved = command_invocation(operands) operands = [literal_destructive_operand(operand) for operand in resolved[1]] if any(path_in_archive(operand, directories, ancestors) for operand in operands): return True - return unresolved_archive_expansion(" ".join(segment), assignments) + return unresolved_archive_expansion( + " ".join(searchable(token) for token in segment), assignments + ) def copy_destinations(arguments, directories, assignments=None): @@ -1588,20 +1801,22 @@ def protected_shell_write_reason(tokens, working_directories): return None -def environment_parameter_value(match, assignments): +def environment_parameter_value(match, assignments, tracked_only=False): name = match.group(1) or match.group(2) if name.startswith(SUBSTITUTION_PLACEHOLDER): return match.group(0) # command substitution output is never resolved if name in assignments: return assignments[name] + if tracked_only: + return match.group(0) return os.environ.get(name, match.group(0)) -def expand_environment_parameters(command, assignments=None): +def expand_environment_parameters(command, assignments=None, tracked_only=False): values = assignments or {} def substitute(match): - return environment_parameter_value(match, values) + return environment_parameter_value(match, values, tracked_only) expanded = NAMED_SHELL_PARAMETER_SYNTAX.sub(substitute, command) return CMD_PARAMETER_SYNTAX.sub(substitute, expanded) @@ -1701,7 +1916,7 @@ def shell_assignment_contexts(tokens, initial=None): def shell_segment_contexts(tokens, working_directories, initial=None): """Yield each segment with its working directory and prior shell variables.""" contexts, _ = shell_assignment_contexts(tokens, initial) - segments = list(segment_directories(tokens, working_directories)) + segments = list(segment_directories(tokens, working_directories, contexts)) if len(segments) != len(contexts): raise ValueError("shell command segments cannot be aligned") for (segment, directories), assignments in zip(segments, contexts): @@ -1992,14 +2207,222 @@ def decode_patch_path(raw_path, strip_prefix): return path -def shell_tokens(command): - command = strip_heredoc_bodies(LINE_CONTINUATION.sub(r"\1 ", command)) +PARAMETER_MARKERS = frozenset("$%!") + + +class ShellToken(str): + """A token that keeps, per parameter marker character, whether double quotes enclosed it. + + `redirection` is true for an unquoted redirection operator and for one + ASCII file descriptor digit written against one (the 2 of 2>file, not of + 2 >file). zsh passes a longer or non-ASCII digit word on as an argument. + `literal` is "bare" for a word written with no quote or escape, "quoted" + for a word inside one pair of plain quotes, "mixed" for plain quoted and + unquoted parts where no unquoted part can expand, and None when the shell + may build the word in another way (escapes, an expanding unquoted part, a + line continuation in the word or one that joins it to the next word). + """ + + double_quoted = () + redirection = False + literal = None + + +def double_quoted_markers(command): + """Return, per parameter marker in order, whether plain double quotes enclose it. + + None means the command uses quoting outside the plain constructs this scan + models (plain single quotes, plain double quotes, unquoted text), so no + marker in it can be proven quoted. + """ + flags = [] + quote = None + comment = False + braces = 0 + previous = " " + index = 0 + while index < len(command): + character = command[index] + following = command[index + 1 : index + 2] + if comment: + if character in "\r\n": + comment = False + elif character in "'\"\\`": + return None + elif quote == "'": + if character == "'": + quote = None + elif character == "\\": + if following and following in PARAMETER_MARKERS: + # The shell executes the escaped marker as literal text; the + # token keeps the bare marker, so the pair is never provable. + flags.append(False) + previous = character + index += 2 + continue + elif character == "`" or (character == "$" and following == "("): + return None + elif character == "$" and following == "{": + braces += 1 + elif character == "}" and braces: + braces -= 1 + elif character in "'\"" and braces: + return None + elif character == '"': + quote = None if quote else '"' + elif quote is None: + if character == "'": + quote = "'" + elif character == "$" and following and following in "'\"": + return None + elif character == "<" and following == "<": + return None + elif character == "#" and previous in " \t\r\n;&|()<>": + comment = True + if character in PARAMETER_MARKERS: + flags.append(quote == '"' and not comment) + previous = character + index += 1 + return flags if quote is None and not braces else None + + +def zsh_expansion_flag(text): + """Whether the text holds an unbraced zsh expansion flag ($=x, $~x, $^x, $+x). + + These match no parameter syntax, so without this check they never enter + the parameter proof at all — while zsh word-splits them exactly like the + forms the proof denies. Any other unmatched $ is inert literal text. + """ + return ZSH_EXPANSION_FLAG_SYNTAX.search(text) + + +def parameters_double_quoted(token): + """Whether every parameter in the token expands inside double quotes to one word. + + Only plain $name and ${name} qualify: flags, operators, subscripts, + positional and special parameters can split even inside double quotes. + A backslash-escaped marker stays one flag short of proven: the shell + executes the literal text while the guard's token still shows the bare + marker, so nothing about its expansion is provable. + """ + flags = getattr(token, "double_quoted", ()) + positions = [ + index for index, character in enumerate(token) if character in PARAMETER_MARKERS + ] + if len(flags) != len(positions): + return False + quoted = dict(zip(positions, flags)) + plain = { + match.start(): match.end() for match in NAMED_SHELL_PARAMETER_SYNTAX.finditer(token) + } + if any( + character == "$" and index not in plain for index, character in enumerate(token) + ) or any(token[end : end + 1] == "[" for end in plain.values()): + return False + return all( + quoted[match.start()] and quoted.get(match.end() - 1, True) + for syntax in (SHELL_PARAMETER_SYNTAX, CMD_PARAMETER_SYNTAX) + for match in syntax.finditer(token) + ) + + +def shell_tokens(command, direct=True): + """Tokenize a command; `direct` is False for a string that a second shell parses. + + Only a directly parsed command keeps quote flags: an outer shell expands + parameters before the inner parse, so inner quotes prove nothing. + """ + breaks = set() + if LINE_CONTINUATION.search(command): + mark = next( + character for character in map(chr, range(0xE000, 0xF900)) + if character not in command + ) + marked = strip_heredoc_bodies( + LINE_CONTINUATION.sub(lambda match: match.group(1) + mark, command) + ) + command = strip_heredoc_bodies(LINE_CONTINUATION.sub(r"\1 ", command)) + breaks = ( + {index for index, character in enumerate(marked) if character == mark} + if marked.replace(mark, " ") == command + else None + ) + else: + command = strip_heredoc_bodies(command) + joins = set() + for index in breaks or (): + left, right = index - 1, index + 1 + while left >= 0 and marked[left] == mark: + left -= 1 + while right < len(marked) and marked[right] == mark: + right += 1 + if left >= 0 and marked[left].strip() and marked[right:right + 1].strip(): + joins.add(index) lexer = shlex.shlex(command, posix=True, punctuation_chars="|;&()<>\n\r") lexer.whitespace = " \t" lexer.whitespace_split = True lexer.commenters = "" + tokens = [] + previous_end = 0 try: - tokens = list(lexer) + for token in lexer: + end = lexer.instream.tell() - len(lexer._pushback_chars) + if command[end - 1:end] in (" ", "\t"): + end -= 1 + word_start = previous_end + while word_start < end and command[word_start] in " \t": + word_start += 1 + previous_end = end + word = command[word_start:end] + literal = None + if ( + breaks is not None + and "\\" not in word + and not ANSI_QUOTE_START.search(word) + and breaks.isdisjoint(range(word_start, end)) + and end not in joins + and word_start - 1 not in joins + and PLAIN_QUOTED_SEGMENT.sub(lambda match: match.group(0)[1:-1], word) == token + ): + unquoted = PLAIN_QUOTED_SEGMENT.sub("", word) + if ( + "'" not in unquoted + and '"' not in unquoted + and not GIT_CONFIG_EXPANDING_WORD.search(unquoted) + ): + literal = "mixed" + for quote in ("", "'", '"'): + start = end - len(token) - 2 * len(quote) + if ( + breaks is not None + and start >= 0 + and command[start:end] == quote + token + quote + and (start == 0 or command[start - 1] in " \t|;&()<>\n\r") + and breaks.isdisjoint(range(start, end)) + and end not in joins + and start - 1 not in joins + ): + literal = "quoted" if quote else "bare" + break + start = end - len(token) + before = command[start - 1:start] if start > 0 else "" + redirection = command[start:end] == token and ( + ( + set(token) <= SHELL_WRITE_REDIRECTION_CHARS + and ("<" in token or ">" in token) + and before != "\\" + ) + or ( + len(token) == 1 + and token in "0123456789" + and command[end:end + 1] in ("<", ">") + and (not before or before in " \t|;&()<>\n\r") + ) + ) + token = ShellToken(token) + token.redirection = redirection + token.literal = literal + tokens.append(token) except ValueError as error: raise ValueError( f"shell command cannot be tokenized ({error}); balance the quotes " @@ -2007,7 +2430,20 @@ def shell_tokens(command): ) from None if not tokens: raise ValueError("shell command is empty") - return tokens + flags = double_quoted_markers(command) if direct else None + if flags is None or len(flags) != sum( + character in PARAMETER_MARKERS for token in tokens for character in token + ): + return tokens + marked = [] + for token in tokens: + count = sum(character in PARAMETER_MARKERS for character in token) + redirection, literal = token.redirection, token.literal + token = ShellToken(token) + token.redirection, token.literal = redirection, literal + token.double_quoted, flags = tuple(flags[:count]), flags[count:] + marked.append(token) + return marked def command_segments(tokens): @@ -2067,56 +2503,163 @@ def xargs_command(arguments): def require_read_command(wrapper, wrapped): - if wrapped and not archive_command_is_read_only(" ".join(wrapped), wrapped, True): + segments = list(command_segments(wrapped)) + if wrapped and ( + not segments + or not all( + archive_command_is_read_only(shlex.join(segment), segment, True) + for segment in segments + ) + ): raise ValueError( f"{wrapper} runs {wrapped[0]} on operands the guard cannot check; " f"only read commands may follow {wrapper}, or pass the paths as literals" ) -def command_invocation(segment): +def runner_value_option(prefix, token): + """The option in a runner option word whose value is the next word, if any. + + Raises for an option that writes, edits or enters a file or directory. A + long option matches by each prefix, as getopt accepts it. A short word is + read as getopt reads it: flag letters, then the first value option letter, + whose value is the rest of the word or, with no rest, the next word. + """ + value_options = EXEC_PREFIX_VALUE_OPTIONS[prefix] + file_options = EXEC_PREFIX_FILE_OPTIONS.get(prefix, frozenset()) + denial = ValueError( + f"{prefix} {token} writes, edits or enters a file or directory that the " + "guard cannot check; run the command without this option and redirect " + "its output instead" + ) + if token.startswith("--"): + name = token.split("=", 1)[0] + if any(option.startswith(name) for option in file_options): + raise denial + option = next((option for option in value_options if option.startswith(name)), None) + return option if "=" not in token else None + for position, letter in enumerate(token[1:], start=1): + option = f"-{letter}" + if option in file_options: + raise denial + if option in value_options: + return option if position == len(token) - 1 else None + if letter not in EXEC_PREFIX_FLAG_LETTERS[prefix]: + return None + return None + + +def nonliteral_runner_option(token): + literal = getattr(token, "literal", "bare") + return ( + SHELL_PARAMETER_SYNTAX.search(token) + or CMD_PARAMETER_SYNTAX.search(token) + or zsh_expansion_flag(token) + or literal is None + or (literal == "bare" and GIT_CONFIG_EXPANDING_WORD.search(token)) + ) + + +def unresolved_runner_word(token): + return ( + SHELL_PARAMETER_SYNTAX.search(token) + or CMD_PARAMETER_SYNTAX.search(token) + or zsh_expansion_flag(token) + ) and not parameters_double_quoted(token) + + +def command_invocation(segment, unwrap_runners=True): + """`unwrap_runners` is False where a command runner itself must be judged.""" index = 0 - while index < len(segment) and SHELL_ASSIGNMENT_PATTERN.match(segment[index]): - validate_shell_assignment(segment[index]) + # The shell word `time` is already removed; its only option can remain. + while unwrap_runners and index < len(segment) and segment[index] == "-p": index += 1 - if index >= len(segment): - return None - executable = segment[index].replace("\\", "/").rsplit("/", 1)[-1].casefold() - if executable == "env": - index += 1 - while index < len(segment): - token = segment[index] - if token == "--": - index += 1 + while True: + while index < len(segment) and SHELL_ASSIGNMENT_PATTERN.match(segment[index]): + validate_shell_assignment(segment[index]) + index += 1 + if index >= len(segment): + return None + executable = segment[index].replace("\\", "/").rsplit("/", 1)[-1].casefold() + prefix = executable.removesuffix(".exe") + if executable == "env": + index += 1 + while index < len(segment): + token = segment[index] + if token == "--": + index += 1 + break + if SHELL_ASSIGNMENT_PATTERN.match(token): + validate_shell_assignment(token) + index += 1 + continue + option = token.split("=", 1)[0] + if option in ENV_OPTIONS_WITHOUT_VALUES: + index += 1 + continue + if option in ENV_OPTIONS_WITH_VALUES: + if option in {"-C", "--chdir"}: + raise ValueError( + f"env {option} changes the working directory the guard " + "tracks; use cd with a literal path instead" + ) + index += 1 + if "=" not in token: + if index >= len(segment): + raise ValueError(f"env {option} lacks a value") + index += 1 + continue + if token.startswith("-"): + raise ValueError( + f"env {token} cannot be validated; use env NAME=VALUE " + ) + if SHELL_PARAMETER_SYNTAX.search(token) or zsh_expansion_flag(token): + raise ValueError( + f"env word {token} cannot be resolved by the guard; env " + "assignments must be literal NAME=VALUE words and the " + "program a literal name" + ) break - if SHELL_ASSIGNMENT_PATTERN.match(token): - validate_shell_assignment(token) + elif unwrap_runners and prefix in EXEC_PREFIX_VALUE_OPTIONS: + index += 1 + while index < len(segment): + token = segment[index] + if token == "--": + index += 1 + break + if not token.startswith("-") or token == "-": + break index += 1 - continue - option = token.split("=", 1)[0] - if option in ENV_OPTIONS_WITHOUT_VALUES: + if nonliteral_runner_option(token): + raise ValueError( + f"{prefix} option {token} cannot be resolved by the guard; " + f"{prefix} options must be literal words" + ) + option = runner_value_option(prefix, token) + if option is None or index >= len(segment): + continue + value = segment[index] index += 1 - continue - if option in ENV_OPTIONS_WITH_VALUES: - if option in {"-C", "--chdir"}: + if prefix == "watch" and option in EXEC_PREFIX_INTERVAL_OPTIONS: + if not RUNNER_DURATION.fullmatch(value): + raise ValueError( + f"watch interval {value} cannot be resolved by the guard; " + "the interval must be a literal number" + ) + elif unresolved_runner_word(value): raise ValueError( - f"env {option} changes the working directory the guard " - "tracks; use cd with a literal path instead" + f"{prefix} option value {value} cannot be resolved by the " + f"guard; {prefix} option values must be literal words" + ) + if prefix == "timeout" and index < len(segment): + if not RUNNER_DURATION.fullmatch(segment[index]): + raise ValueError( + f"timeout duration {segment[index]} cannot be resolved by the " + "guard; the duration must be a literal number" ) index += 1 - if "=" not in token: - if index >= len(segment): - raise ValueError(f"env {option} lacks a value") - index += 1 - continue - if token.startswith("-"): - raise ValueError( - f"env {token} cannot be validated; use env NAME=VALUE " - ) + else: break - if index >= len(segment): - return None - executable = segment[index].replace("\\", "/").rsplit("/", 1)[-1].casefold() arguments = segment[index + 1 :] if SHELL_PARAMETER_SYNTAX.search(executable): raise ValueError( @@ -2140,6 +2683,20 @@ def command_invocation(segment): return executable, arguments +def require_resolved_git_word(word, mixed=True): + """Deny a git word the shell can turn into other or several arguments.""" + literal = getattr(word, "literal", "bare") + if ( + literal is None + or (literal == "mixed" and not mixed) + or (literal == "bare" and GIT_CONFIG_EXPANDING_WORD.search(word)) + ): + raise ValueError( + f"git argument {word} cannot be resolved by the guard; " + "pass a literal value" + ) + + def git_command(segment, assignments=None): invocation = command_invocation(segment) if invocation is None: @@ -2148,6 +2705,8 @@ def git_command(segment, assignments=None): if executable not in {"git", "git.exe"}: return None resolved_arguments = [] + parameter_arguments = [] + unproven_parameters = [] argument_index = 0 while argument_index < len(arguments): argument = arguments[argument_index] @@ -2155,8 +2714,13 @@ def git_command(segment, assignments=None): if argument == "-C" and argument_index + 1 < len(arguments): argument_index += 1 directory = arguments[argument_index] + require_resolved_git_word(directory) expanded = expand_environment_parameters(directory, assignments) - if SHELL_PARAMETER_SYNTAX.search(expanded) or CMD_PARAMETER_SYNTAX.search(expanded): + if ( + SHELL_PARAMETER_SYNTAX.search(expanded) + or CMD_PARAMETER_SYNTAX.search(expanded) + or zsh_expansion_flag(expanded) + ): raise ValueError( f"git argument {directory} cannot be resolved by the guard; " "pass a literal path" @@ -2170,9 +2734,14 @@ def git_command(segment, assignments=None): ) resolved_arguments.extend((argument, expanded)) elif attached_directory: + require_resolved_git_word(argument) directory = argument[2:] expanded = expand_environment_parameters(directory, assignments) - if SHELL_PARAMETER_SYNTAX.search(expanded) or CMD_PARAMETER_SYNTAX.search(expanded): + if ( + SHELL_PARAMETER_SYNTAX.search(expanded) + or CMD_PARAMETER_SYNTAX.search(expanded) + or zsh_expansion_flag(expanded) + ): raise ValueError( f"git argument {directory} cannot be resolved by the guard; " "pass a literal path" @@ -2186,12 +2755,29 @@ def git_command(segment, assignments=None): ) resolved_arguments.append(f"-C{expanded}") else: - if SHELL_PARAMETER_SYNTAX.search(argument) or CMD_PARAMETER_SYNTAX.search(argument): - raise ValueError( - f"git argument {argument} cannot be resolved by the guard; pass a " - "literal value (for commit messages use -F )" - ) - resolved_arguments.append(expand_environment_parameters(argument, assignments)) + parameter = ( + SHELL_PARAMETER_SYNTAX.search(argument) + or CMD_PARAMETER_SYNTAX.search(argument) + or zsh_expansion_flag(argument) + ) + if parameter: + parameter_arguments.append(argument) + literal = re.split(r"[$%!]", argument, maxsplit=1)[0] + expanded = expand_environment_parameters(argument, assignments) + if ( + not literal + or (literal.startswith("-") and "=" not in literal) + or git_option_in(literal, GIT_PARAMETER_DENIED_OPTIONS) + or git_option_in(arguments[argument_index - 1], GIT_PARAMETER_DENIED_OPTIONS) + or (literal.startswith("-c") and literal != "-c") + or "::" in literal + or not parameters_double_quoted(argument) + or SUBSTITUTION_PLACEHOLDER in argument + or SUBSTITUTION_PLACEHOLDER in expanded + ): + unproven_parameters.append(argument) + expanded = expand_environment_parameters(argument, assignments) + resolved_arguments.append(argument if expanded == argument else expanded) argument_index += 1 arguments = resolved_arguments index = 0 @@ -2204,18 +2790,32 @@ def git_command(segment, assignments=None): option = token.split("=", 1)[0] value = token.split("=", 1)[1] if "=" in token else None index += 1 + if option != "-C": + require_resolved_git_word(token) if option in GIT_GLOBAL_OPTIONS_WITH_VALUES and value is None: if index >= len(arguments): raise ValueError(f"git option {option} lacks a value") value = arguments[index] + if option != "-C": + require_resolved_git_word(value) index += 1 if option in {"-c", "--config-env"}: - config_key = str(value).split("=", 1)[0].casefold() + config_key, _, config_value = str(value).partition("=") + config_key = config_key.casefold() if config_key.startswith("alias."): raise ValueError( f"git -c {config_key} defines an alias the guard cannot inspect; " "run the underlying git command directly" ) + if executed_config_key(config_key) and not ( + option == "-c" + and not loaded_config_key(config_key) + and config_value.casefold() in GIT_CONFIG_DISABLING_VALUES + ): + raise ValueError( + f"git -c {config_key} configures a program git executes; " + "the guard cannot allow it" + ) if config_key == "clean.requireforce": raise ValueError( "git -c clean.requireForce disables the git clean safety check; " @@ -2224,8 +2824,47 @@ def git_command(segment, assignments=None): if option in GIT_GLOBAL_OPTIONS_WITH_VALUES: git_options.extend((option, str(value))) if index >= len(arguments): + if parameter_arguments: + raise ValueError( + f"git argument {parameter_arguments[0]} cannot be resolved by the guard; " + "pass a literal value (for commit messages use -F )" + ) return None - return arguments[index].casefold(), arguments[index + 1:], git_options + require_resolved_git_word(arguments[index], mixed=False) + subcommand = arguments[index].casefold() + if subcommand == "config": + for argument in arguments[index + 1:]: + literal = getattr(argument, "literal", None) + if not getattr(argument, "redirection", False) and ( + literal in (None, "mixed") + or ( + literal == "bare" + and (argument.startswith("#") or GIT_CONFIG_EXPANDING_WORD.search(argument)) + ) + ): + raise ValueError( + f"git argument {argument} cannot be resolved by the guard; " + "pass a literal value" + ) + denied = denied_git_config_argument(arguments[index + 1:]) + if denied is not None: + raise ValueError( + f"git config {denied} persists a program git executes; " + "the guard cannot allow it" + ) + read_only_subcommands = ARCHIVE_READ_GIT_COMMANDS | CLOSED_READ_GIT_COMMANDS + if subcommand in read_only_subcommands: + # Read-only subcommands still write through --output. A parameter + # passes only inside plain double quotes, where its expansion stays + # one word; an unquoted one can word-split into an option, and + # command substitution output is never known. + parameter_arguments = unproven_parameters + if parameter_arguments: + raise ValueError( + f"git argument {parameter_arguments[0]} cannot be resolved by the guard; " + "pass a literal value (for commit messages use -F )" + ) + return subcommand, arguments[index + 1:], git_options def wrapped_command_tokens(segment, expand_parameters=True): @@ -2234,7 +2873,7 @@ def wrapped_command_tokens(segment, expand_parameters=True): return None executable, arguments = invocation if executable == "eval": - return shell_tokens(" ".join(arguments)) if arguments else None + return shell_tokens(" ".join(arguments), direct=False) if arguments else None if executable in COMMAND_WRAPPERS: if executable == "command" and arguments[:1] in (["-v"], ["-V"]): return None @@ -2259,7 +2898,7 @@ def wrapped_command_tokens(segment, expand_parameters=True): executable=executable, option=argument ) ) - return shell_tokens(arguments[index + 1]) + return shell_tokens(arguments[index + 1], direct=False) raise ValueError(SHELL_FILE_REASON.format(executable=executable)) if executable in POWERSHELL_WRAPPERS or executable in {"cmd", "cmd.exe"}: switches = {"-c", "-command", "/c", "/k"} @@ -2278,8 +2917,8 @@ def wrapped_command_tokens(segment, expand_parameters=True): for argument in payload ] if len(payload) == 1: - return shell_tokens(payload[0]) - return payload + return shell_tokens(payload[0], direct=False) + return [str(argument) for argument in payload] raise ValueError(SHELL_FILE_REASON.format(executable=executable)) return None @@ -2368,7 +3007,7 @@ def destructive_git_reason( "git command directly" ) reason = destructive_git_reason( - ["git", *git_options, *shell_tokens(alias), *arguments], + ["git", *git_options, *shell_tokens(alias, direct=False), *arguments], resolved_aliases | {command}, assignments, ) @@ -2525,13 +3164,15 @@ def bundled_helper_invocation(command, tokens, working_directories, helpers=PIPE def archive_command_is_read_only( - command, tokens, archive_context=False, git_commands=ARCHIVE_READ_GIT_COMMANDS + command, tokens, archive_context=False, git_commands=ARCHIVE_READ_GIT_COMMANDS, + source=None, ): + """`source` is the unexpanded (segment, assignments) whose tokens keep their quote flags.""" if archive_context and AMBIGUOUS_SHELL_SYNTAX.search(command): return False if not archive_context: return True - invocation = command_invocation(tokens) + invocation = command_invocation(tokens, unwrap_runners=False) if invocation is None: return True executable, arguments = invocation @@ -2554,15 +3195,44 @@ def archive_command_is_read_only( return True if executable != "git": return False - git = git_command(tokens) + git = git_command(*(source or (tokens,))) if git is None: return False subcommand, git_arguments, _options = git if subcommand not in git_commands: return False return not any( - token in GIT_READ_WRITE_OPTIONS or token.startswith("--output=") - for token in git_arguments + git_option_in(token, GIT_READ_WRITE_OPTIONS) for token in git_arguments + ) + + +def archive_write_attempt(segment, source=None): + """Whether a segment denied in archive context attempts a write there.""" + if any( + token + and (set(token) <= SHELL_WRITE_REDIRECTION_CHARS + or re.fullmatch(r"\d*[<>]+&?", token)) + for token in segment + ): + return True + invocation = command_invocation(segment) + if invocation is None: + return False + executable, arguments = invocation + executable = executable.removesuffix(".exe") + if executable in SHELL_WRITE_COMMANDS: + return True + if executable in IN_PLACE_EDITORS and ( + has_short_option(arguments, "i") + or any(argument.startswith("--in-place") for argument in arguments) + ): + return True + git = git_command(*(source or (segment,))) + if git is None: + return False + subcommand, git_arguments, _options = git + return subcommand not in ARCHIVE_READ_GIT_COMMANDS or any( + git_option_in(token, GIT_READ_WRITE_OPTIONS) for token in git_arguments ) @@ -2790,12 +3460,15 @@ def command_denial(command, working_directories, allow_destructive=True): ) ) if not archive_command_is_read_only( - shlex.join(resolved_segment), resolved_segment, archive_context + shlex.join(resolved_segment), resolved_segment, archive_context, + source=(segment, assignments), ) and not ( archive_context and bundled_helper ): - return ARCHIVE_REASON + if archive_write_attempt(resolved_segment, (segment, assignments)): + return ARCHIVE_REASON + return ARCHIVE_UNPROVEN_READ_REASON reason = destructive_git_reason(tokens) or protected_shell_write_reason( tokens, working_directories ) diff --git a/scripts/install.py b/scripts/install.py index 1567e1ef3..9f460b4fc 100644 --- a/scripts/install.py +++ b/scripts/install.py @@ -3283,6 +3283,31 @@ def install( return results +def _runtime_provenance(arguments, source_root: Path): + """Owner-recorded provenance from --runtime-provenance-* upgrade flags. + + When the supplied source root is a git checkout and no patch reference + was given, its HEAD sha is recorded with one best-effort rev-parse. + Provenance is display/audit metadata on the runtime declaration only; it + never joins the pinned digest or runtime validation. + """ + supplied = (arguments.runtime_provenance_source, arguments.runtime_provenance_patch, + arguments.runtime_provenance_note) + recorded = {key: value for key, value in zip(runtime_store.PROVENANCE_KEYS, supplied) if value} + if not recorded: + return None + if "patch_ref" not in recorded: + try: + head = subprocess.run(["git", "rev-parse", "--show-toplevel", "HEAD"], cwd=source_root, + capture_output=True, encoding="utf-8", check=True) + toplevel, _, sha = head.stdout.strip().partition("\n") + if sha and Path(toplevel).resolve() == Path(source_root).resolve(): + recorded["patch_ref"] = sha + except (OSError, subprocess.CalledProcessError): + pass + return recorded + + def parser() -> argparse.ArgumentParser: argument_parser = argparse.ArgumentParser(description=__doc__) for target in TARGETS: @@ -3299,6 +3324,9 @@ def parser() -> argparse.ArgumentParser: runtime_actions = argument_parser.add_mutually_exclusive_group() for action in ("restore", "upgrade", "migrate"): runtime_actions.add_argument(f"--runtime-{action}", action="store_true") + argument_parser.add_argument("--runtime-provenance-source") + argument_parser.add_argument("--runtime-provenance-patch") + argument_parser.add_argument("--runtime-provenance-note") argument_parser.add_argument("--hooks", action="store_true") argument_parser.add_argument("--hooks-init", action="store_true") argument_parser.add_argument("--hooks-refresh", action="store_true") @@ -3344,6 +3372,9 @@ def main(argv: Optional[Sequence[str]] = None) -> int: ) runtime_action = next((action for action in ("restore", "upgrade", "migrate") if getattr(arguments, "runtime_" + action)), None) + provenance = _runtime_provenance(arguments, source_root) + if provenance is not None and runtime_action != "upgrade": + argument_parser.error("--runtime-provenance-* requires --runtime-upgrade") if runtime_action: if project is None: argument_parser.error(f"--runtime-{runtime_action} requires --project") @@ -3351,7 +3382,8 @@ def main(argv: Optional[Sequence[str]] = None) -> int: _validate_project_git_root(project) locks, created = ([], []) if arguments.dry_run else _acquire_install_locks([project / HOOKS_DIRECTORY]) try: - pin = runtime_store.operate(source_root, project, runtime_action, dry_run=arguments.dry_run) + pin = runtime_store.operate(source_root, project, runtime_action, + dry_run=arguments.dry_run, provenance=provenance) finally: _release_install_locks(locks, created) print(f"runtime: {runtime_action} {pin['version']} ({pin['digest']})" diff --git a/scripts/integration.py b/scripts/integration.py index 0dd2a356b..c1b0971d5 100644 --- a/scripts/integration.py +++ b/scripts/integration.py @@ -9,6 +9,7 @@ import json import re +import contextlib from pathlib import Path, PurePosixPath from typing import Optional @@ -1939,6 +1940,10 @@ def retire_member(coordinator: Path, member: str, archive_path: str, reviewed_he for kind, ref in (("push", bound_ref), ("push", "refs/heads/main"), ("push", f"refs/tags/{tag_name}"), ("delete", bound_ref)): members.clear_authorization(checkout, project, kind, ref) + # The last sidecar retires here at milestone end; isolation's retires also + # release the pin whenever the workspace is idle, so a new root can apply. + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(checkout) return {"member": member, "retired": bound} diff --git a/scripts/isolation.py b/scripts/isolation.py index 50a9ef684..3805f809a 100644 --- a/scripts/isolation.py +++ b/scripts/isolation.py @@ -15,6 +15,7 @@ import io import json import shlex +import shutil import os import re import stat @@ -171,6 +172,12 @@ def sidecar_root(primary: Path, kind: str, name: str, *, pin: bool = False) -> P raise IsolationError(str(error)) from error +def release_workspace_quietly(primary: Path) -> None: + """Best-effort placement hygiene after a sidecar retires; never blocks landing.""" + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(primary) + + def validate_task_id(task_id: str) -> str: if not TASK_ID_RE.fullmatch(task_id): raise IsolationError(f"invalid task id: {task_id}") @@ -477,6 +484,70 @@ def _same_directory(path: Path, expected: os.stat_result) -> bool: return stat.S_ISDIR(current.st_mode) and os.path.samestat(current, expected) +def _long_path(absolute: str) -> str: + if absolute.startswith("\\\\?\\"): + return absolute + if absolute.startswith("\\\\"): + return "\\\\?\\UNC\\" + absolute[2:] + return "\\\\?\\" + absolute + + +def _force_rmtree(path: Path) -> None: + """Delete a sidecar tree directly, tolerating read-only files and long paths.""" + def _writable(function, target, _error): + os.chmod(target, stat.S_IWRITE) + function(target) + + # The \\?\ prefix sidesteps Windows MAX_PATH on deep sidecar trees. + target = Path(_long_path(os.path.abspath(path))) if os.name == "nt" else path + try: + shutil.rmtree(target, onerror=_writable) + except OSError as error: + raise IsolationError(f"could not delete {path}: {error}") from error + + +def _remove_worktree(primary: Path, destination: Path, force: bool) -> None: + """Remove a sidecar worktree, tolerating populated submodules and locked trees. + + git refuses to remove worktrees whose submodules were ever initialized, and + forced removal can still fail deleting deep trees (Windows "Directory not + empty"). A dirty sidecar keeps git's refusal unless `force`, so forced + retries only run for a provably clean tree. The direct-delete fallback runs + only for git's tree-deletion failures, so any other git error leaves the + sidecar intact for diagnosis. + """ + tree_delete_failure = ("failed to delete", "directory not empty", "permission denied") + + def failure(*flags: str) -> str: + result = run_git(primary, "worktree", "remove", *flags, str(destination)) + return (result.stderr or result.stdout).strip() if result.returncode else "" + + detail = failure("--force") if force else failure() + if not detail: + return + if not force and destination.is_dir(): + status = run_git( + destination, "status", "--porcelain", "--untracked-files=all" + ) + if status.returncode != 0 or status.stdout.strip(): + raise IsolationError(detail or "git worktree remove failed") + run_git(destination, "submodule", "deinit", "--all", "--force") + detail = failure("--force") + if not detail: + return + if destination.is_dir() and any( + marker in detail.lower() for marker in tree_delete_failure + ): + _force_rmtree(destination) + detail = failure("--force") + if detail and not destination.is_dir(): + pruned = run_git(primary, "worktree", "prune") + if pruned.returncode == 0: + detail = "" + if detail or destination.is_dir() or destination.resolve() in _registered_worktrees(primary): + raise IsolationError(detail or "git worktree remove failed") + + def _cleanup_reserved_worktree( primary: Path, branch: str, @@ -762,9 +833,7 @@ def retire_member_task( if destination.resolve() in registered: if registered[destination.resolve()] != ref: raise IsolationError(f"member task sidecar is on another branch: {destination}") - removed = run_git(checkout, "worktree", "remove", *(("--force",) if force else ()), str(destination)) - if removed.returncode != 0: - raise IsolationError((removed.stderr or removed.stdout).strip() or "git worktree remove failed") + _remove_worktree(checkout, destination, force) authorization = run_git(checkout, "update-ref", "-d", member_task_authorization_ref(project, task_id)) if authorization.returncode != 0: raise IsolationError((authorization.stderr or authorization.stdout).strip() or "git update-ref failed") @@ -772,6 +841,7 @@ def retire_member_task( deleted = run_git(checkout, "update-ref", "-d", ref, tip.stdout.strip()) if deleted.returncode != 0: raise IsolationError((deleted.stderr or deleted.stdout).strip() or "git update-ref failed") + release_workspace_quietly(checkout) return copy_rejected @@ -1392,8 +1462,9 @@ def retire_member_verify(created: Dict[str, Dict[str, str]]) -> None: """Remove the member siblings of a coordinator verify sidecar; they hold no output.""" for record in reversed(list(created.values())): checkout, destination, branch = _member_verify_sidecar(record) - git_output(checkout, "worktree", "remove", "--force", str(destination)) + _remove_worktree(checkout, destination, True) git_output(checkout, "branch", "-D", branch) + release_workspace_quietly(checkout) def check_member_verify(created: Dict[str, Dict[str, str]]) -> None: @@ -1425,6 +1496,89 @@ def clean_verify(primary: Path, worktree: Path, base: str, branch: str) -> Dict[ return {"worktree": str(worktree), "branch": branch, "base": expected, "removed": removed} +def ensure_verify_sidecar(primary: Path, base: str, branch: str) -> Dict[str, object]: + """Return the named verify sidecar, recreating it when it is missing. + + Finish needs the sidecar prepare-task created. A deleted sidecar must be + rebuildable from the recorded base without a clean primary: the serial + task's own product changes are exactly the tree finish reproduces next, + so this must not reuse isolate_verify's clean-tree gate. + """ + primary = require_directory(primary, "primary worktree") + if worktree_root(primary) != primary: + raise IsolationError(f"primary is not its Git root: {primary}") + resolved_base = require_commit(primary, require_full_sha(base)) + if not branch.startswith(VERIFY_BRANCH_PREFIX): + raise IsolationError("verify sidecar reproduction requires a verify branch") + destination = _sidecar_path_for_branch(primary, branch) + ref = f"refs/heads/{branch}" + registered = _registered_worktrees(primary) + if destination.is_symlink() or (destination.exists() and not destination.is_dir()): + raise IsolationError(f"worktree path already exists: {destination}") + if destination.is_dir(): + try: + # A leftover empty directory (a half-deleted sidecar) clears; + # anything non-empty must be the sidecar itself. + destination.rmdir() + except OSError: + try: + same = ( + worktree_root(destination) == destination + and destination.resolve() in registered + and registered[destination.resolve()] == ref + and common_git_dir(destination) == common_git_dir(primary) + ) + except IsolationError: + same = False + if not same: + raise IsolationError( + f"verify sidecar ownership changed: {destination}" + ) from None + if ( + require_attached(destination) != branch + or current_sha(destination) != resolved_base + ): + raise IsolationError(f"verify sidecar HEAD changed: {destination}") + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + if destination.resolve() in registered: + cleared = run_git(primary, "worktree", "remove", str(destination)) + if cleared.returncode != 0: + run_git(primary, "worktree", "prune") + if destination.resolve() in _registered_worktrees(primary): + raise IsolationError( + (cleared.stderr or cleared.stdout).strip() + or "could not clear stale worktree registration" + ) + existing = run_git(primary, "show-ref", "--verify", "--quiet", ref) + if existing.returncode == 0: + deleted = run_git(primary, "update-ref", "-d", ref) + if deleted.returncode != 0: + raise IsolationError( + (deleted.stderr or deleted.stdout).strip() or "git update-ref failed" + ) + create_named_worktree(primary, branch, destination, resolved_base) + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + + +def reproduce_verify_sidecar(sidecar_path: Path, tree: str) -> None: + """Copy the primary's current task tree into the verify sidecar (clean_verify's inverse).""" + git_output(sidecar_path, "read-tree", "-u", "--reset", tree) + if git_output(sidecar_path, "write-tree") != tree: + raise IsolationError("sidecar reproduction differs from the primary task diff") + + def _split_paths(block: str) -> Set[str]: return {line.strip() for line in block.splitlines() if line.strip()} @@ -1537,14 +1691,24 @@ def _commit_pending( def validate_allowed_changes( - repo: Path, base: str, pending: Set[str], allowed: Set[str] + repo: Path, + base: str, + pending: Set[str], + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> None: - if not pending: + # Tolerated paths (primary bookkeeping) may stay uncommitted; the next + # `.project` checkpoint commits them. + landing = pending - tolerated + if not landing: raise IsolationError("no changes to land") - since_base = committed_paths_since(repo, base) | pending + since_base = committed_paths_since(repo, base) | landing unexpected = sorted(path for path in since_base if path not in allowed) if unexpected: - raise IsolationError("unexpected paths: " + ", ".join(unexpected)) + raise IsolationError( + "unexpected paths: " + ", ".join(unexpected) + + "; landing allows only the task file and its declared files" + ) def _clean_blob_oid(repo: Path, path: str, text: str, *, write: bool) -> str: arguments = ["hash-object"] if write: @@ -1564,21 +1728,23 @@ def commit_allowed_changes( subject: str, allowed: Set[str], body: str, + tolerated: Set[str] = frozenset(), ) -> tuple[str, str, Dict[str, Optional[tuple[str, str]]]]: require_attached(repo) pending = uncommitted_paths(repo) - validate_allowed_changes(repo, base, pending, allowed) + validate_allowed_changes(repo, base, pending, allowed, tolerated) + staging = pending - tolerated reset = run_git(repo, "reset", "-q", "HEAD") if reset.returncode != 0: raise IsolationError("could not clear the index before landing") - for path in sorted(pending): + for path in sorted(staging): added = run_git(repo, "add", "-A", "--", path) if added.returncode != 0: raise IsolationError(f"could not stage {path}") staged = _split_paths( git_output(repo, "diff", "--cached", "--name-only", "--relative") ) - if staged != pending: + if staged != staging: raise IsolationError("staged paths do not match the uncommitted change set") verified_tree = git_output(repo, "write-tree") verified_delta = _tree_delta(repo, base, verified_tree) @@ -1724,6 +1890,17 @@ def activate_task( if task_branch is None: if branch.startswith(TASK_BRANCH_PREFIX): raise IsolationError("parallel task activation requires --task-branch") + live = [ + path + for path, ref in _registered_worktrees(worktree).items() + if ref == f"refs/heads/{task_branch_name(task_id)}" + and path != worktree + ] + if live: + raise IsolationError( + "task has an active parallel worktree; serial activation in " + f"another worktree is refused (activate {live[0]} with --task-branch)" + ) elif task_branch != task_branch_name(task_id) or branch != task_branch: raise IsolationError("task activation branch does not match the task worktree") normalized_task = relative_posix(task_file) @@ -1991,7 +2168,12 @@ def _restore_rejected_commit( def _stamp_and_commit( - worktree: Path, base: str, subject: str, task_file: str, allowed: Set[str] + worktree: Path, + base: str, + subject: str, + task_file: str, + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> tuple[str, Dict[str, Optional[tuple[str, str]]]]: task_path = worktree / task_file task_text, task_mode = _read_task_text(task_path) @@ -2012,14 +2194,14 @@ def _stamp_and_commit( pending = uncommitted_paths(worktree) if stamped != task_bytes: pending.add(task_file) - validate_allowed_changes(worktree, base, pending, allowed) + validate_allowed_changes(worktree, base, pending, allowed, tolerated) original_head = current_sha(worktree) index_tree = git_output(worktree, "write-tree") try: _replace_regular_file(task_path, stamped, task_mode) - body = task_commit_body(task_file, uncommitted_paths(worktree), base) + body = task_commit_body(task_file, uncommitted_paths(worktree) - tolerated, base) commit, verified_tree, verified_delta = commit_allowed_changes( - worktree, base, subject, allowed, body + worktree, base, subject, allowed, body, tolerated ) proof_error = _landing_commit_proof_error( worktree, @@ -2097,6 +2279,7 @@ def land( subject, relative_posix(task_file), allowed, + BOOKKEEPING_PATHS, ) return { "bound_branch": bound, @@ -2391,7 +2574,7 @@ def _landing_retry_error( if task_file not in pending: return "interrupted landing does not retain the task change" try: - validate_allowed_changes(repo, base, pending, contract_paths) + validate_allowed_changes(repo, base, pending, contract_paths, BOOKKEEPING_PATHS) except IsolationError as validation_error: return str(validation_error) return None @@ -4430,6 +4613,7 @@ def retire( or f"could not delete {branch}" ) branch_retired = True + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": branch, @@ -4463,22 +4647,13 @@ def retire( raise IsolationError( "worktree is dirty; pass --force only from the retry-retirement path" ) - remove = run_git( - primary, - "worktree", - "remove", - *(("--force",) if force else ()), - str(resolved_worktree), - ) - if remove.returncode != 0: - raise IsolationError( - (remove.stderr or remove.stdout).strip() or "git worktree remove failed" - ) + _remove_worktree(primary, resolved_worktree, force) deleted = run_git(primary, "branch", "-D", retire_branch) if deleted.returncode != 0: raise IsolationError( (deleted.stderr or deleted.stdout).strip() or f"could not delete {retire_branch}" ) + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": retire_branch, diff --git a/scripts/members.py b/scripts/members.py index d427c6ee4..7f516c846 100644 --- a/scripts/members.py +++ b/scripts/members.py @@ -255,6 +255,7 @@ def check_member( coordinator_name: str, checkout: Path, recorded_remote: Optional[str] = None, + refuse_reserved_refs: bool = True, ) -> str: """Refuse a checkout that cannot join; return its origin URL.""" if checkout.is_symlink() or not checkout.is_dir(): @@ -294,13 +295,14 @@ def check_member( raise MembersError(f"member STATE.md is unreadable: {error}") from error if state.milestone is not None and (state.phase, state.status) != ("shipped", "done"): raise MembersError(f"member has an active milestone: {state.milestone}") - prefixes = member_ref_prefixes(coordinator_name) + tuple( - "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS - ) - refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() - colliding = [ref for ref in refs if ref.startswith(prefixes)] - if colliding: - raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) + if refuse_reserved_refs: + prefixes = member_ref_prefixes(coordinator_name) + tuple( + "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS + ) + refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() + colliding = [ref for ref in refs if ref.startswith(prefixes)] + if colliding: + raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) return remote @@ -620,7 +622,7 @@ def validate_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) try: role = member_role(checkout) except MembersError as error: @@ -639,7 +641,7 @@ def repair_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) _refuse_foreign_marker(checkout.resolve(), root, state.project, member["name"]) for member in members: checkout = Path(member["checkout"]).resolve() diff --git a/scripts/runtime_store.py b/scripts/runtime_store.py index 9a2594d26..4fab50ca1 100644 --- a/scripts/runtime_store.py +++ b/scripts/runtime_store.py @@ -27,6 +27,15 @@ ) GUARDS = ("guard_hook.py", "git_guard.py") +# Optional owner-recorded provenance on .gsd-path/runtime.json declarations, +# e.g. {"source": "", "patch_ref": "", "note": ""}. Provenance is display/audit +# metadata ONLY: it never joins the hashed manifest input (manifest_digest +# covers exactly {"version", "files"}), never acts as an alternate accepted +# digest, and never bypasses or weakens validate_runtime. A declaration +# without provenance is stock and stays exactly as valid as one with it. +PROVENANCE_KEYS = ("source", "patch_ref", "note") + # Access denied and sharing violation: on Windows a scanner or indexer that # briefly opens a freshly written file blocks renaming the directory holding it. _WINDOWS_BUSY_ERRORS = (5, 32) @@ -220,7 +229,7 @@ def recover_migration(project): journal.unlink() -def operate(source, project, action, *, dry_run=False): +def operate(source, project, action, *, dry_run=False, provenance=None): if project.is_symlink() or (project / ".gsd-path").is_symlink(): raise ValueError("unsafe project runtime directory") if action == "restore": @@ -228,8 +237,17 @@ def operate(source, project, action, *, dry_run=False): publish(source, expected=pin, dry_run=dry_run, repair=True) return pin if action == "upgrade": - status_runtime.declaration(project) + previous = status_runtime.declaration(project) pin = publish(source, dry_run=dry_run) + # Explicit provenance replaces the recorded object. Without new flags, + # an existing entry carries forward only while the pinned digest is + # unchanged — a different source drops stale hotfix metadata instead + # of stamping it onto bytes it no longer describes. Metadata only — + # it never changes the pinned digest. + if provenance is None and previous.get("digest") == pin["digest"]: + provenance = previous.get("provenance") + if provenance: + pin = {**pin, "provenance": provenance} if not dry_run: launcher = project / ".gsd-path/status_runtime.py" # Exact bytes: read_text() would turn CRLF into LF, leaving the diff --git a/scripts/worktree_paths.py b/scripts/worktree_paths.py index 19653ca92..b2cfda32f 100644 --- a/scripts/worktree_paths.py +++ b/scripts/worktree_paths.py @@ -103,6 +103,33 @@ def _workspace(primary: Path, pin: bool) -> Path: return workspace +def release_workspace(primary: Path) -> bool: + """Forget the pinned placement receipt once no linked worktrees remain. + + Lets GSD_PATH_WORKTREE_ROOT (or a new default) apply again at the next + milestone's first allocation. Returns True when a receipt was removed; + any remaining linked worktree (including one only still registered) keeps + the pin. + """ + primary = primary.resolve() + common = Path(_git(primary, "rev-parse", "--git-common-dir")) + common = (primary / common).resolve() + receipt = common / "gsd-path" / "workspaces" / f"{_identity(primary)}.json" + if not receipt.is_file(): + return False + output, nul_separated = _worktree_list_porcelain(primary) + records = output.split("\0\0") if nul_separated else [ + block for block in output.strip().split("\n\n") if block.strip() + ] + for record in records: + fields = record.split("\0") if nul_separated else record.splitlines() + for field in fields: + if field.startswith("worktree ") and Path(field[len("worktree "):]).resolve() != primary: + return False + receipt.unlink() + return True + + def worktree_path(primary: Path, kind: str, name: str, *, pin: bool = False) -> Path: primary = primary.resolve() if kind not in {"task", "verify", "integrate", "bound"} or not name or Path(name).name != name or name in {".", ".."}: diff --git a/skills/gsd-path-build/scripts/check_handoffs.py b/skills/gsd-path-build/scripts/check_handoffs.py index f86fa4306..743f7cdf9 100644 --- a/skills/gsd-path-build/scripts/check_handoffs.py +++ b/skills/gsd-path-build/scripts/check_handoffs.py @@ -1386,6 +1386,20 @@ def _task_deps(task_text: str, task_id: str) -> List[str]: return _inline_ids(rendered, "T", f"{task_id} deps") +def _dependency_reachable(graph: Dict[str, List[str]], start: str, target: str) -> bool: + seen: Set[str] = set() + pending = list(graph[start]) + while pending: + task_id = pending.pop() + if task_id in seen: + continue + if task_id == target: + return True + seen.add(task_id) + pending.extend(graph[task_id]) + return False + + def _canonical_repo_path(value: str, label: str) -> str: cleaned = value.strip().strip("`\"'") path = PurePosixPath(cleaned) @@ -1509,6 +1523,12 @@ def _validate_task_graph( and _task_scalar(tasks[right], right, "status") == "done" ): continue + # Dependency-ordered tasks serialize safely; only unordered + # same-wave overlap races on a shared file. + if _dependency_reachable(graph, left, right) or _dependency_reachable( + graph, right, left + ): + continue raise HandoffError( f"same-wave file overlap between {left} and {right}: {', '.join(overlap)}" ) diff --git a/skills/gsd-path-build/scripts/dispatch_driver.py b/skills/gsd-path-build/scripts/dispatch_driver.py index 010358d9d..c18b1e008 100644 --- a/skills/gsd-path-build/scripts/dispatch_driver.py +++ b/skills/gsd-path-build/scripts/dispatch_driver.py @@ -702,14 +702,14 @@ def finish_task(primary: Path, state: Dict[str, object]) -> Dict[str, object]: if not command: raise DriverStop("task has no Verify command", task=task_id) if state["mode"] == "serial": - # ponytail: sidecar reproduction is git plumbing that belongs in isolation.py as the - # inverse of clean_verify; lift it there when a second caller appears. sidecar = state["sidecar"] - sidecar_path = Path(str(sidecar["worktree"])) + ensured = isolation.ensure_verify_sidecar(primary, base, str(sidecar["branch"])) + sidecar_path = Path(str(ensured["worktree"])) tree = snapshot_tree(primary) - isolation.git_output(sidecar_path, "read-tree", "-u", "--reset", tree) - if isolation.git_output(sidecar_path, "write-tree") != tree: - raise DriverStop("sidecar reproduction differs from the primary task diff", task=task_id) + try: + isolation.reproduce_verify_sidecar(sidecar_path, tree) + except isolation.IsolationError as error: + raise DriverStop(str(error), task=task_id) from error execution = run_verify(command, sidecar_path) location = f"sidecar {sidecar['branch']}" isolation.clean_verify(primary, sidecar_path, base, str(sidecar["branch"])) @@ -2036,7 +2036,7 @@ def fix_tasks(primary: Path, options: argparse.Namespace) -> Dict[str, object]: texts = contracts._task_texts(primary, options.project_dir) groups = {group["locator"]: group for group in findings["groups"]} today = now()[:10] - # Same-wave file overlap is forbidden, so repairs land in a newly appended wave. + # Unordered same-wave file overlap is rejected, so repairs land in a newly appended wave. plan_path = project / "plan/PLAN.md" plan_text = plan_path.read_text(encoding="utf-8") depths, waves = contracts._plan_waves(plan_text) diff --git a/skills/gsd-path-build/scripts/integration.py b/skills/gsd-path-build/scripts/integration.py index 0dd2a356b..c1b0971d5 100644 --- a/skills/gsd-path-build/scripts/integration.py +++ b/skills/gsd-path-build/scripts/integration.py @@ -9,6 +9,7 @@ import json import re +import contextlib from pathlib import Path, PurePosixPath from typing import Optional @@ -1939,6 +1940,10 @@ def retire_member(coordinator: Path, member: str, archive_path: str, reviewed_he for kind, ref in (("push", bound_ref), ("push", "refs/heads/main"), ("push", f"refs/tags/{tag_name}"), ("delete", bound_ref)): members.clear_authorization(checkout, project, kind, ref) + # The last sidecar retires here at milestone end; isolation's retires also + # release the pin whenever the workspace is idle, so a new root can apply. + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(checkout) return {"member": member, "retired": bound} diff --git a/skills/gsd-path-build/scripts/isolation.py b/skills/gsd-path-build/scripts/isolation.py index 50a9ef684..3805f809a 100644 --- a/skills/gsd-path-build/scripts/isolation.py +++ b/skills/gsd-path-build/scripts/isolation.py @@ -15,6 +15,7 @@ import io import json import shlex +import shutil import os import re import stat @@ -171,6 +172,12 @@ def sidecar_root(primary: Path, kind: str, name: str, *, pin: bool = False) -> P raise IsolationError(str(error)) from error +def release_workspace_quietly(primary: Path) -> None: + """Best-effort placement hygiene after a sidecar retires; never blocks landing.""" + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(primary) + + def validate_task_id(task_id: str) -> str: if not TASK_ID_RE.fullmatch(task_id): raise IsolationError(f"invalid task id: {task_id}") @@ -477,6 +484,70 @@ def _same_directory(path: Path, expected: os.stat_result) -> bool: return stat.S_ISDIR(current.st_mode) and os.path.samestat(current, expected) +def _long_path(absolute: str) -> str: + if absolute.startswith("\\\\?\\"): + return absolute + if absolute.startswith("\\\\"): + return "\\\\?\\UNC\\" + absolute[2:] + return "\\\\?\\" + absolute + + +def _force_rmtree(path: Path) -> None: + """Delete a sidecar tree directly, tolerating read-only files and long paths.""" + def _writable(function, target, _error): + os.chmod(target, stat.S_IWRITE) + function(target) + + # The \\?\ prefix sidesteps Windows MAX_PATH on deep sidecar trees. + target = Path(_long_path(os.path.abspath(path))) if os.name == "nt" else path + try: + shutil.rmtree(target, onerror=_writable) + except OSError as error: + raise IsolationError(f"could not delete {path}: {error}") from error + + +def _remove_worktree(primary: Path, destination: Path, force: bool) -> None: + """Remove a sidecar worktree, tolerating populated submodules and locked trees. + + git refuses to remove worktrees whose submodules were ever initialized, and + forced removal can still fail deleting deep trees (Windows "Directory not + empty"). A dirty sidecar keeps git's refusal unless `force`, so forced + retries only run for a provably clean tree. The direct-delete fallback runs + only for git's tree-deletion failures, so any other git error leaves the + sidecar intact for diagnosis. + """ + tree_delete_failure = ("failed to delete", "directory not empty", "permission denied") + + def failure(*flags: str) -> str: + result = run_git(primary, "worktree", "remove", *flags, str(destination)) + return (result.stderr or result.stdout).strip() if result.returncode else "" + + detail = failure("--force") if force else failure() + if not detail: + return + if not force and destination.is_dir(): + status = run_git( + destination, "status", "--porcelain", "--untracked-files=all" + ) + if status.returncode != 0 or status.stdout.strip(): + raise IsolationError(detail or "git worktree remove failed") + run_git(destination, "submodule", "deinit", "--all", "--force") + detail = failure("--force") + if not detail: + return + if destination.is_dir() and any( + marker in detail.lower() for marker in tree_delete_failure + ): + _force_rmtree(destination) + detail = failure("--force") + if detail and not destination.is_dir(): + pruned = run_git(primary, "worktree", "prune") + if pruned.returncode == 0: + detail = "" + if detail or destination.is_dir() or destination.resolve() in _registered_worktrees(primary): + raise IsolationError(detail or "git worktree remove failed") + + def _cleanup_reserved_worktree( primary: Path, branch: str, @@ -762,9 +833,7 @@ def retire_member_task( if destination.resolve() in registered: if registered[destination.resolve()] != ref: raise IsolationError(f"member task sidecar is on another branch: {destination}") - removed = run_git(checkout, "worktree", "remove", *(("--force",) if force else ()), str(destination)) - if removed.returncode != 0: - raise IsolationError((removed.stderr or removed.stdout).strip() or "git worktree remove failed") + _remove_worktree(checkout, destination, force) authorization = run_git(checkout, "update-ref", "-d", member_task_authorization_ref(project, task_id)) if authorization.returncode != 0: raise IsolationError((authorization.stderr or authorization.stdout).strip() or "git update-ref failed") @@ -772,6 +841,7 @@ def retire_member_task( deleted = run_git(checkout, "update-ref", "-d", ref, tip.stdout.strip()) if deleted.returncode != 0: raise IsolationError((deleted.stderr or deleted.stdout).strip() or "git update-ref failed") + release_workspace_quietly(checkout) return copy_rejected @@ -1392,8 +1462,9 @@ def retire_member_verify(created: Dict[str, Dict[str, str]]) -> None: """Remove the member siblings of a coordinator verify sidecar; they hold no output.""" for record in reversed(list(created.values())): checkout, destination, branch = _member_verify_sidecar(record) - git_output(checkout, "worktree", "remove", "--force", str(destination)) + _remove_worktree(checkout, destination, True) git_output(checkout, "branch", "-D", branch) + release_workspace_quietly(checkout) def check_member_verify(created: Dict[str, Dict[str, str]]) -> None: @@ -1425,6 +1496,89 @@ def clean_verify(primary: Path, worktree: Path, base: str, branch: str) -> Dict[ return {"worktree": str(worktree), "branch": branch, "base": expected, "removed": removed} +def ensure_verify_sidecar(primary: Path, base: str, branch: str) -> Dict[str, object]: + """Return the named verify sidecar, recreating it when it is missing. + + Finish needs the sidecar prepare-task created. A deleted sidecar must be + rebuildable from the recorded base without a clean primary: the serial + task's own product changes are exactly the tree finish reproduces next, + so this must not reuse isolate_verify's clean-tree gate. + """ + primary = require_directory(primary, "primary worktree") + if worktree_root(primary) != primary: + raise IsolationError(f"primary is not its Git root: {primary}") + resolved_base = require_commit(primary, require_full_sha(base)) + if not branch.startswith(VERIFY_BRANCH_PREFIX): + raise IsolationError("verify sidecar reproduction requires a verify branch") + destination = _sidecar_path_for_branch(primary, branch) + ref = f"refs/heads/{branch}" + registered = _registered_worktrees(primary) + if destination.is_symlink() or (destination.exists() and not destination.is_dir()): + raise IsolationError(f"worktree path already exists: {destination}") + if destination.is_dir(): + try: + # A leftover empty directory (a half-deleted sidecar) clears; + # anything non-empty must be the sidecar itself. + destination.rmdir() + except OSError: + try: + same = ( + worktree_root(destination) == destination + and destination.resolve() in registered + and registered[destination.resolve()] == ref + and common_git_dir(destination) == common_git_dir(primary) + ) + except IsolationError: + same = False + if not same: + raise IsolationError( + f"verify sidecar ownership changed: {destination}" + ) from None + if ( + require_attached(destination) != branch + or current_sha(destination) != resolved_base + ): + raise IsolationError(f"verify sidecar HEAD changed: {destination}") + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + if destination.resolve() in registered: + cleared = run_git(primary, "worktree", "remove", str(destination)) + if cleared.returncode != 0: + run_git(primary, "worktree", "prune") + if destination.resolve() in _registered_worktrees(primary): + raise IsolationError( + (cleared.stderr or cleared.stdout).strip() + or "could not clear stale worktree registration" + ) + existing = run_git(primary, "show-ref", "--verify", "--quiet", ref) + if existing.returncode == 0: + deleted = run_git(primary, "update-ref", "-d", ref) + if deleted.returncode != 0: + raise IsolationError( + (deleted.stderr or deleted.stdout).strip() or "git update-ref failed" + ) + create_named_worktree(primary, branch, destination, resolved_base) + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + + +def reproduce_verify_sidecar(sidecar_path: Path, tree: str) -> None: + """Copy the primary's current task tree into the verify sidecar (clean_verify's inverse).""" + git_output(sidecar_path, "read-tree", "-u", "--reset", tree) + if git_output(sidecar_path, "write-tree") != tree: + raise IsolationError("sidecar reproduction differs from the primary task diff") + + def _split_paths(block: str) -> Set[str]: return {line.strip() for line in block.splitlines() if line.strip()} @@ -1537,14 +1691,24 @@ def _commit_pending( def validate_allowed_changes( - repo: Path, base: str, pending: Set[str], allowed: Set[str] + repo: Path, + base: str, + pending: Set[str], + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> None: - if not pending: + # Tolerated paths (primary bookkeeping) may stay uncommitted; the next + # `.project` checkpoint commits them. + landing = pending - tolerated + if not landing: raise IsolationError("no changes to land") - since_base = committed_paths_since(repo, base) | pending + since_base = committed_paths_since(repo, base) | landing unexpected = sorted(path for path in since_base if path not in allowed) if unexpected: - raise IsolationError("unexpected paths: " + ", ".join(unexpected)) + raise IsolationError( + "unexpected paths: " + ", ".join(unexpected) + + "; landing allows only the task file and its declared files" + ) def _clean_blob_oid(repo: Path, path: str, text: str, *, write: bool) -> str: arguments = ["hash-object"] if write: @@ -1564,21 +1728,23 @@ def commit_allowed_changes( subject: str, allowed: Set[str], body: str, + tolerated: Set[str] = frozenset(), ) -> tuple[str, str, Dict[str, Optional[tuple[str, str]]]]: require_attached(repo) pending = uncommitted_paths(repo) - validate_allowed_changes(repo, base, pending, allowed) + validate_allowed_changes(repo, base, pending, allowed, tolerated) + staging = pending - tolerated reset = run_git(repo, "reset", "-q", "HEAD") if reset.returncode != 0: raise IsolationError("could not clear the index before landing") - for path in sorted(pending): + for path in sorted(staging): added = run_git(repo, "add", "-A", "--", path) if added.returncode != 0: raise IsolationError(f"could not stage {path}") staged = _split_paths( git_output(repo, "diff", "--cached", "--name-only", "--relative") ) - if staged != pending: + if staged != staging: raise IsolationError("staged paths do not match the uncommitted change set") verified_tree = git_output(repo, "write-tree") verified_delta = _tree_delta(repo, base, verified_tree) @@ -1724,6 +1890,17 @@ def activate_task( if task_branch is None: if branch.startswith(TASK_BRANCH_PREFIX): raise IsolationError("parallel task activation requires --task-branch") + live = [ + path + for path, ref in _registered_worktrees(worktree).items() + if ref == f"refs/heads/{task_branch_name(task_id)}" + and path != worktree + ] + if live: + raise IsolationError( + "task has an active parallel worktree; serial activation in " + f"another worktree is refused (activate {live[0]} with --task-branch)" + ) elif task_branch != task_branch_name(task_id) or branch != task_branch: raise IsolationError("task activation branch does not match the task worktree") normalized_task = relative_posix(task_file) @@ -1991,7 +2168,12 @@ def _restore_rejected_commit( def _stamp_and_commit( - worktree: Path, base: str, subject: str, task_file: str, allowed: Set[str] + worktree: Path, + base: str, + subject: str, + task_file: str, + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> tuple[str, Dict[str, Optional[tuple[str, str]]]]: task_path = worktree / task_file task_text, task_mode = _read_task_text(task_path) @@ -2012,14 +2194,14 @@ def _stamp_and_commit( pending = uncommitted_paths(worktree) if stamped != task_bytes: pending.add(task_file) - validate_allowed_changes(worktree, base, pending, allowed) + validate_allowed_changes(worktree, base, pending, allowed, tolerated) original_head = current_sha(worktree) index_tree = git_output(worktree, "write-tree") try: _replace_regular_file(task_path, stamped, task_mode) - body = task_commit_body(task_file, uncommitted_paths(worktree), base) + body = task_commit_body(task_file, uncommitted_paths(worktree) - tolerated, base) commit, verified_tree, verified_delta = commit_allowed_changes( - worktree, base, subject, allowed, body + worktree, base, subject, allowed, body, tolerated ) proof_error = _landing_commit_proof_error( worktree, @@ -2097,6 +2279,7 @@ def land( subject, relative_posix(task_file), allowed, + BOOKKEEPING_PATHS, ) return { "bound_branch": bound, @@ -2391,7 +2574,7 @@ def _landing_retry_error( if task_file not in pending: return "interrupted landing does not retain the task change" try: - validate_allowed_changes(repo, base, pending, contract_paths) + validate_allowed_changes(repo, base, pending, contract_paths, BOOKKEEPING_PATHS) except IsolationError as validation_error: return str(validation_error) return None @@ -4430,6 +4613,7 @@ def retire( or f"could not delete {branch}" ) branch_retired = True + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": branch, @@ -4463,22 +4647,13 @@ def retire( raise IsolationError( "worktree is dirty; pass --force only from the retry-retirement path" ) - remove = run_git( - primary, - "worktree", - "remove", - *(("--force",) if force else ()), - str(resolved_worktree), - ) - if remove.returncode != 0: - raise IsolationError( - (remove.stderr or remove.stdout).strip() or "git worktree remove failed" - ) + _remove_worktree(primary, resolved_worktree, force) deleted = run_git(primary, "branch", "-D", retire_branch) if deleted.returncode != 0: raise IsolationError( (deleted.stderr or deleted.stdout).strip() or f"could not delete {retire_branch}" ) + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": retire_branch, diff --git a/skills/gsd-path-build/scripts/members.py b/skills/gsd-path-build/scripts/members.py index d427c6ee4..7f516c846 100644 --- a/skills/gsd-path-build/scripts/members.py +++ b/skills/gsd-path-build/scripts/members.py @@ -255,6 +255,7 @@ def check_member( coordinator_name: str, checkout: Path, recorded_remote: Optional[str] = None, + refuse_reserved_refs: bool = True, ) -> str: """Refuse a checkout that cannot join; return its origin URL.""" if checkout.is_symlink() or not checkout.is_dir(): @@ -294,13 +295,14 @@ def check_member( raise MembersError(f"member STATE.md is unreadable: {error}") from error if state.milestone is not None and (state.phase, state.status) != ("shipped", "done"): raise MembersError(f"member has an active milestone: {state.milestone}") - prefixes = member_ref_prefixes(coordinator_name) + tuple( - "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS - ) - refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() - colliding = [ref for ref in refs if ref.startswith(prefixes)] - if colliding: - raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) + if refuse_reserved_refs: + prefixes = member_ref_prefixes(coordinator_name) + tuple( + "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS + ) + refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() + colliding = [ref for ref in refs if ref.startswith(prefixes)] + if colliding: + raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) return remote @@ -620,7 +622,7 @@ def validate_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) try: role = member_role(checkout) except MembersError as error: @@ -639,7 +641,7 @@ def repair_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) _refuse_foreign_marker(checkout.resolve(), root, state.project, member["name"]) for member in members: checkout = Path(member["checkout"]).resolve() diff --git a/skills/gsd-path-build/scripts/worktree_paths.py b/skills/gsd-path-build/scripts/worktree_paths.py index 19653ca92..b2cfda32f 100644 --- a/skills/gsd-path-build/scripts/worktree_paths.py +++ b/skills/gsd-path-build/scripts/worktree_paths.py @@ -103,6 +103,33 @@ def _workspace(primary: Path, pin: bool) -> Path: return workspace +def release_workspace(primary: Path) -> bool: + """Forget the pinned placement receipt once no linked worktrees remain. + + Lets GSD_PATH_WORKTREE_ROOT (or a new default) apply again at the next + milestone's first allocation. Returns True when a receipt was removed; + any remaining linked worktree (including one only still registered) keeps + the pin. + """ + primary = primary.resolve() + common = Path(_git(primary, "rev-parse", "--git-common-dir")) + common = (primary / common).resolve() + receipt = common / "gsd-path" / "workspaces" / f"{_identity(primary)}.json" + if not receipt.is_file(): + return False + output, nul_separated = _worktree_list_porcelain(primary) + records = output.split("\0\0") if nul_separated else [ + block for block in output.strip().split("\n\n") if block.strip() + ] + for record in records: + fields = record.split("\0") if nul_separated else record.splitlines() + for field in fields: + if field.startswith("worktree ") and Path(field[len("worktree "):]).resolve() != primary: + return False + receipt.unlink() + return True + + def worktree_path(primary: Path, kind: str, name: str, *, pin: bool = False) -> Path: primary = primary.resolve() if kind not in {"task", "verify", "integrate", "bound"} or not name or Path(name).name != name or name in {".", ".."}: diff --git a/skills/gsd-path-decide/scripts/check_handoffs.py b/skills/gsd-path-decide/scripts/check_handoffs.py index f86fa4306..743f7cdf9 100644 --- a/skills/gsd-path-decide/scripts/check_handoffs.py +++ b/skills/gsd-path-decide/scripts/check_handoffs.py @@ -1386,6 +1386,20 @@ def _task_deps(task_text: str, task_id: str) -> List[str]: return _inline_ids(rendered, "T", f"{task_id} deps") +def _dependency_reachable(graph: Dict[str, List[str]], start: str, target: str) -> bool: + seen: Set[str] = set() + pending = list(graph[start]) + while pending: + task_id = pending.pop() + if task_id in seen: + continue + if task_id == target: + return True + seen.add(task_id) + pending.extend(graph[task_id]) + return False + + def _canonical_repo_path(value: str, label: str) -> str: cleaned = value.strip().strip("`\"'") path = PurePosixPath(cleaned) @@ -1509,6 +1523,12 @@ def _validate_task_graph( and _task_scalar(tasks[right], right, "status") == "done" ): continue + # Dependency-ordered tasks serialize safely; only unordered + # same-wave overlap races on a shared file. + if _dependency_reachable(graph, left, right) or _dependency_reachable( + graph, right, left + ): + continue raise HandoffError( f"same-wave file overlap between {left} and {right}: {', '.join(overlap)}" ) diff --git a/skills/gsd-path-decide/scripts/integration.py b/skills/gsd-path-decide/scripts/integration.py index 0dd2a356b..c1b0971d5 100644 --- a/skills/gsd-path-decide/scripts/integration.py +++ b/skills/gsd-path-decide/scripts/integration.py @@ -9,6 +9,7 @@ import json import re +import contextlib from pathlib import Path, PurePosixPath from typing import Optional @@ -1939,6 +1940,10 @@ def retire_member(coordinator: Path, member: str, archive_path: str, reviewed_he for kind, ref in (("push", bound_ref), ("push", "refs/heads/main"), ("push", f"refs/tags/{tag_name}"), ("delete", bound_ref)): members.clear_authorization(checkout, project, kind, ref) + # The last sidecar retires here at milestone end; isolation's retires also + # release the pin whenever the workspace is idle, so a new root can apply. + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(checkout) return {"member": member, "retired": bound} diff --git a/skills/gsd-path-decide/scripts/isolation.py b/skills/gsd-path-decide/scripts/isolation.py index 50a9ef684..3805f809a 100644 --- a/skills/gsd-path-decide/scripts/isolation.py +++ b/skills/gsd-path-decide/scripts/isolation.py @@ -15,6 +15,7 @@ import io import json import shlex +import shutil import os import re import stat @@ -171,6 +172,12 @@ def sidecar_root(primary: Path, kind: str, name: str, *, pin: bool = False) -> P raise IsolationError(str(error)) from error +def release_workspace_quietly(primary: Path) -> None: + """Best-effort placement hygiene after a sidecar retires; never blocks landing.""" + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(primary) + + def validate_task_id(task_id: str) -> str: if not TASK_ID_RE.fullmatch(task_id): raise IsolationError(f"invalid task id: {task_id}") @@ -477,6 +484,70 @@ def _same_directory(path: Path, expected: os.stat_result) -> bool: return stat.S_ISDIR(current.st_mode) and os.path.samestat(current, expected) +def _long_path(absolute: str) -> str: + if absolute.startswith("\\\\?\\"): + return absolute + if absolute.startswith("\\\\"): + return "\\\\?\\UNC\\" + absolute[2:] + return "\\\\?\\" + absolute + + +def _force_rmtree(path: Path) -> None: + """Delete a sidecar tree directly, tolerating read-only files and long paths.""" + def _writable(function, target, _error): + os.chmod(target, stat.S_IWRITE) + function(target) + + # The \\?\ prefix sidesteps Windows MAX_PATH on deep sidecar trees. + target = Path(_long_path(os.path.abspath(path))) if os.name == "nt" else path + try: + shutil.rmtree(target, onerror=_writable) + except OSError as error: + raise IsolationError(f"could not delete {path}: {error}") from error + + +def _remove_worktree(primary: Path, destination: Path, force: bool) -> None: + """Remove a sidecar worktree, tolerating populated submodules and locked trees. + + git refuses to remove worktrees whose submodules were ever initialized, and + forced removal can still fail deleting deep trees (Windows "Directory not + empty"). A dirty sidecar keeps git's refusal unless `force`, so forced + retries only run for a provably clean tree. The direct-delete fallback runs + only for git's tree-deletion failures, so any other git error leaves the + sidecar intact for diagnosis. + """ + tree_delete_failure = ("failed to delete", "directory not empty", "permission denied") + + def failure(*flags: str) -> str: + result = run_git(primary, "worktree", "remove", *flags, str(destination)) + return (result.stderr or result.stdout).strip() if result.returncode else "" + + detail = failure("--force") if force else failure() + if not detail: + return + if not force and destination.is_dir(): + status = run_git( + destination, "status", "--porcelain", "--untracked-files=all" + ) + if status.returncode != 0 or status.stdout.strip(): + raise IsolationError(detail or "git worktree remove failed") + run_git(destination, "submodule", "deinit", "--all", "--force") + detail = failure("--force") + if not detail: + return + if destination.is_dir() and any( + marker in detail.lower() for marker in tree_delete_failure + ): + _force_rmtree(destination) + detail = failure("--force") + if detail and not destination.is_dir(): + pruned = run_git(primary, "worktree", "prune") + if pruned.returncode == 0: + detail = "" + if detail or destination.is_dir() or destination.resolve() in _registered_worktrees(primary): + raise IsolationError(detail or "git worktree remove failed") + + def _cleanup_reserved_worktree( primary: Path, branch: str, @@ -762,9 +833,7 @@ def retire_member_task( if destination.resolve() in registered: if registered[destination.resolve()] != ref: raise IsolationError(f"member task sidecar is on another branch: {destination}") - removed = run_git(checkout, "worktree", "remove", *(("--force",) if force else ()), str(destination)) - if removed.returncode != 0: - raise IsolationError((removed.stderr or removed.stdout).strip() or "git worktree remove failed") + _remove_worktree(checkout, destination, force) authorization = run_git(checkout, "update-ref", "-d", member_task_authorization_ref(project, task_id)) if authorization.returncode != 0: raise IsolationError((authorization.stderr or authorization.stdout).strip() or "git update-ref failed") @@ -772,6 +841,7 @@ def retire_member_task( deleted = run_git(checkout, "update-ref", "-d", ref, tip.stdout.strip()) if deleted.returncode != 0: raise IsolationError((deleted.stderr or deleted.stdout).strip() or "git update-ref failed") + release_workspace_quietly(checkout) return copy_rejected @@ -1392,8 +1462,9 @@ def retire_member_verify(created: Dict[str, Dict[str, str]]) -> None: """Remove the member siblings of a coordinator verify sidecar; they hold no output.""" for record in reversed(list(created.values())): checkout, destination, branch = _member_verify_sidecar(record) - git_output(checkout, "worktree", "remove", "--force", str(destination)) + _remove_worktree(checkout, destination, True) git_output(checkout, "branch", "-D", branch) + release_workspace_quietly(checkout) def check_member_verify(created: Dict[str, Dict[str, str]]) -> None: @@ -1425,6 +1496,89 @@ def clean_verify(primary: Path, worktree: Path, base: str, branch: str) -> Dict[ return {"worktree": str(worktree), "branch": branch, "base": expected, "removed": removed} +def ensure_verify_sidecar(primary: Path, base: str, branch: str) -> Dict[str, object]: + """Return the named verify sidecar, recreating it when it is missing. + + Finish needs the sidecar prepare-task created. A deleted sidecar must be + rebuildable from the recorded base without a clean primary: the serial + task's own product changes are exactly the tree finish reproduces next, + so this must not reuse isolate_verify's clean-tree gate. + """ + primary = require_directory(primary, "primary worktree") + if worktree_root(primary) != primary: + raise IsolationError(f"primary is not its Git root: {primary}") + resolved_base = require_commit(primary, require_full_sha(base)) + if not branch.startswith(VERIFY_BRANCH_PREFIX): + raise IsolationError("verify sidecar reproduction requires a verify branch") + destination = _sidecar_path_for_branch(primary, branch) + ref = f"refs/heads/{branch}" + registered = _registered_worktrees(primary) + if destination.is_symlink() or (destination.exists() and not destination.is_dir()): + raise IsolationError(f"worktree path already exists: {destination}") + if destination.is_dir(): + try: + # A leftover empty directory (a half-deleted sidecar) clears; + # anything non-empty must be the sidecar itself. + destination.rmdir() + except OSError: + try: + same = ( + worktree_root(destination) == destination + and destination.resolve() in registered + and registered[destination.resolve()] == ref + and common_git_dir(destination) == common_git_dir(primary) + ) + except IsolationError: + same = False + if not same: + raise IsolationError( + f"verify sidecar ownership changed: {destination}" + ) from None + if ( + require_attached(destination) != branch + or current_sha(destination) != resolved_base + ): + raise IsolationError(f"verify sidecar HEAD changed: {destination}") + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + if destination.resolve() in registered: + cleared = run_git(primary, "worktree", "remove", str(destination)) + if cleared.returncode != 0: + run_git(primary, "worktree", "prune") + if destination.resolve() in _registered_worktrees(primary): + raise IsolationError( + (cleared.stderr or cleared.stdout).strip() + or "could not clear stale worktree registration" + ) + existing = run_git(primary, "show-ref", "--verify", "--quiet", ref) + if existing.returncode == 0: + deleted = run_git(primary, "update-ref", "-d", ref) + if deleted.returncode != 0: + raise IsolationError( + (deleted.stderr or deleted.stdout).strip() or "git update-ref failed" + ) + create_named_worktree(primary, branch, destination, resolved_base) + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + + +def reproduce_verify_sidecar(sidecar_path: Path, tree: str) -> None: + """Copy the primary's current task tree into the verify sidecar (clean_verify's inverse).""" + git_output(sidecar_path, "read-tree", "-u", "--reset", tree) + if git_output(sidecar_path, "write-tree") != tree: + raise IsolationError("sidecar reproduction differs from the primary task diff") + + def _split_paths(block: str) -> Set[str]: return {line.strip() for line in block.splitlines() if line.strip()} @@ -1537,14 +1691,24 @@ def _commit_pending( def validate_allowed_changes( - repo: Path, base: str, pending: Set[str], allowed: Set[str] + repo: Path, + base: str, + pending: Set[str], + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> None: - if not pending: + # Tolerated paths (primary bookkeeping) may stay uncommitted; the next + # `.project` checkpoint commits them. + landing = pending - tolerated + if not landing: raise IsolationError("no changes to land") - since_base = committed_paths_since(repo, base) | pending + since_base = committed_paths_since(repo, base) | landing unexpected = sorted(path for path in since_base if path not in allowed) if unexpected: - raise IsolationError("unexpected paths: " + ", ".join(unexpected)) + raise IsolationError( + "unexpected paths: " + ", ".join(unexpected) + + "; landing allows only the task file and its declared files" + ) def _clean_blob_oid(repo: Path, path: str, text: str, *, write: bool) -> str: arguments = ["hash-object"] if write: @@ -1564,21 +1728,23 @@ def commit_allowed_changes( subject: str, allowed: Set[str], body: str, + tolerated: Set[str] = frozenset(), ) -> tuple[str, str, Dict[str, Optional[tuple[str, str]]]]: require_attached(repo) pending = uncommitted_paths(repo) - validate_allowed_changes(repo, base, pending, allowed) + validate_allowed_changes(repo, base, pending, allowed, tolerated) + staging = pending - tolerated reset = run_git(repo, "reset", "-q", "HEAD") if reset.returncode != 0: raise IsolationError("could not clear the index before landing") - for path in sorted(pending): + for path in sorted(staging): added = run_git(repo, "add", "-A", "--", path) if added.returncode != 0: raise IsolationError(f"could not stage {path}") staged = _split_paths( git_output(repo, "diff", "--cached", "--name-only", "--relative") ) - if staged != pending: + if staged != staging: raise IsolationError("staged paths do not match the uncommitted change set") verified_tree = git_output(repo, "write-tree") verified_delta = _tree_delta(repo, base, verified_tree) @@ -1724,6 +1890,17 @@ def activate_task( if task_branch is None: if branch.startswith(TASK_BRANCH_PREFIX): raise IsolationError("parallel task activation requires --task-branch") + live = [ + path + for path, ref in _registered_worktrees(worktree).items() + if ref == f"refs/heads/{task_branch_name(task_id)}" + and path != worktree + ] + if live: + raise IsolationError( + "task has an active parallel worktree; serial activation in " + f"another worktree is refused (activate {live[0]} with --task-branch)" + ) elif task_branch != task_branch_name(task_id) or branch != task_branch: raise IsolationError("task activation branch does not match the task worktree") normalized_task = relative_posix(task_file) @@ -1991,7 +2168,12 @@ def _restore_rejected_commit( def _stamp_and_commit( - worktree: Path, base: str, subject: str, task_file: str, allowed: Set[str] + worktree: Path, + base: str, + subject: str, + task_file: str, + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> tuple[str, Dict[str, Optional[tuple[str, str]]]]: task_path = worktree / task_file task_text, task_mode = _read_task_text(task_path) @@ -2012,14 +2194,14 @@ def _stamp_and_commit( pending = uncommitted_paths(worktree) if stamped != task_bytes: pending.add(task_file) - validate_allowed_changes(worktree, base, pending, allowed) + validate_allowed_changes(worktree, base, pending, allowed, tolerated) original_head = current_sha(worktree) index_tree = git_output(worktree, "write-tree") try: _replace_regular_file(task_path, stamped, task_mode) - body = task_commit_body(task_file, uncommitted_paths(worktree), base) + body = task_commit_body(task_file, uncommitted_paths(worktree) - tolerated, base) commit, verified_tree, verified_delta = commit_allowed_changes( - worktree, base, subject, allowed, body + worktree, base, subject, allowed, body, tolerated ) proof_error = _landing_commit_proof_error( worktree, @@ -2097,6 +2279,7 @@ def land( subject, relative_posix(task_file), allowed, + BOOKKEEPING_PATHS, ) return { "bound_branch": bound, @@ -2391,7 +2574,7 @@ def _landing_retry_error( if task_file not in pending: return "interrupted landing does not retain the task change" try: - validate_allowed_changes(repo, base, pending, contract_paths) + validate_allowed_changes(repo, base, pending, contract_paths, BOOKKEEPING_PATHS) except IsolationError as validation_error: return str(validation_error) return None @@ -4430,6 +4613,7 @@ def retire( or f"could not delete {branch}" ) branch_retired = True + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": branch, @@ -4463,22 +4647,13 @@ def retire( raise IsolationError( "worktree is dirty; pass --force only from the retry-retirement path" ) - remove = run_git( - primary, - "worktree", - "remove", - *(("--force",) if force else ()), - str(resolved_worktree), - ) - if remove.returncode != 0: - raise IsolationError( - (remove.stderr or remove.stdout).strip() or "git worktree remove failed" - ) + _remove_worktree(primary, resolved_worktree, force) deleted = run_git(primary, "branch", "-D", retire_branch) if deleted.returncode != 0: raise IsolationError( (deleted.stderr or deleted.stdout).strip() or f"could not delete {retire_branch}" ) + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": retire_branch, diff --git a/skills/gsd-path-decide/scripts/members.py b/skills/gsd-path-decide/scripts/members.py index d427c6ee4..7f516c846 100644 --- a/skills/gsd-path-decide/scripts/members.py +++ b/skills/gsd-path-decide/scripts/members.py @@ -255,6 +255,7 @@ def check_member( coordinator_name: str, checkout: Path, recorded_remote: Optional[str] = None, + refuse_reserved_refs: bool = True, ) -> str: """Refuse a checkout that cannot join; return its origin URL.""" if checkout.is_symlink() or not checkout.is_dir(): @@ -294,13 +295,14 @@ def check_member( raise MembersError(f"member STATE.md is unreadable: {error}") from error if state.milestone is not None and (state.phase, state.status) != ("shipped", "done"): raise MembersError(f"member has an active milestone: {state.milestone}") - prefixes = member_ref_prefixes(coordinator_name) + tuple( - "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS - ) - refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() - colliding = [ref for ref in refs if ref.startswith(prefixes)] - if colliding: - raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) + if refuse_reserved_refs: + prefixes = member_ref_prefixes(coordinator_name) + tuple( + "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS + ) + refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() + colliding = [ref for ref in refs if ref.startswith(prefixes)] + if colliding: + raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) return remote @@ -620,7 +622,7 @@ def validate_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) try: role = member_role(checkout) except MembersError as error: @@ -639,7 +641,7 @@ def repair_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) _refuse_foreign_marker(checkout.resolve(), root, state.project, member["name"]) for member in members: checkout = Path(member["checkout"]).resolve() diff --git a/skills/gsd-path-decide/scripts/worktree_paths.py b/skills/gsd-path-decide/scripts/worktree_paths.py index 19653ca92..b2cfda32f 100644 --- a/skills/gsd-path-decide/scripts/worktree_paths.py +++ b/skills/gsd-path-decide/scripts/worktree_paths.py @@ -103,6 +103,33 @@ def _workspace(primary: Path, pin: bool) -> Path: return workspace +def release_workspace(primary: Path) -> bool: + """Forget the pinned placement receipt once no linked worktrees remain. + + Lets GSD_PATH_WORKTREE_ROOT (or a new default) apply again at the next + milestone's first allocation. Returns True when a receipt was removed; + any remaining linked worktree (including one only still registered) keeps + the pin. + """ + primary = primary.resolve() + common = Path(_git(primary, "rev-parse", "--git-common-dir")) + common = (primary / common).resolve() + receipt = common / "gsd-path" / "workspaces" / f"{_identity(primary)}.json" + if not receipt.is_file(): + return False + output, nul_separated = _worktree_list_porcelain(primary) + records = output.split("\0\0") if nul_separated else [ + block for block in output.strip().split("\n\n") if block.strip() + ] + for record in records: + fields = record.split("\0") if nul_separated else record.splitlines() + for field in fields: + if field.startswith("worktree ") and Path(field[len("worktree "):]).resolve() != primary: + return False + receipt.unlink() + return True + + def worktree_path(primary: Path, kind: str, name: str, *, pin: bool = False) -> Path: primary = primary.resolve() if kind not in {"task", "verify", "integrate", "bound"} or not name or Path(name).name != name or name in {".", ".."}: diff --git a/skills/gsd-path-define/scripts/check_handoffs.py b/skills/gsd-path-define/scripts/check_handoffs.py index f86fa4306..743f7cdf9 100644 --- a/skills/gsd-path-define/scripts/check_handoffs.py +++ b/skills/gsd-path-define/scripts/check_handoffs.py @@ -1386,6 +1386,20 @@ def _task_deps(task_text: str, task_id: str) -> List[str]: return _inline_ids(rendered, "T", f"{task_id} deps") +def _dependency_reachable(graph: Dict[str, List[str]], start: str, target: str) -> bool: + seen: Set[str] = set() + pending = list(graph[start]) + while pending: + task_id = pending.pop() + if task_id in seen: + continue + if task_id == target: + return True + seen.add(task_id) + pending.extend(graph[task_id]) + return False + + def _canonical_repo_path(value: str, label: str) -> str: cleaned = value.strip().strip("`\"'") path = PurePosixPath(cleaned) @@ -1509,6 +1523,12 @@ def _validate_task_graph( and _task_scalar(tasks[right], right, "status") == "done" ): continue + # Dependency-ordered tasks serialize safely; only unordered + # same-wave overlap races on a shared file. + if _dependency_reachable(graph, left, right) or _dependency_reachable( + graph, right, left + ): + continue raise HandoffError( f"same-wave file overlap between {left} and {right}: {', '.join(overlap)}" ) diff --git a/skills/gsd-path-define/scripts/integration.py b/skills/gsd-path-define/scripts/integration.py index 0dd2a356b..c1b0971d5 100644 --- a/skills/gsd-path-define/scripts/integration.py +++ b/skills/gsd-path-define/scripts/integration.py @@ -9,6 +9,7 @@ import json import re +import contextlib from pathlib import Path, PurePosixPath from typing import Optional @@ -1939,6 +1940,10 @@ def retire_member(coordinator: Path, member: str, archive_path: str, reviewed_he for kind, ref in (("push", bound_ref), ("push", "refs/heads/main"), ("push", f"refs/tags/{tag_name}"), ("delete", bound_ref)): members.clear_authorization(checkout, project, kind, ref) + # The last sidecar retires here at milestone end; isolation's retires also + # release the pin whenever the workspace is idle, so a new root can apply. + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(checkout) return {"member": member, "retired": bound} diff --git a/skills/gsd-path-define/scripts/isolation.py b/skills/gsd-path-define/scripts/isolation.py index 50a9ef684..3805f809a 100644 --- a/skills/gsd-path-define/scripts/isolation.py +++ b/skills/gsd-path-define/scripts/isolation.py @@ -15,6 +15,7 @@ import io import json import shlex +import shutil import os import re import stat @@ -171,6 +172,12 @@ def sidecar_root(primary: Path, kind: str, name: str, *, pin: bool = False) -> P raise IsolationError(str(error)) from error +def release_workspace_quietly(primary: Path) -> None: + """Best-effort placement hygiene after a sidecar retires; never blocks landing.""" + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(primary) + + def validate_task_id(task_id: str) -> str: if not TASK_ID_RE.fullmatch(task_id): raise IsolationError(f"invalid task id: {task_id}") @@ -477,6 +484,70 @@ def _same_directory(path: Path, expected: os.stat_result) -> bool: return stat.S_ISDIR(current.st_mode) and os.path.samestat(current, expected) +def _long_path(absolute: str) -> str: + if absolute.startswith("\\\\?\\"): + return absolute + if absolute.startswith("\\\\"): + return "\\\\?\\UNC\\" + absolute[2:] + return "\\\\?\\" + absolute + + +def _force_rmtree(path: Path) -> None: + """Delete a sidecar tree directly, tolerating read-only files and long paths.""" + def _writable(function, target, _error): + os.chmod(target, stat.S_IWRITE) + function(target) + + # The \\?\ prefix sidesteps Windows MAX_PATH on deep sidecar trees. + target = Path(_long_path(os.path.abspath(path))) if os.name == "nt" else path + try: + shutil.rmtree(target, onerror=_writable) + except OSError as error: + raise IsolationError(f"could not delete {path}: {error}") from error + + +def _remove_worktree(primary: Path, destination: Path, force: bool) -> None: + """Remove a sidecar worktree, tolerating populated submodules and locked trees. + + git refuses to remove worktrees whose submodules were ever initialized, and + forced removal can still fail deleting deep trees (Windows "Directory not + empty"). A dirty sidecar keeps git's refusal unless `force`, so forced + retries only run for a provably clean tree. The direct-delete fallback runs + only for git's tree-deletion failures, so any other git error leaves the + sidecar intact for diagnosis. + """ + tree_delete_failure = ("failed to delete", "directory not empty", "permission denied") + + def failure(*flags: str) -> str: + result = run_git(primary, "worktree", "remove", *flags, str(destination)) + return (result.stderr or result.stdout).strip() if result.returncode else "" + + detail = failure("--force") if force else failure() + if not detail: + return + if not force and destination.is_dir(): + status = run_git( + destination, "status", "--porcelain", "--untracked-files=all" + ) + if status.returncode != 0 or status.stdout.strip(): + raise IsolationError(detail or "git worktree remove failed") + run_git(destination, "submodule", "deinit", "--all", "--force") + detail = failure("--force") + if not detail: + return + if destination.is_dir() and any( + marker in detail.lower() for marker in tree_delete_failure + ): + _force_rmtree(destination) + detail = failure("--force") + if detail and not destination.is_dir(): + pruned = run_git(primary, "worktree", "prune") + if pruned.returncode == 0: + detail = "" + if detail or destination.is_dir() or destination.resolve() in _registered_worktrees(primary): + raise IsolationError(detail or "git worktree remove failed") + + def _cleanup_reserved_worktree( primary: Path, branch: str, @@ -762,9 +833,7 @@ def retire_member_task( if destination.resolve() in registered: if registered[destination.resolve()] != ref: raise IsolationError(f"member task sidecar is on another branch: {destination}") - removed = run_git(checkout, "worktree", "remove", *(("--force",) if force else ()), str(destination)) - if removed.returncode != 0: - raise IsolationError((removed.stderr or removed.stdout).strip() or "git worktree remove failed") + _remove_worktree(checkout, destination, force) authorization = run_git(checkout, "update-ref", "-d", member_task_authorization_ref(project, task_id)) if authorization.returncode != 0: raise IsolationError((authorization.stderr or authorization.stdout).strip() or "git update-ref failed") @@ -772,6 +841,7 @@ def retire_member_task( deleted = run_git(checkout, "update-ref", "-d", ref, tip.stdout.strip()) if deleted.returncode != 0: raise IsolationError((deleted.stderr or deleted.stdout).strip() or "git update-ref failed") + release_workspace_quietly(checkout) return copy_rejected @@ -1392,8 +1462,9 @@ def retire_member_verify(created: Dict[str, Dict[str, str]]) -> None: """Remove the member siblings of a coordinator verify sidecar; they hold no output.""" for record in reversed(list(created.values())): checkout, destination, branch = _member_verify_sidecar(record) - git_output(checkout, "worktree", "remove", "--force", str(destination)) + _remove_worktree(checkout, destination, True) git_output(checkout, "branch", "-D", branch) + release_workspace_quietly(checkout) def check_member_verify(created: Dict[str, Dict[str, str]]) -> None: @@ -1425,6 +1496,89 @@ def clean_verify(primary: Path, worktree: Path, base: str, branch: str) -> Dict[ return {"worktree": str(worktree), "branch": branch, "base": expected, "removed": removed} +def ensure_verify_sidecar(primary: Path, base: str, branch: str) -> Dict[str, object]: + """Return the named verify sidecar, recreating it when it is missing. + + Finish needs the sidecar prepare-task created. A deleted sidecar must be + rebuildable from the recorded base without a clean primary: the serial + task's own product changes are exactly the tree finish reproduces next, + so this must not reuse isolate_verify's clean-tree gate. + """ + primary = require_directory(primary, "primary worktree") + if worktree_root(primary) != primary: + raise IsolationError(f"primary is not its Git root: {primary}") + resolved_base = require_commit(primary, require_full_sha(base)) + if not branch.startswith(VERIFY_BRANCH_PREFIX): + raise IsolationError("verify sidecar reproduction requires a verify branch") + destination = _sidecar_path_for_branch(primary, branch) + ref = f"refs/heads/{branch}" + registered = _registered_worktrees(primary) + if destination.is_symlink() or (destination.exists() and not destination.is_dir()): + raise IsolationError(f"worktree path already exists: {destination}") + if destination.is_dir(): + try: + # A leftover empty directory (a half-deleted sidecar) clears; + # anything non-empty must be the sidecar itself. + destination.rmdir() + except OSError: + try: + same = ( + worktree_root(destination) == destination + and destination.resolve() in registered + and registered[destination.resolve()] == ref + and common_git_dir(destination) == common_git_dir(primary) + ) + except IsolationError: + same = False + if not same: + raise IsolationError( + f"verify sidecar ownership changed: {destination}" + ) from None + if ( + require_attached(destination) != branch + or current_sha(destination) != resolved_base + ): + raise IsolationError(f"verify sidecar HEAD changed: {destination}") + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + if destination.resolve() in registered: + cleared = run_git(primary, "worktree", "remove", str(destination)) + if cleared.returncode != 0: + run_git(primary, "worktree", "prune") + if destination.resolve() in _registered_worktrees(primary): + raise IsolationError( + (cleared.stderr or cleared.stdout).strip() + or "could not clear stale worktree registration" + ) + existing = run_git(primary, "show-ref", "--verify", "--quiet", ref) + if existing.returncode == 0: + deleted = run_git(primary, "update-ref", "-d", ref) + if deleted.returncode != 0: + raise IsolationError( + (deleted.stderr or deleted.stdout).strip() or "git update-ref failed" + ) + create_named_worktree(primary, branch, destination, resolved_base) + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + + +def reproduce_verify_sidecar(sidecar_path: Path, tree: str) -> None: + """Copy the primary's current task tree into the verify sidecar (clean_verify's inverse).""" + git_output(sidecar_path, "read-tree", "-u", "--reset", tree) + if git_output(sidecar_path, "write-tree") != tree: + raise IsolationError("sidecar reproduction differs from the primary task diff") + + def _split_paths(block: str) -> Set[str]: return {line.strip() for line in block.splitlines() if line.strip()} @@ -1537,14 +1691,24 @@ def _commit_pending( def validate_allowed_changes( - repo: Path, base: str, pending: Set[str], allowed: Set[str] + repo: Path, + base: str, + pending: Set[str], + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> None: - if not pending: + # Tolerated paths (primary bookkeeping) may stay uncommitted; the next + # `.project` checkpoint commits them. + landing = pending - tolerated + if not landing: raise IsolationError("no changes to land") - since_base = committed_paths_since(repo, base) | pending + since_base = committed_paths_since(repo, base) | landing unexpected = sorted(path for path in since_base if path not in allowed) if unexpected: - raise IsolationError("unexpected paths: " + ", ".join(unexpected)) + raise IsolationError( + "unexpected paths: " + ", ".join(unexpected) + + "; landing allows only the task file and its declared files" + ) def _clean_blob_oid(repo: Path, path: str, text: str, *, write: bool) -> str: arguments = ["hash-object"] if write: @@ -1564,21 +1728,23 @@ def commit_allowed_changes( subject: str, allowed: Set[str], body: str, + tolerated: Set[str] = frozenset(), ) -> tuple[str, str, Dict[str, Optional[tuple[str, str]]]]: require_attached(repo) pending = uncommitted_paths(repo) - validate_allowed_changes(repo, base, pending, allowed) + validate_allowed_changes(repo, base, pending, allowed, tolerated) + staging = pending - tolerated reset = run_git(repo, "reset", "-q", "HEAD") if reset.returncode != 0: raise IsolationError("could not clear the index before landing") - for path in sorted(pending): + for path in sorted(staging): added = run_git(repo, "add", "-A", "--", path) if added.returncode != 0: raise IsolationError(f"could not stage {path}") staged = _split_paths( git_output(repo, "diff", "--cached", "--name-only", "--relative") ) - if staged != pending: + if staged != staging: raise IsolationError("staged paths do not match the uncommitted change set") verified_tree = git_output(repo, "write-tree") verified_delta = _tree_delta(repo, base, verified_tree) @@ -1724,6 +1890,17 @@ def activate_task( if task_branch is None: if branch.startswith(TASK_BRANCH_PREFIX): raise IsolationError("parallel task activation requires --task-branch") + live = [ + path + for path, ref in _registered_worktrees(worktree).items() + if ref == f"refs/heads/{task_branch_name(task_id)}" + and path != worktree + ] + if live: + raise IsolationError( + "task has an active parallel worktree; serial activation in " + f"another worktree is refused (activate {live[0]} with --task-branch)" + ) elif task_branch != task_branch_name(task_id) or branch != task_branch: raise IsolationError("task activation branch does not match the task worktree") normalized_task = relative_posix(task_file) @@ -1991,7 +2168,12 @@ def _restore_rejected_commit( def _stamp_and_commit( - worktree: Path, base: str, subject: str, task_file: str, allowed: Set[str] + worktree: Path, + base: str, + subject: str, + task_file: str, + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> tuple[str, Dict[str, Optional[tuple[str, str]]]]: task_path = worktree / task_file task_text, task_mode = _read_task_text(task_path) @@ -2012,14 +2194,14 @@ def _stamp_and_commit( pending = uncommitted_paths(worktree) if stamped != task_bytes: pending.add(task_file) - validate_allowed_changes(worktree, base, pending, allowed) + validate_allowed_changes(worktree, base, pending, allowed, tolerated) original_head = current_sha(worktree) index_tree = git_output(worktree, "write-tree") try: _replace_regular_file(task_path, stamped, task_mode) - body = task_commit_body(task_file, uncommitted_paths(worktree), base) + body = task_commit_body(task_file, uncommitted_paths(worktree) - tolerated, base) commit, verified_tree, verified_delta = commit_allowed_changes( - worktree, base, subject, allowed, body + worktree, base, subject, allowed, body, tolerated ) proof_error = _landing_commit_proof_error( worktree, @@ -2097,6 +2279,7 @@ def land( subject, relative_posix(task_file), allowed, + BOOKKEEPING_PATHS, ) return { "bound_branch": bound, @@ -2391,7 +2574,7 @@ def _landing_retry_error( if task_file not in pending: return "interrupted landing does not retain the task change" try: - validate_allowed_changes(repo, base, pending, contract_paths) + validate_allowed_changes(repo, base, pending, contract_paths, BOOKKEEPING_PATHS) except IsolationError as validation_error: return str(validation_error) return None @@ -4430,6 +4613,7 @@ def retire( or f"could not delete {branch}" ) branch_retired = True + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": branch, @@ -4463,22 +4647,13 @@ def retire( raise IsolationError( "worktree is dirty; pass --force only from the retry-retirement path" ) - remove = run_git( - primary, - "worktree", - "remove", - *(("--force",) if force else ()), - str(resolved_worktree), - ) - if remove.returncode != 0: - raise IsolationError( - (remove.stderr or remove.stdout).strip() or "git worktree remove failed" - ) + _remove_worktree(primary, resolved_worktree, force) deleted = run_git(primary, "branch", "-D", retire_branch) if deleted.returncode != 0: raise IsolationError( (deleted.stderr or deleted.stdout).strip() or f"could not delete {retire_branch}" ) + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": retire_branch, diff --git a/skills/gsd-path-define/scripts/members.py b/skills/gsd-path-define/scripts/members.py index d427c6ee4..7f516c846 100644 --- a/skills/gsd-path-define/scripts/members.py +++ b/skills/gsd-path-define/scripts/members.py @@ -255,6 +255,7 @@ def check_member( coordinator_name: str, checkout: Path, recorded_remote: Optional[str] = None, + refuse_reserved_refs: bool = True, ) -> str: """Refuse a checkout that cannot join; return its origin URL.""" if checkout.is_symlink() or not checkout.is_dir(): @@ -294,13 +295,14 @@ def check_member( raise MembersError(f"member STATE.md is unreadable: {error}") from error if state.milestone is not None and (state.phase, state.status) != ("shipped", "done"): raise MembersError(f"member has an active milestone: {state.milestone}") - prefixes = member_ref_prefixes(coordinator_name) + tuple( - "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS - ) - refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() - colliding = [ref for ref in refs if ref.startswith(prefixes)] - if colliding: - raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) + if refuse_reserved_refs: + prefixes = member_ref_prefixes(coordinator_name) + tuple( + "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS + ) + refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() + colliding = [ref for ref in refs if ref.startswith(prefixes)] + if colliding: + raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) return remote @@ -620,7 +622,7 @@ def validate_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) try: role = member_role(checkout) except MembersError as error: @@ -639,7 +641,7 @@ def repair_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) _refuse_foreign_marker(checkout.resolve(), root, state.project, member["name"]) for member in members: checkout = Path(member["checkout"]).resolve() diff --git a/skills/gsd-path-define/scripts/worktree_paths.py b/skills/gsd-path-define/scripts/worktree_paths.py index 19653ca92..b2cfda32f 100644 --- a/skills/gsd-path-define/scripts/worktree_paths.py +++ b/skills/gsd-path-define/scripts/worktree_paths.py @@ -103,6 +103,33 @@ def _workspace(primary: Path, pin: bool) -> Path: return workspace +def release_workspace(primary: Path) -> bool: + """Forget the pinned placement receipt once no linked worktrees remain. + + Lets GSD_PATH_WORKTREE_ROOT (or a new default) apply again at the next + milestone's first allocation. Returns True when a receipt was removed; + any remaining linked worktree (including one only still registered) keeps + the pin. + """ + primary = primary.resolve() + common = Path(_git(primary, "rev-parse", "--git-common-dir")) + common = (primary / common).resolve() + receipt = common / "gsd-path" / "workspaces" / f"{_identity(primary)}.json" + if not receipt.is_file(): + return False + output, nul_separated = _worktree_list_porcelain(primary) + records = output.split("\0\0") if nul_separated else [ + block for block in output.strip().split("\n\n") if block.strip() + ] + for record in records: + fields = record.split("\0") if nul_separated else record.splitlines() + for field in fields: + if field.startswith("worktree ") and Path(field[len("worktree "):]).resolve() != primary: + return False + receipt.unlink() + return True + + def worktree_path(primary: Path, kind: str, name: str, *, pin: bool = False) -> Path: primary = primary.resolve() if kind not in {"task", "verify", "integrate", "bound"} or not name or Path(name).name != name or name in {".", ".."}: diff --git a/skills/gsd-path-discuss/scripts/worktree_paths.py b/skills/gsd-path-discuss/scripts/worktree_paths.py index 19653ca92..b2cfda32f 100644 --- a/skills/gsd-path-discuss/scripts/worktree_paths.py +++ b/skills/gsd-path-discuss/scripts/worktree_paths.py @@ -103,6 +103,33 @@ def _workspace(primary: Path, pin: bool) -> Path: return workspace +def release_workspace(primary: Path) -> bool: + """Forget the pinned placement receipt once no linked worktrees remain. + + Lets GSD_PATH_WORKTREE_ROOT (or a new default) apply again at the next + milestone's first allocation. Returns True when a receipt was removed; + any remaining linked worktree (including one only still registered) keeps + the pin. + """ + primary = primary.resolve() + common = Path(_git(primary, "rev-parse", "--git-common-dir")) + common = (primary / common).resolve() + receipt = common / "gsd-path" / "workspaces" / f"{_identity(primary)}.json" + if not receipt.is_file(): + return False + output, nul_separated = _worktree_list_porcelain(primary) + records = output.split("\0\0") if nul_separated else [ + block for block in output.strip().split("\n\n") if block.strip() + ] + for record in records: + fields = record.split("\0") if nul_separated else record.splitlines() + for field in fields: + if field.startswith("worktree ") and Path(field[len("worktree "):]).resolve() != primary: + return False + receipt.unlink() + return True + + def worktree_path(primary: Path, kind: str, name: str, *, pin: bool = False) -> Path: primary = primary.resolve() if kind not in {"task", "verify", "integrate", "bound"} or not name or Path(name).name != name or name in {".", ".."}: diff --git a/skills/gsd-path-docs-audit/scripts/check_handoffs.py b/skills/gsd-path-docs-audit/scripts/check_handoffs.py index f86fa4306..743f7cdf9 100644 --- a/skills/gsd-path-docs-audit/scripts/check_handoffs.py +++ b/skills/gsd-path-docs-audit/scripts/check_handoffs.py @@ -1386,6 +1386,20 @@ def _task_deps(task_text: str, task_id: str) -> List[str]: return _inline_ids(rendered, "T", f"{task_id} deps") +def _dependency_reachable(graph: Dict[str, List[str]], start: str, target: str) -> bool: + seen: Set[str] = set() + pending = list(graph[start]) + while pending: + task_id = pending.pop() + if task_id in seen: + continue + if task_id == target: + return True + seen.add(task_id) + pending.extend(graph[task_id]) + return False + + def _canonical_repo_path(value: str, label: str) -> str: cleaned = value.strip().strip("`\"'") path = PurePosixPath(cleaned) @@ -1509,6 +1523,12 @@ def _validate_task_graph( and _task_scalar(tasks[right], right, "status") == "done" ): continue + # Dependency-ordered tasks serialize safely; only unordered + # same-wave overlap races on a shared file. + if _dependency_reachable(graph, left, right) or _dependency_reachable( + graph, right, left + ): + continue raise HandoffError( f"same-wave file overlap between {left} and {right}: {', '.join(overlap)}" ) diff --git a/skills/gsd-path-docs-audit/scripts/integration.py b/skills/gsd-path-docs-audit/scripts/integration.py index 0dd2a356b..c1b0971d5 100644 --- a/skills/gsd-path-docs-audit/scripts/integration.py +++ b/skills/gsd-path-docs-audit/scripts/integration.py @@ -9,6 +9,7 @@ import json import re +import contextlib from pathlib import Path, PurePosixPath from typing import Optional @@ -1939,6 +1940,10 @@ def retire_member(coordinator: Path, member: str, archive_path: str, reviewed_he for kind, ref in (("push", bound_ref), ("push", "refs/heads/main"), ("push", f"refs/tags/{tag_name}"), ("delete", bound_ref)): members.clear_authorization(checkout, project, kind, ref) + # The last sidecar retires here at milestone end; isolation's retires also + # release the pin whenever the workspace is idle, so a new root can apply. + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(checkout) return {"member": member, "retired": bound} diff --git a/skills/gsd-path-docs-audit/scripts/isolation.py b/skills/gsd-path-docs-audit/scripts/isolation.py index 50a9ef684..3805f809a 100644 --- a/skills/gsd-path-docs-audit/scripts/isolation.py +++ b/skills/gsd-path-docs-audit/scripts/isolation.py @@ -15,6 +15,7 @@ import io import json import shlex +import shutil import os import re import stat @@ -171,6 +172,12 @@ def sidecar_root(primary: Path, kind: str, name: str, *, pin: bool = False) -> P raise IsolationError(str(error)) from error +def release_workspace_quietly(primary: Path) -> None: + """Best-effort placement hygiene after a sidecar retires; never blocks landing.""" + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(primary) + + def validate_task_id(task_id: str) -> str: if not TASK_ID_RE.fullmatch(task_id): raise IsolationError(f"invalid task id: {task_id}") @@ -477,6 +484,70 @@ def _same_directory(path: Path, expected: os.stat_result) -> bool: return stat.S_ISDIR(current.st_mode) and os.path.samestat(current, expected) +def _long_path(absolute: str) -> str: + if absolute.startswith("\\\\?\\"): + return absolute + if absolute.startswith("\\\\"): + return "\\\\?\\UNC\\" + absolute[2:] + return "\\\\?\\" + absolute + + +def _force_rmtree(path: Path) -> None: + """Delete a sidecar tree directly, tolerating read-only files and long paths.""" + def _writable(function, target, _error): + os.chmod(target, stat.S_IWRITE) + function(target) + + # The \\?\ prefix sidesteps Windows MAX_PATH on deep sidecar trees. + target = Path(_long_path(os.path.abspath(path))) if os.name == "nt" else path + try: + shutil.rmtree(target, onerror=_writable) + except OSError as error: + raise IsolationError(f"could not delete {path}: {error}") from error + + +def _remove_worktree(primary: Path, destination: Path, force: bool) -> None: + """Remove a sidecar worktree, tolerating populated submodules and locked trees. + + git refuses to remove worktrees whose submodules were ever initialized, and + forced removal can still fail deleting deep trees (Windows "Directory not + empty"). A dirty sidecar keeps git's refusal unless `force`, so forced + retries only run for a provably clean tree. The direct-delete fallback runs + only for git's tree-deletion failures, so any other git error leaves the + sidecar intact for diagnosis. + """ + tree_delete_failure = ("failed to delete", "directory not empty", "permission denied") + + def failure(*flags: str) -> str: + result = run_git(primary, "worktree", "remove", *flags, str(destination)) + return (result.stderr or result.stdout).strip() if result.returncode else "" + + detail = failure("--force") if force else failure() + if not detail: + return + if not force and destination.is_dir(): + status = run_git( + destination, "status", "--porcelain", "--untracked-files=all" + ) + if status.returncode != 0 or status.stdout.strip(): + raise IsolationError(detail or "git worktree remove failed") + run_git(destination, "submodule", "deinit", "--all", "--force") + detail = failure("--force") + if not detail: + return + if destination.is_dir() and any( + marker in detail.lower() for marker in tree_delete_failure + ): + _force_rmtree(destination) + detail = failure("--force") + if detail and not destination.is_dir(): + pruned = run_git(primary, "worktree", "prune") + if pruned.returncode == 0: + detail = "" + if detail or destination.is_dir() or destination.resolve() in _registered_worktrees(primary): + raise IsolationError(detail or "git worktree remove failed") + + def _cleanup_reserved_worktree( primary: Path, branch: str, @@ -762,9 +833,7 @@ def retire_member_task( if destination.resolve() in registered: if registered[destination.resolve()] != ref: raise IsolationError(f"member task sidecar is on another branch: {destination}") - removed = run_git(checkout, "worktree", "remove", *(("--force",) if force else ()), str(destination)) - if removed.returncode != 0: - raise IsolationError((removed.stderr or removed.stdout).strip() or "git worktree remove failed") + _remove_worktree(checkout, destination, force) authorization = run_git(checkout, "update-ref", "-d", member_task_authorization_ref(project, task_id)) if authorization.returncode != 0: raise IsolationError((authorization.stderr or authorization.stdout).strip() or "git update-ref failed") @@ -772,6 +841,7 @@ def retire_member_task( deleted = run_git(checkout, "update-ref", "-d", ref, tip.stdout.strip()) if deleted.returncode != 0: raise IsolationError((deleted.stderr or deleted.stdout).strip() or "git update-ref failed") + release_workspace_quietly(checkout) return copy_rejected @@ -1392,8 +1462,9 @@ def retire_member_verify(created: Dict[str, Dict[str, str]]) -> None: """Remove the member siblings of a coordinator verify sidecar; they hold no output.""" for record in reversed(list(created.values())): checkout, destination, branch = _member_verify_sidecar(record) - git_output(checkout, "worktree", "remove", "--force", str(destination)) + _remove_worktree(checkout, destination, True) git_output(checkout, "branch", "-D", branch) + release_workspace_quietly(checkout) def check_member_verify(created: Dict[str, Dict[str, str]]) -> None: @@ -1425,6 +1496,89 @@ def clean_verify(primary: Path, worktree: Path, base: str, branch: str) -> Dict[ return {"worktree": str(worktree), "branch": branch, "base": expected, "removed": removed} +def ensure_verify_sidecar(primary: Path, base: str, branch: str) -> Dict[str, object]: + """Return the named verify sidecar, recreating it when it is missing. + + Finish needs the sidecar prepare-task created. A deleted sidecar must be + rebuildable from the recorded base without a clean primary: the serial + task's own product changes are exactly the tree finish reproduces next, + so this must not reuse isolate_verify's clean-tree gate. + """ + primary = require_directory(primary, "primary worktree") + if worktree_root(primary) != primary: + raise IsolationError(f"primary is not its Git root: {primary}") + resolved_base = require_commit(primary, require_full_sha(base)) + if not branch.startswith(VERIFY_BRANCH_PREFIX): + raise IsolationError("verify sidecar reproduction requires a verify branch") + destination = _sidecar_path_for_branch(primary, branch) + ref = f"refs/heads/{branch}" + registered = _registered_worktrees(primary) + if destination.is_symlink() or (destination.exists() and not destination.is_dir()): + raise IsolationError(f"worktree path already exists: {destination}") + if destination.is_dir(): + try: + # A leftover empty directory (a half-deleted sidecar) clears; + # anything non-empty must be the sidecar itself. + destination.rmdir() + except OSError: + try: + same = ( + worktree_root(destination) == destination + and destination.resolve() in registered + and registered[destination.resolve()] == ref + and common_git_dir(destination) == common_git_dir(primary) + ) + except IsolationError: + same = False + if not same: + raise IsolationError( + f"verify sidecar ownership changed: {destination}" + ) from None + if ( + require_attached(destination) != branch + or current_sha(destination) != resolved_base + ): + raise IsolationError(f"verify sidecar HEAD changed: {destination}") + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + if destination.resolve() in registered: + cleared = run_git(primary, "worktree", "remove", str(destination)) + if cleared.returncode != 0: + run_git(primary, "worktree", "prune") + if destination.resolve() in _registered_worktrees(primary): + raise IsolationError( + (cleared.stderr or cleared.stdout).strip() + or "could not clear stale worktree registration" + ) + existing = run_git(primary, "show-ref", "--verify", "--quiet", ref) + if existing.returncode == 0: + deleted = run_git(primary, "update-ref", "-d", ref) + if deleted.returncode != 0: + raise IsolationError( + (deleted.stderr or deleted.stdout).strip() or "git update-ref failed" + ) + create_named_worktree(primary, branch, destination, resolved_base) + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + + +def reproduce_verify_sidecar(sidecar_path: Path, tree: str) -> None: + """Copy the primary's current task tree into the verify sidecar (clean_verify's inverse).""" + git_output(sidecar_path, "read-tree", "-u", "--reset", tree) + if git_output(sidecar_path, "write-tree") != tree: + raise IsolationError("sidecar reproduction differs from the primary task diff") + + def _split_paths(block: str) -> Set[str]: return {line.strip() for line in block.splitlines() if line.strip()} @@ -1537,14 +1691,24 @@ def _commit_pending( def validate_allowed_changes( - repo: Path, base: str, pending: Set[str], allowed: Set[str] + repo: Path, + base: str, + pending: Set[str], + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> None: - if not pending: + # Tolerated paths (primary bookkeeping) may stay uncommitted; the next + # `.project` checkpoint commits them. + landing = pending - tolerated + if not landing: raise IsolationError("no changes to land") - since_base = committed_paths_since(repo, base) | pending + since_base = committed_paths_since(repo, base) | landing unexpected = sorted(path for path in since_base if path not in allowed) if unexpected: - raise IsolationError("unexpected paths: " + ", ".join(unexpected)) + raise IsolationError( + "unexpected paths: " + ", ".join(unexpected) + + "; landing allows only the task file and its declared files" + ) def _clean_blob_oid(repo: Path, path: str, text: str, *, write: bool) -> str: arguments = ["hash-object"] if write: @@ -1564,21 +1728,23 @@ def commit_allowed_changes( subject: str, allowed: Set[str], body: str, + tolerated: Set[str] = frozenset(), ) -> tuple[str, str, Dict[str, Optional[tuple[str, str]]]]: require_attached(repo) pending = uncommitted_paths(repo) - validate_allowed_changes(repo, base, pending, allowed) + validate_allowed_changes(repo, base, pending, allowed, tolerated) + staging = pending - tolerated reset = run_git(repo, "reset", "-q", "HEAD") if reset.returncode != 0: raise IsolationError("could not clear the index before landing") - for path in sorted(pending): + for path in sorted(staging): added = run_git(repo, "add", "-A", "--", path) if added.returncode != 0: raise IsolationError(f"could not stage {path}") staged = _split_paths( git_output(repo, "diff", "--cached", "--name-only", "--relative") ) - if staged != pending: + if staged != staging: raise IsolationError("staged paths do not match the uncommitted change set") verified_tree = git_output(repo, "write-tree") verified_delta = _tree_delta(repo, base, verified_tree) @@ -1724,6 +1890,17 @@ def activate_task( if task_branch is None: if branch.startswith(TASK_BRANCH_PREFIX): raise IsolationError("parallel task activation requires --task-branch") + live = [ + path + for path, ref in _registered_worktrees(worktree).items() + if ref == f"refs/heads/{task_branch_name(task_id)}" + and path != worktree + ] + if live: + raise IsolationError( + "task has an active parallel worktree; serial activation in " + f"another worktree is refused (activate {live[0]} with --task-branch)" + ) elif task_branch != task_branch_name(task_id) or branch != task_branch: raise IsolationError("task activation branch does not match the task worktree") normalized_task = relative_posix(task_file) @@ -1991,7 +2168,12 @@ def _restore_rejected_commit( def _stamp_and_commit( - worktree: Path, base: str, subject: str, task_file: str, allowed: Set[str] + worktree: Path, + base: str, + subject: str, + task_file: str, + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> tuple[str, Dict[str, Optional[tuple[str, str]]]]: task_path = worktree / task_file task_text, task_mode = _read_task_text(task_path) @@ -2012,14 +2194,14 @@ def _stamp_and_commit( pending = uncommitted_paths(worktree) if stamped != task_bytes: pending.add(task_file) - validate_allowed_changes(worktree, base, pending, allowed) + validate_allowed_changes(worktree, base, pending, allowed, tolerated) original_head = current_sha(worktree) index_tree = git_output(worktree, "write-tree") try: _replace_regular_file(task_path, stamped, task_mode) - body = task_commit_body(task_file, uncommitted_paths(worktree), base) + body = task_commit_body(task_file, uncommitted_paths(worktree) - tolerated, base) commit, verified_tree, verified_delta = commit_allowed_changes( - worktree, base, subject, allowed, body + worktree, base, subject, allowed, body, tolerated ) proof_error = _landing_commit_proof_error( worktree, @@ -2097,6 +2279,7 @@ def land( subject, relative_posix(task_file), allowed, + BOOKKEEPING_PATHS, ) return { "bound_branch": bound, @@ -2391,7 +2574,7 @@ def _landing_retry_error( if task_file not in pending: return "interrupted landing does not retain the task change" try: - validate_allowed_changes(repo, base, pending, contract_paths) + validate_allowed_changes(repo, base, pending, contract_paths, BOOKKEEPING_PATHS) except IsolationError as validation_error: return str(validation_error) return None @@ -4430,6 +4613,7 @@ def retire( or f"could not delete {branch}" ) branch_retired = True + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": branch, @@ -4463,22 +4647,13 @@ def retire( raise IsolationError( "worktree is dirty; pass --force only from the retry-retirement path" ) - remove = run_git( - primary, - "worktree", - "remove", - *(("--force",) if force else ()), - str(resolved_worktree), - ) - if remove.returncode != 0: - raise IsolationError( - (remove.stderr or remove.stdout).strip() or "git worktree remove failed" - ) + _remove_worktree(primary, resolved_worktree, force) deleted = run_git(primary, "branch", "-D", retire_branch) if deleted.returncode != 0: raise IsolationError( (deleted.stderr or deleted.stdout).strip() or f"could not delete {retire_branch}" ) + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": retire_branch, diff --git a/skills/gsd-path-docs-audit/scripts/members.py b/skills/gsd-path-docs-audit/scripts/members.py index d427c6ee4..7f516c846 100644 --- a/skills/gsd-path-docs-audit/scripts/members.py +++ b/skills/gsd-path-docs-audit/scripts/members.py @@ -255,6 +255,7 @@ def check_member( coordinator_name: str, checkout: Path, recorded_remote: Optional[str] = None, + refuse_reserved_refs: bool = True, ) -> str: """Refuse a checkout that cannot join; return its origin URL.""" if checkout.is_symlink() or not checkout.is_dir(): @@ -294,13 +295,14 @@ def check_member( raise MembersError(f"member STATE.md is unreadable: {error}") from error if state.milestone is not None and (state.phase, state.status) != ("shipped", "done"): raise MembersError(f"member has an active milestone: {state.milestone}") - prefixes = member_ref_prefixes(coordinator_name) + tuple( - "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS - ) - refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() - colliding = [ref for ref in refs if ref.startswith(prefixes)] - if colliding: - raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) + if refuse_reserved_refs: + prefixes = member_ref_prefixes(coordinator_name) + tuple( + "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS + ) + refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() + colliding = [ref for ref in refs if ref.startswith(prefixes)] + if colliding: + raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) return remote @@ -620,7 +622,7 @@ def validate_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) try: role = member_role(checkout) except MembersError as error: @@ -639,7 +641,7 @@ def repair_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) _refuse_foreign_marker(checkout.resolve(), root, state.project, member["name"]) for member in members: checkout = Path(member["checkout"]).resolve() diff --git a/skills/gsd-path-docs-audit/scripts/worktree_paths.py b/skills/gsd-path-docs-audit/scripts/worktree_paths.py index 19653ca92..b2cfda32f 100644 --- a/skills/gsd-path-docs-audit/scripts/worktree_paths.py +++ b/skills/gsd-path-docs-audit/scripts/worktree_paths.py @@ -103,6 +103,33 @@ def _workspace(primary: Path, pin: bool) -> Path: return workspace +def release_workspace(primary: Path) -> bool: + """Forget the pinned placement receipt once no linked worktrees remain. + + Lets GSD_PATH_WORKTREE_ROOT (or a new default) apply again at the next + milestone's first allocation. Returns True when a receipt was removed; + any remaining linked worktree (including one only still registered) keeps + the pin. + """ + primary = primary.resolve() + common = Path(_git(primary, "rev-parse", "--git-common-dir")) + common = (primary / common).resolve() + receipt = common / "gsd-path" / "workspaces" / f"{_identity(primary)}.json" + if not receipt.is_file(): + return False + output, nul_separated = _worktree_list_porcelain(primary) + records = output.split("\0\0") if nul_separated else [ + block for block in output.strip().split("\n\n") if block.strip() + ] + for record in records: + fields = record.split("\0") if nul_separated else record.splitlines() + for field in fields: + if field.startswith("worktree ") and Path(field[len("worktree "):]).resolve() != primary: + return False + receipt.unlink() + return True + + def worktree_path(primary: Path, kind: str, name: str, *, pin: bool = False) -> Path: primary = primary.resolve() if kind not in {"task", "verify", "integrate", "bound"} or not name or Path(name).name != name or name in {".", ".."}: diff --git a/skills/gsd-path-forensics/scripts/worktree_paths.py b/skills/gsd-path-forensics/scripts/worktree_paths.py index 19653ca92..b2cfda32f 100644 --- a/skills/gsd-path-forensics/scripts/worktree_paths.py +++ b/skills/gsd-path-forensics/scripts/worktree_paths.py @@ -103,6 +103,33 @@ def _workspace(primary: Path, pin: bool) -> Path: return workspace +def release_workspace(primary: Path) -> bool: + """Forget the pinned placement receipt once no linked worktrees remain. + + Lets GSD_PATH_WORKTREE_ROOT (or a new default) apply again at the next + milestone's first allocation. Returns True when a receipt was removed; + any remaining linked worktree (including one only still registered) keeps + the pin. + """ + primary = primary.resolve() + common = Path(_git(primary, "rev-parse", "--git-common-dir")) + common = (primary / common).resolve() + receipt = common / "gsd-path" / "workspaces" / f"{_identity(primary)}.json" + if not receipt.is_file(): + return False + output, nul_separated = _worktree_list_porcelain(primary) + records = output.split("\0\0") if nul_separated else [ + block for block in output.strip().split("\n\n") if block.strip() + ] + for record in records: + fields = record.split("\0") if nul_separated else record.splitlines() + for field in fields: + if field.startswith("worktree ") and Path(field[len("worktree "):]).resolve() != primary: + return False + receipt.unlink() + return True + + def worktree_path(primary: Path, kind: str, name: str, *, pin: bool = False) -> Path: primary = primary.resolve() if kind not in {"task", "verify", "integrate", "bound"} or not name or Path(name).name != name or name in {".", ".."}: diff --git a/skills/gsd-path-inspect/scripts/check_handoffs.py b/skills/gsd-path-inspect/scripts/check_handoffs.py index f86fa4306..743f7cdf9 100644 --- a/skills/gsd-path-inspect/scripts/check_handoffs.py +++ b/skills/gsd-path-inspect/scripts/check_handoffs.py @@ -1386,6 +1386,20 @@ def _task_deps(task_text: str, task_id: str) -> List[str]: return _inline_ids(rendered, "T", f"{task_id} deps") +def _dependency_reachable(graph: Dict[str, List[str]], start: str, target: str) -> bool: + seen: Set[str] = set() + pending = list(graph[start]) + while pending: + task_id = pending.pop() + if task_id in seen: + continue + if task_id == target: + return True + seen.add(task_id) + pending.extend(graph[task_id]) + return False + + def _canonical_repo_path(value: str, label: str) -> str: cleaned = value.strip().strip("`\"'") path = PurePosixPath(cleaned) @@ -1509,6 +1523,12 @@ def _validate_task_graph( and _task_scalar(tasks[right], right, "status") == "done" ): continue + # Dependency-ordered tasks serialize safely; only unordered + # same-wave overlap races on a shared file. + if _dependency_reachable(graph, left, right) or _dependency_reachable( + graph, right, left + ): + continue raise HandoffError( f"same-wave file overlap between {left} and {right}: {', '.join(overlap)}" ) diff --git a/skills/gsd-path-inspect/scripts/integration.py b/skills/gsd-path-inspect/scripts/integration.py index 0dd2a356b..c1b0971d5 100644 --- a/skills/gsd-path-inspect/scripts/integration.py +++ b/skills/gsd-path-inspect/scripts/integration.py @@ -9,6 +9,7 @@ import json import re +import contextlib from pathlib import Path, PurePosixPath from typing import Optional @@ -1939,6 +1940,10 @@ def retire_member(coordinator: Path, member: str, archive_path: str, reviewed_he for kind, ref in (("push", bound_ref), ("push", "refs/heads/main"), ("push", f"refs/tags/{tag_name}"), ("delete", bound_ref)): members.clear_authorization(checkout, project, kind, ref) + # The last sidecar retires here at milestone end; isolation's retires also + # release the pin whenever the workspace is idle, so a new root can apply. + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(checkout) return {"member": member, "retired": bound} diff --git a/skills/gsd-path-inspect/scripts/isolation.py b/skills/gsd-path-inspect/scripts/isolation.py index 50a9ef684..3805f809a 100644 --- a/skills/gsd-path-inspect/scripts/isolation.py +++ b/skills/gsd-path-inspect/scripts/isolation.py @@ -15,6 +15,7 @@ import io import json import shlex +import shutil import os import re import stat @@ -171,6 +172,12 @@ def sidecar_root(primary: Path, kind: str, name: str, *, pin: bool = False) -> P raise IsolationError(str(error)) from error +def release_workspace_quietly(primary: Path) -> None: + """Best-effort placement hygiene after a sidecar retires; never blocks landing.""" + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(primary) + + def validate_task_id(task_id: str) -> str: if not TASK_ID_RE.fullmatch(task_id): raise IsolationError(f"invalid task id: {task_id}") @@ -477,6 +484,70 @@ def _same_directory(path: Path, expected: os.stat_result) -> bool: return stat.S_ISDIR(current.st_mode) and os.path.samestat(current, expected) +def _long_path(absolute: str) -> str: + if absolute.startswith("\\\\?\\"): + return absolute + if absolute.startswith("\\\\"): + return "\\\\?\\UNC\\" + absolute[2:] + return "\\\\?\\" + absolute + + +def _force_rmtree(path: Path) -> None: + """Delete a sidecar tree directly, tolerating read-only files and long paths.""" + def _writable(function, target, _error): + os.chmod(target, stat.S_IWRITE) + function(target) + + # The \\?\ prefix sidesteps Windows MAX_PATH on deep sidecar trees. + target = Path(_long_path(os.path.abspath(path))) if os.name == "nt" else path + try: + shutil.rmtree(target, onerror=_writable) + except OSError as error: + raise IsolationError(f"could not delete {path}: {error}") from error + + +def _remove_worktree(primary: Path, destination: Path, force: bool) -> None: + """Remove a sidecar worktree, tolerating populated submodules and locked trees. + + git refuses to remove worktrees whose submodules were ever initialized, and + forced removal can still fail deleting deep trees (Windows "Directory not + empty"). A dirty sidecar keeps git's refusal unless `force`, so forced + retries only run for a provably clean tree. The direct-delete fallback runs + only for git's tree-deletion failures, so any other git error leaves the + sidecar intact for diagnosis. + """ + tree_delete_failure = ("failed to delete", "directory not empty", "permission denied") + + def failure(*flags: str) -> str: + result = run_git(primary, "worktree", "remove", *flags, str(destination)) + return (result.stderr or result.stdout).strip() if result.returncode else "" + + detail = failure("--force") if force else failure() + if not detail: + return + if not force and destination.is_dir(): + status = run_git( + destination, "status", "--porcelain", "--untracked-files=all" + ) + if status.returncode != 0 or status.stdout.strip(): + raise IsolationError(detail or "git worktree remove failed") + run_git(destination, "submodule", "deinit", "--all", "--force") + detail = failure("--force") + if not detail: + return + if destination.is_dir() and any( + marker in detail.lower() for marker in tree_delete_failure + ): + _force_rmtree(destination) + detail = failure("--force") + if detail and not destination.is_dir(): + pruned = run_git(primary, "worktree", "prune") + if pruned.returncode == 0: + detail = "" + if detail or destination.is_dir() or destination.resolve() in _registered_worktrees(primary): + raise IsolationError(detail or "git worktree remove failed") + + def _cleanup_reserved_worktree( primary: Path, branch: str, @@ -762,9 +833,7 @@ def retire_member_task( if destination.resolve() in registered: if registered[destination.resolve()] != ref: raise IsolationError(f"member task sidecar is on another branch: {destination}") - removed = run_git(checkout, "worktree", "remove", *(("--force",) if force else ()), str(destination)) - if removed.returncode != 0: - raise IsolationError((removed.stderr or removed.stdout).strip() or "git worktree remove failed") + _remove_worktree(checkout, destination, force) authorization = run_git(checkout, "update-ref", "-d", member_task_authorization_ref(project, task_id)) if authorization.returncode != 0: raise IsolationError((authorization.stderr or authorization.stdout).strip() or "git update-ref failed") @@ -772,6 +841,7 @@ def retire_member_task( deleted = run_git(checkout, "update-ref", "-d", ref, tip.stdout.strip()) if deleted.returncode != 0: raise IsolationError((deleted.stderr or deleted.stdout).strip() or "git update-ref failed") + release_workspace_quietly(checkout) return copy_rejected @@ -1392,8 +1462,9 @@ def retire_member_verify(created: Dict[str, Dict[str, str]]) -> None: """Remove the member siblings of a coordinator verify sidecar; they hold no output.""" for record in reversed(list(created.values())): checkout, destination, branch = _member_verify_sidecar(record) - git_output(checkout, "worktree", "remove", "--force", str(destination)) + _remove_worktree(checkout, destination, True) git_output(checkout, "branch", "-D", branch) + release_workspace_quietly(checkout) def check_member_verify(created: Dict[str, Dict[str, str]]) -> None: @@ -1425,6 +1496,89 @@ def clean_verify(primary: Path, worktree: Path, base: str, branch: str) -> Dict[ return {"worktree": str(worktree), "branch": branch, "base": expected, "removed": removed} +def ensure_verify_sidecar(primary: Path, base: str, branch: str) -> Dict[str, object]: + """Return the named verify sidecar, recreating it when it is missing. + + Finish needs the sidecar prepare-task created. A deleted sidecar must be + rebuildable from the recorded base without a clean primary: the serial + task's own product changes are exactly the tree finish reproduces next, + so this must not reuse isolate_verify's clean-tree gate. + """ + primary = require_directory(primary, "primary worktree") + if worktree_root(primary) != primary: + raise IsolationError(f"primary is not its Git root: {primary}") + resolved_base = require_commit(primary, require_full_sha(base)) + if not branch.startswith(VERIFY_BRANCH_PREFIX): + raise IsolationError("verify sidecar reproduction requires a verify branch") + destination = _sidecar_path_for_branch(primary, branch) + ref = f"refs/heads/{branch}" + registered = _registered_worktrees(primary) + if destination.is_symlink() or (destination.exists() and not destination.is_dir()): + raise IsolationError(f"worktree path already exists: {destination}") + if destination.is_dir(): + try: + # A leftover empty directory (a half-deleted sidecar) clears; + # anything non-empty must be the sidecar itself. + destination.rmdir() + except OSError: + try: + same = ( + worktree_root(destination) == destination + and destination.resolve() in registered + and registered[destination.resolve()] == ref + and common_git_dir(destination) == common_git_dir(primary) + ) + except IsolationError: + same = False + if not same: + raise IsolationError( + f"verify sidecar ownership changed: {destination}" + ) from None + if ( + require_attached(destination) != branch + or current_sha(destination) != resolved_base + ): + raise IsolationError(f"verify sidecar HEAD changed: {destination}") + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + if destination.resolve() in registered: + cleared = run_git(primary, "worktree", "remove", str(destination)) + if cleared.returncode != 0: + run_git(primary, "worktree", "prune") + if destination.resolve() in _registered_worktrees(primary): + raise IsolationError( + (cleared.stderr or cleared.stdout).strip() + or "could not clear stale worktree registration" + ) + existing = run_git(primary, "show-ref", "--verify", "--quiet", ref) + if existing.returncode == 0: + deleted = run_git(primary, "update-ref", "-d", ref) + if deleted.returncode != 0: + raise IsolationError( + (deleted.stderr or deleted.stdout).strip() or "git update-ref failed" + ) + create_named_worktree(primary, branch, destination, resolved_base) + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + + +def reproduce_verify_sidecar(sidecar_path: Path, tree: str) -> None: + """Copy the primary's current task tree into the verify sidecar (clean_verify's inverse).""" + git_output(sidecar_path, "read-tree", "-u", "--reset", tree) + if git_output(sidecar_path, "write-tree") != tree: + raise IsolationError("sidecar reproduction differs from the primary task diff") + + def _split_paths(block: str) -> Set[str]: return {line.strip() for line in block.splitlines() if line.strip()} @@ -1537,14 +1691,24 @@ def _commit_pending( def validate_allowed_changes( - repo: Path, base: str, pending: Set[str], allowed: Set[str] + repo: Path, + base: str, + pending: Set[str], + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> None: - if not pending: + # Tolerated paths (primary bookkeeping) may stay uncommitted; the next + # `.project` checkpoint commits them. + landing = pending - tolerated + if not landing: raise IsolationError("no changes to land") - since_base = committed_paths_since(repo, base) | pending + since_base = committed_paths_since(repo, base) | landing unexpected = sorted(path for path in since_base if path not in allowed) if unexpected: - raise IsolationError("unexpected paths: " + ", ".join(unexpected)) + raise IsolationError( + "unexpected paths: " + ", ".join(unexpected) + + "; landing allows only the task file and its declared files" + ) def _clean_blob_oid(repo: Path, path: str, text: str, *, write: bool) -> str: arguments = ["hash-object"] if write: @@ -1564,21 +1728,23 @@ def commit_allowed_changes( subject: str, allowed: Set[str], body: str, + tolerated: Set[str] = frozenset(), ) -> tuple[str, str, Dict[str, Optional[tuple[str, str]]]]: require_attached(repo) pending = uncommitted_paths(repo) - validate_allowed_changes(repo, base, pending, allowed) + validate_allowed_changes(repo, base, pending, allowed, tolerated) + staging = pending - tolerated reset = run_git(repo, "reset", "-q", "HEAD") if reset.returncode != 0: raise IsolationError("could not clear the index before landing") - for path in sorted(pending): + for path in sorted(staging): added = run_git(repo, "add", "-A", "--", path) if added.returncode != 0: raise IsolationError(f"could not stage {path}") staged = _split_paths( git_output(repo, "diff", "--cached", "--name-only", "--relative") ) - if staged != pending: + if staged != staging: raise IsolationError("staged paths do not match the uncommitted change set") verified_tree = git_output(repo, "write-tree") verified_delta = _tree_delta(repo, base, verified_tree) @@ -1724,6 +1890,17 @@ def activate_task( if task_branch is None: if branch.startswith(TASK_BRANCH_PREFIX): raise IsolationError("parallel task activation requires --task-branch") + live = [ + path + for path, ref in _registered_worktrees(worktree).items() + if ref == f"refs/heads/{task_branch_name(task_id)}" + and path != worktree + ] + if live: + raise IsolationError( + "task has an active parallel worktree; serial activation in " + f"another worktree is refused (activate {live[0]} with --task-branch)" + ) elif task_branch != task_branch_name(task_id) or branch != task_branch: raise IsolationError("task activation branch does not match the task worktree") normalized_task = relative_posix(task_file) @@ -1991,7 +2168,12 @@ def _restore_rejected_commit( def _stamp_and_commit( - worktree: Path, base: str, subject: str, task_file: str, allowed: Set[str] + worktree: Path, + base: str, + subject: str, + task_file: str, + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> tuple[str, Dict[str, Optional[tuple[str, str]]]]: task_path = worktree / task_file task_text, task_mode = _read_task_text(task_path) @@ -2012,14 +2194,14 @@ def _stamp_and_commit( pending = uncommitted_paths(worktree) if stamped != task_bytes: pending.add(task_file) - validate_allowed_changes(worktree, base, pending, allowed) + validate_allowed_changes(worktree, base, pending, allowed, tolerated) original_head = current_sha(worktree) index_tree = git_output(worktree, "write-tree") try: _replace_regular_file(task_path, stamped, task_mode) - body = task_commit_body(task_file, uncommitted_paths(worktree), base) + body = task_commit_body(task_file, uncommitted_paths(worktree) - tolerated, base) commit, verified_tree, verified_delta = commit_allowed_changes( - worktree, base, subject, allowed, body + worktree, base, subject, allowed, body, tolerated ) proof_error = _landing_commit_proof_error( worktree, @@ -2097,6 +2279,7 @@ def land( subject, relative_posix(task_file), allowed, + BOOKKEEPING_PATHS, ) return { "bound_branch": bound, @@ -2391,7 +2574,7 @@ def _landing_retry_error( if task_file not in pending: return "interrupted landing does not retain the task change" try: - validate_allowed_changes(repo, base, pending, contract_paths) + validate_allowed_changes(repo, base, pending, contract_paths, BOOKKEEPING_PATHS) except IsolationError as validation_error: return str(validation_error) return None @@ -4430,6 +4613,7 @@ def retire( or f"could not delete {branch}" ) branch_retired = True + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": branch, @@ -4463,22 +4647,13 @@ def retire( raise IsolationError( "worktree is dirty; pass --force only from the retry-retirement path" ) - remove = run_git( - primary, - "worktree", - "remove", - *(("--force",) if force else ()), - str(resolved_worktree), - ) - if remove.returncode != 0: - raise IsolationError( - (remove.stderr or remove.stdout).strip() or "git worktree remove failed" - ) + _remove_worktree(primary, resolved_worktree, force) deleted = run_git(primary, "branch", "-D", retire_branch) if deleted.returncode != 0: raise IsolationError( (deleted.stderr or deleted.stdout).strip() or f"could not delete {retire_branch}" ) + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": retire_branch, diff --git a/skills/gsd-path-inspect/scripts/members.py b/skills/gsd-path-inspect/scripts/members.py index d427c6ee4..7f516c846 100644 --- a/skills/gsd-path-inspect/scripts/members.py +++ b/skills/gsd-path-inspect/scripts/members.py @@ -255,6 +255,7 @@ def check_member( coordinator_name: str, checkout: Path, recorded_remote: Optional[str] = None, + refuse_reserved_refs: bool = True, ) -> str: """Refuse a checkout that cannot join; return its origin URL.""" if checkout.is_symlink() or not checkout.is_dir(): @@ -294,13 +295,14 @@ def check_member( raise MembersError(f"member STATE.md is unreadable: {error}") from error if state.milestone is not None and (state.phase, state.status) != ("shipped", "done"): raise MembersError(f"member has an active milestone: {state.milestone}") - prefixes = member_ref_prefixes(coordinator_name) + tuple( - "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS - ) - refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() - colliding = [ref for ref in refs if ref.startswith(prefixes)] - if colliding: - raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) + if refuse_reserved_refs: + prefixes = member_ref_prefixes(coordinator_name) + tuple( + "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS + ) + refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() + colliding = [ref for ref in refs if ref.startswith(prefixes)] + if colliding: + raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) return remote @@ -620,7 +622,7 @@ def validate_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) try: role = member_role(checkout) except MembersError as error: @@ -639,7 +641,7 @@ def repair_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) _refuse_foreign_marker(checkout.resolve(), root, state.project, member["name"]) for member in members: checkout = Path(member["checkout"]).resolve() diff --git a/skills/gsd-path-inspect/scripts/worktree_paths.py b/skills/gsd-path-inspect/scripts/worktree_paths.py index 19653ca92..b2cfda32f 100644 --- a/skills/gsd-path-inspect/scripts/worktree_paths.py +++ b/skills/gsd-path-inspect/scripts/worktree_paths.py @@ -103,6 +103,33 @@ def _workspace(primary: Path, pin: bool) -> Path: return workspace +def release_workspace(primary: Path) -> bool: + """Forget the pinned placement receipt once no linked worktrees remain. + + Lets GSD_PATH_WORKTREE_ROOT (or a new default) apply again at the next + milestone's first allocation. Returns True when a receipt was removed; + any remaining linked worktree (including one only still registered) keeps + the pin. + """ + primary = primary.resolve() + common = Path(_git(primary, "rev-parse", "--git-common-dir")) + common = (primary / common).resolve() + receipt = common / "gsd-path" / "workspaces" / f"{_identity(primary)}.json" + if not receipt.is_file(): + return False + output, nul_separated = _worktree_list_porcelain(primary) + records = output.split("\0\0") if nul_separated else [ + block for block in output.strip().split("\n\n") if block.strip() + ] + for record in records: + fields = record.split("\0") if nul_separated else record.splitlines() + for field in fields: + if field.startswith("worktree ") and Path(field[len("worktree "):]).resolve() != primary: + return False + receipt.unlink() + return True + + def worktree_path(primary: Path, kind: str, name: str, *, pin: bool = False) -> Path: primary = primary.resolve() if kind not in {"task", "verify", "integrate", "bound"} or not name or Path(name).name != name or name in {".", ".."}: diff --git a/skills/gsd-path-loop/scripts/worktree_paths.py b/skills/gsd-path-loop/scripts/worktree_paths.py index 19653ca92..b2cfda32f 100644 --- a/skills/gsd-path-loop/scripts/worktree_paths.py +++ b/skills/gsd-path-loop/scripts/worktree_paths.py @@ -103,6 +103,33 @@ def _workspace(primary: Path, pin: bool) -> Path: return workspace +def release_workspace(primary: Path) -> bool: + """Forget the pinned placement receipt once no linked worktrees remain. + + Lets GSD_PATH_WORKTREE_ROOT (or a new default) apply again at the next + milestone's first allocation. Returns True when a receipt was removed; + any remaining linked worktree (including one only still registered) keeps + the pin. + """ + primary = primary.resolve() + common = Path(_git(primary, "rev-parse", "--git-common-dir")) + common = (primary / common).resolve() + receipt = common / "gsd-path" / "workspaces" / f"{_identity(primary)}.json" + if not receipt.is_file(): + return False + output, nul_separated = _worktree_list_porcelain(primary) + records = output.split("\0\0") if nul_separated else [ + block for block in output.strip().split("\n\n") if block.strip() + ] + for record in records: + fields = record.split("\0") if nul_separated else record.splitlines() + for field in fields: + if field.startswith("worktree ") and Path(field[len("worktree "):]).resolve() != primary: + return False + receipt.unlink() + return True + + def worktree_path(primary: Path, kind: str, name: str, *, pin: bool = False) -> Path: primary = primary.resolve() if kind not in {"task", "verify", "integrate", "bound"} or not name or Path(name).name != name or name in {".", ".."}: diff --git a/skills/gsd-path-plan/SKILL.md b/skills/gsd-path-plan/SKILL.md index 654eacd1a..a87ea5149 100644 --- a/skills/gsd-path-plan/SKILL.md +++ b/skills/gsd-path-plan/SKILL.md @@ -115,9 +115,10 @@ as stated in Lookahead mode. In patch mode, tasks already `done` keep their landed metadata; only the appended patch tasks must start clean. - Require every dependency id to exist, forbid later-wave dependencies, - detect cycles, and forbid file overlap between planned tasks in the same - wave. Same-wave dependency chains are allowed only when their files do - not overlap; the build executes them in dependency layers. + detect cycles, and forbid unordered file overlap between planned tasks + in the same wave. Same-wave tasks may share files only when one + transitively depends on the other through `deps`; the build executes + them in dependency layers. - Reject a declared dependency that carries neither data nor a prerequisite effect. A `deps` edge is valid only when the dependent consumes a symbol, signature, schema, endpoint, file format, or path named in that diff --git a/skills/gsd-path-plan/references/planner.md b/skills/gsd-path-plan/references/planner.md index 5f4421976..e98a5c613 100644 --- a/skills/gsd-path-plan/references/planner.md +++ b/skills/gsd-path-plan/references/planner.md @@ -38,8 +38,9 @@ the how. security-critical; `verify-only` suits polish and low-risk feature waves whose Verify commands meaningfully cover the criteria. -Every dependency must be in an earlier wave or in the same wave with no file -overlap. Produce an acyclic graph; same-wave chains execute in layers. +Every dependency must be in an earlier wave or in the same wave. Same-wave +tasks may share files only when one transitively depends on the other. Produce +an acyclic graph; same-wave chains execute in layers. Declare a dependency only when something real crosses it: data — the dependent consumes a symbol, signature, schema, endpoint, file format, or path named in @@ -81,7 +82,8 @@ plan order above, which still keeps plan-invalidating risk in wave 1. held-out. Put each `judgment` prohibition's Must not and Detail in the owning task's Approach. - Make `files` exhaustive, including imports, routes, generated artifacts, - tests, and wiring. Require disjoint files for same-wave tasks. + tests, and wiring. Require disjoint files for same-wave tasks that are not + dependency-ordered. - Write an Interface contract in every task: `None` for independent tasks; for any tasks that exchange a symbol, signature, schema, endpoint, file format, or path, the exact shapes both sides code against — identical text diff --git a/skills/gsd-path-plan/scripts/check_handoffs.py b/skills/gsd-path-plan/scripts/check_handoffs.py index f86fa4306..743f7cdf9 100644 --- a/skills/gsd-path-plan/scripts/check_handoffs.py +++ b/skills/gsd-path-plan/scripts/check_handoffs.py @@ -1386,6 +1386,20 @@ def _task_deps(task_text: str, task_id: str) -> List[str]: return _inline_ids(rendered, "T", f"{task_id} deps") +def _dependency_reachable(graph: Dict[str, List[str]], start: str, target: str) -> bool: + seen: Set[str] = set() + pending = list(graph[start]) + while pending: + task_id = pending.pop() + if task_id in seen: + continue + if task_id == target: + return True + seen.add(task_id) + pending.extend(graph[task_id]) + return False + + def _canonical_repo_path(value: str, label: str) -> str: cleaned = value.strip().strip("`\"'") path = PurePosixPath(cleaned) @@ -1509,6 +1523,12 @@ def _validate_task_graph( and _task_scalar(tasks[right], right, "status") == "done" ): continue + # Dependency-ordered tasks serialize safely; only unordered + # same-wave overlap races on a shared file. + if _dependency_reachable(graph, left, right) or _dependency_reachable( + graph, right, left + ): + continue raise HandoffError( f"same-wave file overlap between {left} and {right}: {', '.join(overlap)}" ) diff --git a/skills/gsd-path-plan/scripts/integration.py b/skills/gsd-path-plan/scripts/integration.py index 0dd2a356b..c1b0971d5 100644 --- a/skills/gsd-path-plan/scripts/integration.py +++ b/skills/gsd-path-plan/scripts/integration.py @@ -9,6 +9,7 @@ import json import re +import contextlib from pathlib import Path, PurePosixPath from typing import Optional @@ -1939,6 +1940,10 @@ def retire_member(coordinator: Path, member: str, archive_path: str, reviewed_he for kind, ref in (("push", bound_ref), ("push", "refs/heads/main"), ("push", f"refs/tags/{tag_name}"), ("delete", bound_ref)): members.clear_authorization(checkout, project, kind, ref) + # The last sidecar retires here at milestone end; isolation's retires also + # release the pin whenever the workspace is idle, so a new root can apply. + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(checkout) return {"member": member, "retired": bound} diff --git a/skills/gsd-path-plan/scripts/isolation.py b/skills/gsd-path-plan/scripts/isolation.py index 50a9ef684..3805f809a 100644 --- a/skills/gsd-path-plan/scripts/isolation.py +++ b/skills/gsd-path-plan/scripts/isolation.py @@ -15,6 +15,7 @@ import io import json import shlex +import shutil import os import re import stat @@ -171,6 +172,12 @@ def sidecar_root(primary: Path, kind: str, name: str, *, pin: bool = False) -> P raise IsolationError(str(error)) from error +def release_workspace_quietly(primary: Path) -> None: + """Best-effort placement hygiene after a sidecar retires; never blocks landing.""" + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(primary) + + def validate_task_id(task_id: str) -> str: if not TASK_ID_RE.fullmatch(task_id): raise IsolationError(f"invalid task id: {task_id}") @@ -477,6 +484,70 @@ def _same_directory(path: Path, expected: os.stat_result) -> bool: return stat.S_ISDIR(current.st_mode) and os.path.samestat(current, expected) +def _long_path(absolute: str) -> str: + if absolute.startswith("\\\\?\\"): + return absolute + if absolute.startswith("\\\\"): + return "\\\\?\\UNC\\" + absolute[2:] + return "\\\\?\\" + absolute + + +def _force_rmtree(path: Path) -> None: + """Delete a sidecar tree directly, tolerating read-only files and long paths.""" + def _writable(function, target, _error): + os.chmod(target, stat.S_IWRITE) + function(target) + + # The \\?\ prefix sidesteps Windows MAX_PATH on deep sidecar trees. + target = Path(_long_path(os.path.abspath(path))) if os.name == "nt" else path + try: + shutil.rmtree(target, onerror=_writable) + except OSError as error: + raise IsolationError(f"could not delete {path}: {error}") from error + + +def _remove_worktree(primary: Path, destination: Path, force: bool) -> None: + """Remove a sidecar worktree, tolerating populated submodules and locked trees. + + git refuses to remove worktrees whose submodules were ever initialized, and + forced removal can still fail deleting deep trees (Windows "Directory not + empty"). A dirty sidecar keeps git's refusal unless `force`, so forced + retries only run for a provably clean tree. The direct-delete fallback runs + only for git's tree-deletion failures, so any other git error leaves the + sidecar intact for diagnosis. + """ + tree_delete_failure = ("failed to delete", "directory not empty", "permission denied") + + def failure(*flags: str) -> str: + result = run_git(primary, "worktree", "remove", *flags, str(destination)) + return (result.stderr or result.stdout).strip() if result.returncode else "" + + detail = failure("--force") if force else failure() + if not detail: + return + if not force and destination.is_dir(): + status = run_git( + destination, "status", "--porcelain", "--untracked-files=all" + ) + if status.returncode != 0 or status.stdout.strip(): + raise IsolationError(detail or "git worktree remove failed") + run_git(destination, "submodule", "deinit", "--all", "--force") + detail = failure("--force") + if not detail: + return + if destination.is_dir() and any( + marker in detail.lower() for marker in tree_delete_failure + ): + _force_rmtree(destination) + detail = failure("--force") + if detail and not destination.is_dir(): + pruned = run_git(primary, "worktree", "prune") + if pruned.returncode == 0: + detail = "" + if detail or destination.is_dir() or destination.resolve() in _registered_worktrees(primary): + raise IsolationError(detail or "git worktree remove failed") + + def _cleanup_reserved_worktree( primary: Path, branch: str, @@ -762,9 +833,7 @@ def retire_member_task( if destination.resolve() in registered: if registered[destination.resolve()] != ref: raise IsolationError(f"member task sidecar is on another branch: {destination}") - removed = run_git(checkout, "worktree", "remove", *(("--force",) if force else ()), str(destination)) - if removed.returncode != 0: - raise IsolationError((removed.stderr or removed.stdout).strip() or "git worktree remove failed") + _remove_worktree(checkout, destination, force) authorization = run_git(checkout, "update-ref", "-d", member_task_authorization_ref(project, task_id)) if authorization.returncode != 0: raise IsolationError((authorization.stderr or authorization.stdout).strip() or "git update-ref failed") @@ -772,6 +841,7 @@ def retire_member_task( deleted = run_git(checkout, "update-ref", "-d", ref, tip.stdout.strip()) if deleted.returncode != 0: raise IsolationError((deleted.stderr or deleted.stdout).strip() or "git update-ref failed") + release_workspace_quietly(checkout) return copy_rejected @@ -1392,8 +1462,9 @@ def retire_member_verify(created: Dict[str, Dict[str, str]]) -> None: """Remove the member siblings of a coordinator verify sidecar; they hold no output.""" for record in reversed(list(created.values())): checkout, destination, branch = _member_verify_sidecar(record) - git_output(checkout, "worktree", "remove", "--force", str(destination)) + _remove_worktree(checkout, destination, True) git_output(checkout, "branch", "-D", branch) + release_workspace_quietly(checkout) def check_member_verify(created: Dict[str, Dict[str, str]]) -> None: @@ -1425,6 +1496,89 @@ def clean_verify(primary: Path, worktree: Path, base: str, branch: str) -> Dict[ return {"worktree": str(worktree), "branch": branch, "base": expected, "removed": removed} +def ensure_verify_sidecar(primary: Path, base: str, branch: str) -> Dict[str, object]: + """Return the named verify sidecar, recreating it when it is missing. + + Finish needs the sidecar prepare-task created. A deleted sidecar must be + rebuildable from the recorded base without a clean primary: the serial + task's own product changes are exactly the tree finish reproduces next, + so this must not reuse isolate_verify's clean-tree gate. + """ + primary = require_directory(primary, "primary worktree") + if worktree_root(primary) != primary: + raise IsolationError(f"primary is not its Git root: {primary}") + resolved_base = require_commit(primary, require_full_sha(base)) + if not branch.startswith(VERIFY_BRANCH_PREFIX): + raise IsolationError("verify sidecar reproduction requires a verify branch") + destination = _sidecar_path_for_branch(primary, branch) + ref = f"refs/heads/{branch}" + registered = _registered_worktrees(primary) + if destination.is_symlink() or (destination.exists() and not destination.is_dir()): + raise IsolationError(f"worktree path already exists: {destination}") + if destination.is_dir(): + try: + # A leftover empty directory (a half-deleted sidecar) clears; + # anything non-empty must be the sidecar itself. + destination.rmdir() + except OSError: + try: + same = ( + worktree_root(destination) == destination + and destination.resolve() in registered + and registered[destination.resolve()] == ref + and common_git_dir(destination) == common_git_dir(primary) + ) + except IsolationError: + same = False + if not same: + raise IsolationError( + f"verify sidecar ownership changed: {destination}" + ) from None + if ( + require_attached(destination) != branch + or current_sha(destination) != resolved_base + ): + raise IsolationError(f"verify sidecar HEAD changed: {destination}") + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + if destination.resolve() in registered: + cleared = run_git(primary, "worktree", "remove", str(destination)) + if cleared.returncode != 0: + run_git(primary, "worktree", "prune") + if destination.resolve() in _registered_worktrees(primary): + raise IsolationError( + (cleared.stderr or cleared.stdout).strip() + or "could not clear stale worktree registration" + ) + existing = run_git(primary, "show-ref", "--verify", "--quiet", ref) + if existing.returncode == 0: + deleted = run_git(primary, "update-ref", "-d", ref) + if deleted.returncode != 0: + raise IsolationError( + (deleted.stderr or deleted.stdout).strip() or "git update-ref failed" + ) + create_named_worktree(primary, branch, destination, resolved_base) + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + + +def reproduce_verify_sidecar(sidecar_path: Path, tree: str) -> None: + """Copy the primary's current task tree into the verify sidecar (clean_verify's inverse).""" + git_output(sidecar_path, "read-tree", "-u", "--reset", tree) + if git_output(sidecar_path, "write-tree") != tree: + raise IsolationError("sidecar reproduction differs from the primary task diff") + + def _split_paths(block: str) -> Set[str]: return {line.strip() for line in block.splitlines() if line.strip()} @@ -1537,14 +1691,24 @@ def _commit_pending( def validate_allowed_changes( - repo: Path, base: str, pending: Set[str], allowed: Set[str] + repo: Path, + base: str, + pending: Set[str], + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> None: - if not pending: + # Tolerated paths (primary bookkeeping) may stay uncommitted; the next + # `.project` checkpoint commits them. + landing = pending - tolerated + if not landing: raise IsolationError("no changes to land") - since_base = committed_paths_since(repo, base) | pending + since_base = committed_paths_since(repo, base) | landing unexpected = sorted(path for path in since_base if path not in allowed) if unexpected: - raise IsolationError("unexpected paths: " + ", ".join(unexpected)) + raise IsolationError( + "unexpected paths: " + ", ".join(unexpected) + + "; landing allows only the task file and its declared files" + ) def _clean_blob_oid(repo: Path, path: str, text: str, *, write: bool) -> str: arguments = ["hash-object"] if write: @@ -1564,21 +1728,23 @@ def commit_allowed_changes( subject: str, allowed: Set[str], body: str, + tolerated: Set[str] = frozenset(), ) -> tuple[str, str, Dict[str, Optional[tuple[str, str]]]]: require_attached(repo) pending = uncommitted_paths(repo) - validate_allowed_changes(repo, base, pending, allowed) + validate_allowed_changes(repo, base, pending, allowed, tolerated) + staging = pending - tolerated reset = run_git(repo, "reset", "-q", "HEAD") if reset.returncode != 0: raise IsolationError("could not clear the index before landing") - for path in sorted(pending): + for path in sorted(staging): added = run_git(repo, "add", "-A", "--", path) if added.returncode != 0: raise IsolationError(f"could not stage {path}") staged = _split_paths( git_output(repo, "diff", "--cached", "--name-only", "--relative") ) - if staged != pending: + if staged != staging: raise IsolationError("staged paths do not match the uncommitted change set") verified_tree = git_output(repo, "write-tree") verified_delta = _tree_delta(repo, base, verified_tree) @@ -1724,6 +1890,17 @@ def activate_task( if task_branch is None: if branch.startswith(TASK_BRANCH_PREFIX): raise IsolationError("parallel task activation requires --task-branch") + live = [ + path + for path, ref in _registered_worktrees(worktree).items() + if ref == f"refs/heads/{task_branch_name(task_id)}" + and path != worktree + ] + if live: + raise IsolationError( + "task has an active parallel worktree; serial activation in " + f"another worktree is refused (activate {live[0]} with --task-branch)" + ) elif task_branch != task_branch_name(task_id) or branch != task_branch: raise IsolationError("task activation branch does not match the task worktree") normalized_task = relative_posix(task_file) @@ -1991,7 +2168,12 @@ def _restore_rejected_commit( def _stamp_and_commit( - worktree: Path, base: str, subject: str, task_file: str, allowed: Set[str] + worktree: Path, + base: str, + subject: str, + task_file: str, + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> tuple[str, Dict[str, Optional[tuple[str, str]]]]: task_path = worktree / task_file task_text, task_mode = _read_task_text(task_path) @@ -2012,14 +2194,14 @@ def _stamp_and_commit( pending = uncommitted_paths(worktree) if stamped != task_bytes: pending.add(task_file) - validate_allowed_changes(worktree, base, pending, allowed) + validate_allowed_changes(worktree, base, pending, allowed, tolerated) original_head = current_sha(worktree) index_tree = git_output(worktree, "write-tree") try: _replace_regular_file(task_path, stamped, task_mode) - body = task_commit_body(task_file, uncommitted_paths(worktree), base) + body = task_commit_body(task_file, uncommitted_paths(worktree) - tolerated, base) commit, verified_tree, verified_delta = commit_allowed_changes( - worktree, base, subject, allowed, body + worktree, base, subject, allowed, body, tolerated ) proof_error = _landing_commit_proof_error( worktree, @@ -2097,6 +2279,7 @@ def land( subject, relative_posix(task_file), allowed, + BOOKKEEPING_PATHS, ) return { "bound_branch": bound, @@ -2391,7 +2574,7 @@ def _landing_retry_error( if task_file not in pending: return "interrupted landing does not retain the task change" try: - validate_allowed_changes(repo, base, pending, contract_paths) + validate_allowed_changes(repo, base, pending, contract_paths, BOOKKEEPING_PATHS) except IsolationError as validation_error: return str(validation_error) return None @@ -4430,6 +4613,7 @@ def retire( or f"could not delete {branch}" ) branch_retired = True + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": branch, @@ -4463,22 +4647,13 @@ def retire( raise IsolationError( "worktree is dirty; pass --force only from the retry-retirement path" ) - remove = run_git( - primary, - "worktree", - "remove", - *(("--force",) if force else ()), - str(resolved_worktree), - ) - if remove.returncode != 0: - raise IsolationError( - (remove.stderr or remove.stdout).strip() or "git worktree remove failed" - ) + _remove_worktree(primary, resolved_worktree, force) deleted = run_git(primary, "branch", "-D", retire_branch) if deleted.returncode != 0: raise IsolationError( (deleted.stderr or deleted.stdout).strip() or f"could not delete {retire_branch}" ) + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": retire_branch, diff --git a/skills/gsd-path-plan/scripts/members.py b/skills/gsd-path-plan/scripts/members.py index d427c6ee4..7f516c846 100644 --- a/skills/gsd-path-plan/scripts/members.py +++ b/skills/gsd-path-plan/scripts/members.py @@ -255,6 +255,7 @@ def check_member( coordinator_name: str, checkout: Path, recorded_remote: Optional[str] = None, + refuse_reserved_refs: bool = True, ) -> str: """Refuse a checkout that cannot join; return its origin URL.""" if checkout.is_symlink() or not checkout.is_dir(): @@ -294,13 +295,14 @@ def check_member( raise MembersError(f"member STATE.md is unreadable: {error}") from error if state.milestone is not None and (state.phase, state.status) != ("shipped", "done"): raise MembersError(f"member has an active milestone: {state.milestone}") - prefixes = member_ref_prefixes(coordinator_name) + tuple( - "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS - ) - refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() - colliding = [ref for ref in refs if ref.startswith(prefixes)] - if colliding: - raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) + if refuse_reserved_refs: + prefixes = member_ref_prefixes(coordinator_name) + tuple( + "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS + ) + refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() + colliding = [ref for ref in refs if ref.startswith(prefixes)] + if colliding: + raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) return remote @@ -620,7 +622,7 @@ def validate_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) try: role = member_role(checkout) except MembersError as error: @@ -639,7 +641,7 @@ def repair_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) _refuse_foreign_marker(checkout.resolve(), root, state.project, member["name"]) for member in members: checkout = Path(member["checkout"]).resolve() diff --git a/skills/gsd-path-plan/scripts/worktree_paths.py b/skills/gsd-path-plan/scripts/worktree_paths.py index 19653ca92..b2cfda32f 100644 --- a/skills/gsd-path-plan/scripts/worktree_paths.py +++ b/skills/gsd-path-plan/scripts/worktree_paths.py @@ -103,6 +103,33 @@ def _workspace(primary: Path, pin: bool) -> Path: return workspace +def release_workspace(primary: Path) -> bool: + """Forget the pinned placement receipt once no linked worktrees remain. + + Lets GSD_PATH_WORKTREE_ROOT (or a new default) apply again at the next + milestone's first allocation. Returns True when a receipt was removed; + any remaining linked worktree (including one only still registered) keeps + the pin. + """ + primary = primary.resolve() + common = Path(_git(primary, "rev-parse", "--git-common-dir")) + common = (primary / common).resolve() + receipt = common / "gsd-path" / "workspaces" / f"{_identity(primary)}.json" + if not receipt.is_file(): + return False + output, nul_separated = _worktree_list_porcelain(primary) + records = output.split("\0\0") if nul_separated else [ + block for block in output.strip().split("\n\n") if block.strip() + ] + for record in records: + fields = record.split("\0") if nul_separated else record.splitlines() + for field in fields: + if field.startswith("worktree ") and Path(field[len("worktree "):]).resolve() != primary: + return False + receipt.unlink() + return True + + def worktree_path(primary: Path, kind: str, name: str, *, pin: bool = False) -> Path: primary = primary.resolve() if kind not in {"task", "verify", "integrate", "bound"} or not name or Path(name).name != name or name in {".", ".."}: diff --git a/skills/gsd-path-plan/templates/plan.md b/skills/gsd-path-plan/templates/plan.md index a2c8ea8ec..2b49a23f4 100644 --- a/skills/gsd-path-plan/templates/plan.md +++ b/skills/gsd-path-plan/templates/plan.md @@ -47,8 +47,8 @@ Goal: Review depth: ## Surface contract diff --git a/skills/gsd-path-research/scripts/check_handoffs.py b/skills/gsd-path-research/scripts/check_handoffs.py index f86fa4306..743f7cdf9 100644 --- a/skills/gsd-path-research/scripts/check_handoffs.py +++ b/skills/gsd-path-research/scripts/check_handoffs.py @@ -1386,6 +1386,20 @@ def _task_deps(task_text: str, task_id: str) -> List[str]: return _inline_ids(rendered, "T", f"{task_id} deps") +def _dependency_reachable(graph: Dict[str, List[str]], start: str, target: str) -> bool: + seen: Set[str] = set() + pending = list(graph[start]) + while pending: + task_id = pending.pop() + if task_id in seen: + continue + if task_id == target: + return True + seen.add(task_id) + pending.extend(graph[task_id]) + return False + + def _canonical_repo_path(value: str, label: str) -> str: cleaned = value.strip().strip("`\"'") path = PurePosixPath(cleaned) @@ -1509,6 +1523,12 @@ def _validate_task_graph( and _task_scalar(tasks[right], right, "status") == "done" ): continue + # Dependency-ordered tasks serialize safely; only unordered + # same-wave overlap races on a shared file. + if _dependency_reachable(graph, left, right) or _dependency_reachable( + graph, right, left + ): + continue raise HandoffError( f"same-wave file overlap between {left} and {right}: {', '.join(overlap)}" ) diff --git a/skills/gsd-path-research/scripts/integration.py b/skills/gsd-path-research/scripts/integration.py index 0dd2a356b..c1b0971d5 100644 --- a/skills/gsd-path-research/scripts/integration.py +++ b/skills/gsd-path-research/scripts/integration.py @@ -9,6 +9,7 @@ import json import re +import contextlib from pathlib import Path, PurePosixPath from typing import Optional @@ -1939,6 +1940,10 @@ def retire_member(coordinator: Path, member: str, archive_path: str, reviewed_he for kind, ref in (("push", bound_ref), ("push", "refs/heads/main"), ("push", f"refs/tags/{tag_name}"), ("delete", bound_ref)): members.clear_authorization(checkout, project, kind, ref) + # The last sidecar retires here at milestone end; isolation's retires also + # release the pin whenever the workspace is idle, so a new root can apply. + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(checkout) return {"member": member, "retired": bound} diff --git a/skills/gsd-path-research/scripts/isolation.py b/skills/gsd-path-research/scripts/isolation.py index 50a9ef684..3805f809a 100644 --- a/skills/gsd-path-research/scripts/isolation.py +++ b/skills/gsd-path-research/scripts/isolation.py @@ -15,6 +15,7 @@ import io import json import shlex +import shutil import os import re import stat @@ -171,6 +172,12 @@ def sidecar_root(primary: Path, kind: str, name: str, *, pin: bool = False) -> P raise IsolationError(str(error)) from error +def release_workspace_quietly(primary: Path) -> None: + """Best-effort placement hygiene after a sidecar retires; never blocks landing.""" + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(primary) + + def validate_task_id(task_id: str) -> str: if not TASK_ID_RE.fullmatch(task_id): raise IsolationError(f"invalid task id: {task_id}") @@ -477,6 +484,70 @@ def _same_directory(path: Path, expected: os.stat_result) -> bool: return stat.S_ISDIR(current.st_mode) and os.path.samestat(current, expected) +def _long_path(absolute: str) -> str: + if absolute.startswith("\\\\?\\"): + return absolute + if absolute.startswith("\\\\"): + return "\\\\?\\UNC\\" + absolute[2:] + return "\\\\?\\" + absolute + + +def _force_rmtree(path: Path) -> None: + """Delete a sidecar tree directly, tolerating read-only files and long paths.""" + def _writable(function, target, _error): + os.chmod(target, stat.S_IWRITE) + function(target) + + # The \\?\ prefix sidesteps Windows MAX_PATH on deep sidecar trees. + target = Path(_long_path(os.path.abspath(path))) if os.name == "nt" else path + try: + shutil.rmtree(target, onerror=_writable) + except OSError as error: + raise IsolationError(f"could not delete {path}: {error}") from error + + +def _remove_worktree(primary: Path, destination: Path, force: bool) -> None: + """Remove a sidecar worktree, tolerating populated submodules and locked trees. + + git refuses to remove worktrees whose submodules were ever initialized, and + forced removal can still fail deleting deep trees (Windows "Directory not + empty"). A dirty sidecar keeps git's refusal unless `force`, so forced + retries only run for a provably clean tree. The direct-delete fallback runs + only for git's tree-deletion failures, so any other git error leaves the + sidecar intact for diagnosis. + """ + tree_delete_failure = ("failed to delete", "directory not empty", "permission denied") + + def failure(*flags: str) -> str: + result = run_git(primary, "worktree", "remove", *flags, str(destination)) + return (result.stderr or result.stdout).strip() if result.returncode else "" + + detail = failure("--force") if force else failure() + if not detail: + return + if not force and destination.is_dir(): + status = run_git( + destination, "status", "--porcelain", "--untracked-files=all" + ) + if status.returncode != 0 or status.stdout.strip(): + raise IsolationError(detail or "git worktree remove failed") + run_git(destination, "submodule", "deinit", "--all", "--force") + detail = failure("--force") + if not detail: + return + if destination.is_dir() and any( + marker in detail.lower() for marker in tree_delete_failure + ): + _force_rmtree(destination) + detail = failure("--force") + if detail and not destination.is_dir(): + pruned = run_git(primary, "worktree", "prune") + if pruned.returncode == 0: + detail = "" + if detail or destination.is_dir() or destination.resolve() in _registered_worktrees(primary): + raise IsolationError(detail or "git worktree remove failed") + + def _cleanup_reserved_worktree( primary: Path, branch: str, @@ -762,9 +833,7 @@ def retire_member_task( if destination.resolve() in registered: if registered[destination.resolve()] != ref: raise IsolationError(f"member task sidecar is on another branch: {destination}") - removed = run_git(checkout, "worktree", "remove", *(("--force",) if force else ()), str(destination)) - if removed.returncode != 0: - raise IsolationError((removed.stderr or removed.stdout).strip() or "git worktree remove failed") + _remove_worktree(checkout, destination, force) authorization = run_git(checkout, "update-ref", "-d", member_task_authorization_ref(project, task_id)) if authorization.returncode != 0: raise IsolationError((authorization.stderr or authorization.stdout).strip() or "git update-ref failed") @@ -772,6 +841,7 @@ def retire_member_task( deleted = run_git(checkout, "update-ref", "-d", ref, tip.stdout.strip()) if deleted.returncode != 0: raise IsolationError((deleted.stderr or deleted.stdout).strip() or "git update-ref failed") + release_workspace_quietly(checkout) return copy_rejected @@ -1392,8 +1462,9 @@ def retire_member_verify(created: Dict[str, Dict[str, str]]) -> None: """Remove the member siblings of a coordinator verify sidecar; they hold no output.""" for record in reversed(list(created.values())): checkout, destination, branch = _member_verify_sidecar(record) - git_output(checkout, "worktree", "remove", "--force", str(destination)) + _remove_worktree(checkout, destination, True) git_output(checkout, "branch", "-D", branch) + release_workspace_quietly(checkout) def check_member_verify(created: Dict[str, Dict[str, str]]) -> None: @@ -1425,6 +1496,89 @@ def clean_verify(primary: Path, worktree: Path, base: str, branch: str) -> Dict[ return {"worktree": str(worktree), "branch": branch, "base": expected, "removed": removed} +def ensure_verify_sidecar(primary: Path, base: str, branch: str) -> Dict[str, object]: + """Return the named verify sidecar, recreating it when it is missing. + + Finish needs the sidecar prepare-task created. A deleted sidecar must be + rebuildable from the recorded base without a clean primary: the serial + task's own product changes are exactly the tree finish reproduces next, + so this must not reuse isolate_verify's clean-tree gate. + """ + primary = require_directory(primary, "primary worktree") + if worktree_root(primary) != primary: + raise IsolationError(f"primary is not its Git root: {primary}") + resolved_base = require_commit(primary, require_full_sha(base)) + if not branch.startswith(VERIFY_BRANCH_PREFIX): + raise IsolationError("verify sidecar reproduction requires a verify branch") + destination = _sidecar_path_for_branch(primary, branch) + ref = f"refs/heads/{branch}" + registered = _registered_worktrees(primary) + if destination.is_symlink() or (destination.exists() and not destination.is_dir()): + raise IsolationError(f"worktree path already exists: {destination}") + if destination.is_dir(): + try: + # A leftover empty directory (a half-deleted sidecar) clears; + # anything non-empty must be the sidecar itself. + destination.rmdir() + except OSError: + try: + same = ( + worktree_root(destination) == destination + and destination.resolve() in registered + and registered[destination.resolve()] == ref + and common_git_dir(destination) == common_git_dir(primary) + ) + except IsolationError: + same = False + if not same: + raise IsolationError( + f"verify sidecar ownership changed: {destination}" + ) from None + if ( + require_attached(destination) != branch + or current_sha(destination) != resolved_base + ): + raise IsolationError(f"verify sidecar HEAD changed: {destination}") + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + if destination.resolve() in registered: + cleared = run_git(primary, "worktree", "remove", str(destination)) + if cleared.returncode != 0: + run_git(primary, "worktree", "prune") + if destination.resolve() in _registered_worktrees(primary): + raise IsolationError( + (cleared.stderr or cleared.stdout).strip() + or "could not clear stale worktree registration" + ) + existing = run_git(primary, "show-ref", "--verify", "--quiet", ref) + if existing.returncode == 0: + deleted = run_git(primary, "update-ref", "-d", ref) + if deleted.returncode != 0: + raise IsolationError( + (deleted.stderr or deleted.stdout).strip() or "git update-ref failed" + ) + create_named_worktree(primary, branch, destination, resolved_base) + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + + +def reproduce_verify_sidecar(sidecar_path: Path, tree: str) -> None: + """Copy the primary's current task tree into the verify sidecar (clean_verify's inverse).""" + git_output(sidecar_path, "read-tree", "-u", "--reset", tree) + if git_output(sidecar_path, "write-tree") != tree: + raise IsolationError("sidecar reproduction differs from the primary task diff") + + def _split_paths(block: str) -> Set[str]: return {line.strip() for line in block.splitlines() if line.strip()} @@ -1537,14 +1691,24 @@ def _commit_pending( def validate_allowed_changes( - repo: Path, base: str, pending: Set[str], allowed: Set[str] + repo: Path, + base: str, + pending: Set[str], + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> None: - if not pending: + # Tolerated paths (primary bookkeeping) may stay uncommitted; the next + # `.project` checkpoint commits them. + landing = pending - tolerated + if not landing: raise IsolationError("no changes to land") - since_base = committed_paths_since(repo, base) | pending + since_base = committed_paths_since(repo, base) | landing unexpected = sorted(path for path in since_base if path not in allowed) if unexpected: - raise IsolationError("unexpected paths: " + ", ".join(unexpected)) + raise IsolationError( + "unexpected paths: " + ", ".join(unexpected) + + "; landing allows only the task file and its declared files" + ) def _clean_blob_oid(repo: Path, path: str, text: str, *, write: bool) -> str: arguments = ["hash-object"] if write: @@ -1564,21 +1728,23 @@ def commit_allowed_changes( subject: str, allowed: Set[str], body: str, + tolerated: Set[str] = frozenset(), ) -> tuple[str, str, Dict[str, Optional[tuple[str, str]]]]: require_attached(repo) pending = uncommitted_paths(repo) - validate_allowed_changes(repo, base, pending, allowed) + validate_allowed_changes(repo, base, pending, allowed, tolerated) + staging = pending - tolerated reset = run_git(repo, "reset", "-q", "HEAD") if reset.returncode != 0: raise IsolationError("could not clear the index before landing") - for path in sorted(pending): + for path in sorted(staging): added = run_git(repo, "add", "-A", "--", path) if added.returncode != 0: raise IsolationError(f"could not stage {path}") staged = _split_paths( git_output(repo, "diff", "--cached", "--name-only", "--relative") ) - if staged != pending: + if staged != staging: raise IsolationError("staged paths do not match the uncommitted change set") verified_tree = git_output(repo, "write-tree") verified_delta = _tree_delta(repo, base, verified_tree) @@ -1724,6 +1890,17 @@ def activate_task( if task_branch is None: if branch.startswith(TASK_BRANCH_PREFIX): raise IsolationError("parallel task activation requires --task-branch") + live = [ + path + for path, ref in _registered_worktrees(worktree).items() + if ref == f"refs/heads/{task_branch_name(task_id)}" + and path != worktree + ] + if live: + raise IsolationError( + "task has an active parallel worktree; serial activation in " + f"another worktree is refused (activate {live[0]} with --task-branch)" + ) elif task_branch != task_branch_name(task_id) or branch != task_branch: raise IsolationError("task activation branch does not match the task worktree") normalized_task = relative_posix(task_file) @@ -1991,7 +2168,12 @@ def _restore_rejected_commit( def _stamp_and_commit( - worktree: Path, base: str, subject: str, task_file: str, allowed: Set[str] + worktree: Path, + base: str, + subject: str, + task_file: str, + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> tuple[str, Dict[str, Optional[tuple[str, str]]]]: task_path = worktree / task_file task_text, task_mode = _read_task_text(task_path) @@ -2012,14 +2194,14 @@ def _stamp_and_commit( pending = uncommitted_paths(worktree) if stamped != task_bytes: pending.add(task_file) - validate_allowed_changes(worktree, base, pending, allowed) + validate_allowed_changes(worktree, base, pending, allowed, tolerated) original_head = current_sha(worktree) index_tree = git_output(worktree, "write-tree") try: _replace_regular_file(task_path, stamped, task_mode) - body = task_commit_body(task_file, uncommitted_paths(worktree), base) + body = task_commit_body(task_file, uncommitted_paths(worktree) - tolerated, base) commit, verified_tree, verified_delta = commit_allowed_changes( - worktree, base, subject, allowed, body + worktree, base, subject, allowed, body, tolerated ) proof_error = _landing_commit_proof_error( worktree, @@ -2097,6 +2279,7 @@ def land( subject, relative_posix(task_file), allowed, + BOOKKEEPING_PATHS, ) return { "bound_branch": bound, @@ -2391,7 +2574,7 @@ def _landing_retry_error( if task_file not in pending: return "interrupted landing does not retain the task change" try: - validate_allowed_changes(repo, base, pending, contract_paths) + validate_allowed_changes(repo, base, pending, contract_paths, BOOKKEEPING_PATHS) except IsolationError as validation_error: return str(validation_error) return None @@ -4430,6 +4613,7 @@ def retire( or f"could not delete {branch}" ) branch_retired = True + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": branch, @@ -4463,22 +4647,13 @@ def retire( raise IsolationError( "worktree is dirty; pass --force only from the retry-retirement path" ) - remove = run_git( - primary, - "worktree", - "remove", - *(("--force",) if force else ()), - str(resolved_worktree), - ) - if remove.returncode != 0: - raise IsolationError( - (remove.stderr or remove.stdout).strip() or "git worktree remove failed" - ) + _remove_worktree(primary, resolved_worktree, force) deleted = run_git(primary, "branch", "-D", retire_branch) if deleted.returncode != 0: raise IsolationError( (deleted.stderr or deleted.stdout).strip() or f"could not delete {retire_branch}" ) + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": retire_branch, diff --git a/skills/gsd-path-research/scripts/members.py b/skills/gsd-path-research/scripts/members.py index d427c6ee4..7f516c846 100644 --- a/skills/gsd-path-research/scripts/members.py +++ b/skills/gsd-path-research/scripts/members.py @@ -255,6 +255,7 @@ def check_member( coordinator_name: str, checkout: Path, recorded_remote: Optional[str] = None, + refuse_reserved_refs: bool = True, ) -> str: """Refuse a checkout that cannot join; return its origin URL.""" if checkout.is_symlink() or not checkout.is_dir(): @@ -294,13 +295,14 @@ def check_member( raise MembersError(f"member STATE.md is unreadable: {error}") from error if state.milestone is not None and (state.phase, state.status) != ("shipped", "done"): raise MembersError(f"member has an active milestone: {state.milestone}") - prefixes = member_ref_prefixes(coordinator_name) + tuple( - "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS - ) - refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() - colliding = [ref for ref in refs if ref.startswith(prefixes)] - if colliding: - raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) + if refuse_reserved_refs: + prefixes = member_ref_prefixes(coordinator_name) + tuple( + "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS + ) + refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() + colliding = [ref for ref in refs if ref.startswith(prefixes)] + if colliding: + raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) return remote @@ -620,7 +622,7 @@ def validate_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) try: role = member_role(checkout) except MembersError as error: @@ -639,7 +641,7 @@ def repair_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) _refuse_foreign_marker(checkout.resolve(), root, state.project, member["name"]) for member in members: checkout = Path(member["checkout"]).resolve() diff --git a/skills/gsd-path-research/scripts/worktree_paths.py b/skills/gsd-path-research/scripts/worktree_paths.py index 19653ca92..b2cfda32f 100644 --- a/skills/gsd-path-research/scripts/worktree_paths.py +++ b/skills/gsd-path-research/scripts/worktree_paths.py @@ -103,6 +103,33 @@ def _workspace(primary: Path, pin: bool) -> Path: return workspace +def release_workspace(primary: Path) -> bool: + """Forget the pinned placement receipt once no linked worktrees remain. + + Lets GSD_PATH_WORKTREE_ROOT (or a new default) apply again at the next + milestone's first allocation. Returns True when a receipt was removed; + any remaining linked worktree (including one only still registered) keeps + the pin. + """ + primary = primary.resolve() + common = Path(_git(primary, "rev-parse", "--git-common-dir")) + common = (primary / common).resolve() + receipt = common / "gsd-path" / "workspaces" / f"{_identity(primary)}.json" + if not receipt.is_file(): + return False + output, nul_separated = _worktree_list_porcelain(primary) + records = output.split("\0\0") if nul_separated else [ + block for block in output.strip().split("\n\n") if block.strip() + ] + for record in records: + fields = record.split("\0") if nul_separated else record.splitlines() + for field in fields: + if field.startswith("worktree ") and Path(field[len("worktree "):]).resolve() != primary: + return False + receipt.unlink() + return True + + def worktree_path(primary: Path, kind: str, name: str, *, pin: bool = False) -> Path: primary = primary.resolve() if kind not in {"task", "verify", "integrate", "bound"} or not name or Path(name).name != name or name in {".", ".."}: diff --git a/skills/gsd-path-roadmap/scripts/check_handoffs.py b/skills/gsd-path-roadmap/scripts/check_handoffs.py index f86fa4306..743f7cdf9 100644 --- a/skills/gsd-path-roadmap/scripts/check_handoffs.py +++ b/skills/gsd-path-roadmap/scripts/check_handoffs.py @@ -1386,6 +1386,20 @@ def _task_deps(task_text: str, task_id: str) -> List[str]: return _inline_ids(rendered, "T", f"{task_id} deps") +def _dependency_reachable(graph: Dict[str, List[str]], start: str, target: str) -> bool: + seen: Set[str] = set() + pending = list(graph[start]) + while pending: + task_id = pending.pop() + if task_id in seen: + continue + if task_id == target: + return True + seen.add(task_id) + pending.extend(graph[task_id]) + return False + + def _canonical_repo_path(value: str, label: str) -> str: cleaned = value.strip().strip("`\"'") path = PurePosixPath(cleaned) @@ -1509,6 +1523,12 @@ def _validate_task_graph( and _task_scalar(tasks[right], right, "status") == "done" ): continue + # Dependency-ordered tasks serialize safely; only unordered + # same-wave overlap races on a shared file. + if _dependency_reachable(graph, left, right) or _dependency_reachable( + graph, right, left + ): + continue raise HandoffError( f"same-wave file overlap between {left} and {right}: {', '.join(overlap)}" ) diff --git a/skills/gsd-path-roadmap/scripts/integration.py b/skills/gsd-path-roadmap/scripts/integration.py index 0dd2a356b..c1b0971d5 100644 --- a/skills/gsd-path-roadmap/scripts/integration.py +++ b/skills/gsd-path-roadmap/scripts/integration.py @@ -9,6 +9,7 @@ import json import re +import contextlib from pathlib import Path, PurePosixPath from typing import Optional @@ -1939,6 +1940,10 @@ def retire_member(coordinator: Path, member: str, archive_path: str, reviewed_he for kind, ref in (("push", bound_ref), ("push", "refs/heads/main"), ("push", f"refs/tags/{tag_name}"), ("delete", bound_ref)): members.clear_authorization(checkout, project, kind, ref) + # The last sidecar retires here at milestone end; isolation's retires also + # release the pin whenever the workspace is idle, so a new root can apply. + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(checkout) return {"member": member, "retired": bound} diff --git a/skills/gsd-path-roadmap/scripts/isolation.py b/skills/gsd-path-roadmap/scripts/isolation.py index 50a9ef684..3805f809a 100644 --- a/skills/gsd-path-roadmap/scripts/isolation.py +++ b/skills/gsd-path-roadmap/scripts/isolation.py @@ -15,6 +15,7 @@ import io import json import shlex +import shutil import os import re import stat @@ -171,6 +172,12 @@ def sidecar_root(primary: Path, kind: str, name: str, *, pin: bool = False) -> P raise IsolationError(str(error)) from error +def release_workspace_quietly(primary: Path) -> None: + """Best-effort placement hygiene after a sidecar retires; never blocks landing.""" + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(primary) + + def validate_task_id(task_id: str) -> str: if not TASK_ID_RE.fullmatch(task_id): raise IsolationError(f"invalid task id: {task_id}") @@ -477,6 +484,70 @@ def _same_directory(path: Path, expected: os.stat_result) -> bool: return stat.S_ISDIR(current.st_mode) and os.path.samestat(current, expected) +def _long_path(absolute: str) -> str: + if absolute.startswith("\\\\?\\"): + return absolute + if absolute.startswith("\\\\"): + return "\\\\?\\UNC\\" + absolute[2:] + return "\\\\?\\" + absolute + + +def _force_rmtree(path: Path) -> None: + """Delete a sidecar tree directly, tolerating read-only files and long paths.""" + def _writable(function, target, _error): + os.chmod(target, stat.S_IWRITE) + function(target) + + # The \\?\ prefix sidesteps Windows MAX_PATH on deep sidecar trees. + target = Path(_long_path(os.path.abspath(path))) if os.name == "nt" else path + try: + shutil.rmtree(target, onerror=_writable) + except OSError as error: + raise IsolationError(f"could not delete {path}: {error}") from error + + +def _remove_worktree(primary: Path, destination: Path, force: bool) -> None: + """Remove a sidecar worktree, tolerating populated submodules and locked trees. + + git refuses to remove worktrees whose submodules were ever initialized, and + forced removal can still fail deleting deep trees (Windows "Directory not + empty"). A dirty sidecar keeps git's refusal unless `force`, so forced + retries only run for a provably clean tree. The direct-delete fallback runs + only for git's tree-deletion failures, so any other git error leaves the + sidecar intact for diagnosis. + """ + tree_delete_failure = ("failed to delete", "directory not empty", "permission denied") + + def failure(*flags: str) -> str: + result = run_git(primary, "worktree", "remove", *flags, str(destination)) + return (result.stderr or result.stdout).strip() if result.returncode else "" + + detail = failure("--force") if force else failure() + if not detail: + return + if not force and destination.is_dir(): + status = run_git( + destination, "status", "--porcelain", "--untracked-files=all" + ) + if status.returncode != 0 or status.stdout.strip(): + raise IsolationError(detail or "git worktree remove failed") + run_git(destination, "submodule", "deinit", "--all", "--force") + detail = failure("--force") + if not detail: + return + if destination.is_dir() and any( + marker in detail.lower() for marker in tree_delete_failure + ): + _force_rmtree(destination) + detail = failure("--force") + if detail and not destination.is_dir(): + pruned = run_git(primary, "worktree", "prune") + if pruned.returncode == 0: + detail = "" + if detail or destination.is_dir() or destination.resolve() in _registered_worktrees(primary): + raise IsolationError(detail or "git worktree remove failed") + + def _cleanup_reserved_worktree( primary: Path, branch: str, @@ -762,9 +833,7 @@ def retire_member_task( if destination.resolve() in registered: if registered[destination.resolve()] != ref: raise IsolationError(f"member task sidecar is on another branch: {destination}") - removed = run_git(checkout, "worktree", "remove", *(("--force",) if force else ()), str(destination)) - if removed.returncode != 0: - raise IsolationError((removed.stderr or removed.stdout).strip() or "git worktree remove failed") + _remove_worktree(checkout, destination, force) authorization = run_git(checkout, "update-ref", "-d", member_task_authorization_ref(project, task_id)) if authorization.returncode != 0: raise IsolationError((authorization.stderr or authorization.stdout).strip() or "git update-ref failed") @@ -772,6 +841,7 @@ def retire_member_task( deleted = run_git(checkout, "update-ref", "-d", ref, tip.stdout.strip()) if deleted.returncode != 0: raise IsolationError((deleted.stderr or deleted.stdout).strip() or "git update-ref failed") + release_workspace_quietly(checkout) return copy_rejected @@ -1392,8 +1462,9 @@ def retire_member_verify(created: Dict[str, Dict[str, str]]) -> None: """Remove the member siblings of a coordinator verify sidecar; they hold no output.""" for record in reversed(list(created.values())): checkout, destination, branch = _member_verify_sidecar(record) - git_output(checkout, "worktree", "remove", "--force", str(destination)) + _remove_worktree(checkout, destination, True) git_output(checkout, "branch", "-D", branch) + release_workspace_quietly(checkout) def check_member_verify(created: Dict[str, Dict[str, str]]) -> None: @@ -1425,6 +1496,89 @@ def clean_verify(primary: Path, worktree: Path, base: str, branch: str) -> Dict[ return {"worktree": str(worktree), "branch": branch, "base": expected, "removed": removed} +def ensure_verify_sidecar(primary: Path, base: str, branch: str) -> Dict[str, object]: + """Return the named verify sidecar, recreating it when it is missing. + + Finish needs the sidecar prepare-task created. A deleted sidecar must be + rebuildable from the recorded base without a clean primary: the serial + task's own product changes are exactly the tree finish reproduces next, + so this must not reuse isolate_verify's clean-tree gate. + """ + primary = require_directory(primary, "primary worktree") + if worktree_root(primary) != primary: + raise IsolationError(f"primary is not its Git root: {primary}") + resolved_base = require_commit(primary, require_full_sha(base)) + if not branch.startswith(VERIFY_BRANCH_PREFIX): + raise IsolationError("verify sidecar reproduction requires a verify branch") + destination = _sidecar_path_for_branch(primary, branch) + ref = f"refs/heads/{branch}" + registered = _registered_worktrees(primary) + if destination.is_symlink() or (destination.exists() and not destination.is_dir()): + raise IsolationError(f"worktree path already exists: {destination}") + if destination.is_dir(): + try: + # A leftover empty directory (a half-deleted sidecar) clears; + # anything non-empty must be the sidecar itself. + destination.rmdir() + except OSError: + try: + same = ( + worktree_root(destination) == destination + and destination.resolve() in registered + and registered[destination.resolve()] == ref + and common_git_dir(destination) == common_git_dir(primary) + ) + except IsolationError: + same = False + if not same: + raise IsolationError( + f"verify sidecar ownership changed: {destination}" + ) from None + if ( + require_attached(destination) != branch + or current_sha(destination) != resolved_base + ): + raise IsolationError(f"verify sidecar HEAD changed: {destination}") + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + if destination.resolve() in registered: + cleared = run_git(primary, "worktree", "remove", str(destination)) + if cleared.returncode != 0: + run_git(primary, "worktree", "prune") + if destination.resolve() in _registered_worktrees(primary): + raise IsolationError( + (cleared.stderr or cleared.stdout).strip() + or "could not clear stale worktree registration" + ) + existing = run_git(primary, "show-ref", "--verify", "--quiet", ref) + if existing.returncode == 0: + deleted = run_git(primary, "update-ref", "-d", ref) + if deleted.returncode != 0: + raise IsolationError( + (deleted.stderr or deleted.stdout).strip() or "git update-ref failed" + ) + create_named_worktree(primary, branch, destination, resolved_base) + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + + +def reproduce_verify_sidecar(sidecar_path: Path, tree: str) -> None: + """Copy the primary's current task tree into the verify sidecar (clean_verify's inverse).""" + git_output(sidecar_path, "read-tree", "-u", "--reset", tree) + if git_output(sidecar_path, "write-tree") != tree: + raise IsolationError("sidecar reproduction differs from the primary task diff") + + def _split_paths(block: str) -> Set[str]: return {line.strip() for line in block.splitlines() if line.strip()} @@ -1537,14 +1691,24 @@ def _commit_pending( def validate_allowed_changes( - repo: Path, base: str, pending: Set[str], allowed: Set[str] + repo: Path, + base: str, + pending: Set[str], + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> None: - if not pending: + # Tolerated paths (primary bookkeeping) may stay uncommitted; the next + # `.project` checkpoint commits them. + landing = pending - tolerated + if not landing: raise IsolationError("no changes to land") - since_base = committed_paths_since(repo, base) | pending + since_base = committed_paths_since(repo, base) | landing unexpected = sorted(path for path in since_base if path not in allowed) if unexpected: - raise IsolationError("unexpected paths: " + ", ".join(unexpected)) + raise IsolationError( + "unexpected paths: " + ", ".join(unexpected) + + "; landing allows only the task file and its declared files" + ) def _clean_blob_oid(repo: Path, path: str, text: str, *, write: bool) -> str: arguments = ["hash-object"] if write: @@ -1564,21 +1728,23 @@ def commit_allowed_changes( subject: str, allowed: Set[str], body: str, + tolerated: Set[str] = frozenset(), ) -> tuple[str, str, Dict[str, Optional[tuple[str, str]]]]: require_attached(repo) pending = uncommitted_paths(repo) - validate_allowed_changes(repo, base, pending, allowed) + validate_allowed_changes(repo, base, pending, allowed, tolerated) + staging = pending - tolerated reset = run_git(repo, "reset", "-q", "HEAD") if reset.returncode != 0: raise IsolationError("could not clear the index before landing") - for path in sorted(pending): + for path in sorted(staging): added = run_git(repo, "add", "-A", "--", path) if added.returncode != 0: raise IsolationError(f"could not stage {path}") staged = _split_paths( git_output(repo, "diff", "--cached", "--name-only", "--relative") ) - if staged != pending: + if staged != staging: raise IsolationError("staged paths do not match the uncommitted change set") verified_tree = git_output(repo, "write-tree") verified_delta = _tree_delta(repo, base, verified_tree) @@ -1724,6 +1890,17 @@ def activate_task( if task_branch is None: if branch.startswith(TASK_BRANCH_PREFIX): raise IsolationError("parallel task activation requires --task-branch") + live = [ + path + for path, ref in _registered_worktrees(worktree).items() + if ref == f"refs/heads/{task_branch_name(task_id)}" + and path != worktree + ] + if live: + raise IsolationError( + "task has an active parallel worktree; serial activation in " + f"another worktree is refused (activate {live[0]} with --task-branch)" + ) elif task_branch != task_branch_name(task_id) or branch != task_branch: raise IsolationError("task activation branch does not match the task worktree") normalized_task = relative_posix(task_file) @@ -1991,7 +2168,12 @@ def _restore_rejected_commit( def _stamp_and_commit( - worktree: Path, base: str, subject: str, task_file: str, allowed: Set[str] + worktree: Path, + base: str, + subject: str, + task_file: str, + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> tuple[str, Dict[str, Optional[tuple[str, str]]]]: task_path = worktree / task_file task_text, task_mode = _read_task_text(task_path) @@ -2012,14 +2194,14 @@ def _stamp_and_commit( pending = uncommitted_paths(worktree) if stamped != task_bytes: pending.add(task_file) - validate_allowed_changes(worktree, base, pending, allowed) + validate_allowed_changes(worktree, base, pending, allowed, tolerated) original_head = current_sha(worktree) index_tree = git_output(worktree, "write-tree") try: _replace_regular_file(task_path, stamped, task_mode) - body = task_commit_body(task_file, uncommitted_paths(worktree), base) + body = task_commit_body(task_file, uncommitted_paths(worktree) - tolerated, base) commit, verified_tree, verified_delta = commit_allowed_changes( - worktree, base, subject, allowed, body + worktree, base, subject, allowed, body, tolerated ) proof_error = _landing_commit_proof_error( worktree, @@ -2097,6 +2279,7 @@ def land( subject, relative_posix(task_file), allowed, + BOOKKEEPING_PATHS, ) return { "bound_branch": bound, @@ -2391,7 +2574,7 @@ def _landing_retry_error( if task_file not in pending: return "interrupted landing does not retain the task change" try: - validate_allowed_changes(repo, base, pending, contract_paths) + validate_allowed_changes(repo, base, pending, contract_paths, BOOKKEEPING_PATHS) except IsolationError as validation_error: return str(validation_error) return None @@ -4430,6 +4613,7 @@ def retire( or f"could not delete {branch}" ) branch_retired = True + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": branch, @@ -4463,22 +4647,13 @@ def retire( raise IsolationError( "worktree is dirty; pass --force only from the retry-retirement path" ) - remove = run_git( - primary, - "worktree", - "remove", - *(("--force",) if force else ()), - str(resolved_worktree), - ) - if remove.returncode != 0: - raise IsolationError( - (remove.stderr or remove.stdout).strip() or "git worktree remove failed" - ) + _remove_worktree(primary, resolved_worktree, force) deleted = run_git(primary, "branch", "-D", retire_branch) if deleted.returncode != 0: raise IsolationError( (deleted.stderr or deleted.stdout).strip() or f"could not delete {retire_branch}" ) + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": retire_branch, diff --git a/skills/gsd-path-roadmap/scripts/members.py b/skills/gsd-path-roadmap/scripts/members.py index d427c6ee4..7f516c846 100644 --- a/skills/gsd-path-roadmap/scripts/members.py +++ b/skills/gsd-path-roadmap/scripts/members.py @@ -255,6 +255,7 @@ def check_member( coordinator_name: str, checkout: Path, recorded_remote: Optional[str] = None, + refuse_reserved_refs: bool = True, ) -> str: """Refuse a checkout that cannot join; return its origin URL.""" if checkout.is_symlink() or not checkout.is_dir(): @@ -294,13 +295,14 @@ def check_member( raise MembersError(f"member STATE.md is unreadable: {error}") from error if state.milestone is not None and (state.phase, state.status) != ("shipped", "done"): raise MembersError(f"member has an active milestone: {state.milestone}") - prefixes = member_ref_prefixes(coordinator_name) + tuple( - "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS - ) - refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() - colliding = [ref for ref in refs if ref.startswith(prefixes)] - if colliding: - raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) + if refuse_reserved_refs: + prefixes = member_ref_prefixes(coordinator_name) + tuple( + "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS + ) + refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() + colliding = [ref for ref in refs if ref.startswith(prefixes)] + if colliding: + raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) return remote @@ -620,7 +622,7 @@ def validate_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) try: role = member_role(checkout) except MembersError as error: @@ -639,7 +641,7 @@ def repair_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) _refuse_foreign_marker(checkout.resolve(), root, state.project, member["name"]) for member in members: checkout = Path(member["checkout"]).resolve() diff --git a/skills/gsd-path-roadmap/scripts/worktree_paths.py b/skills/gsd-path-roadmap/scripts/worktree_paths.py index 19653ca92..b2cfda32f 100644 --- a/skills/gsd-path-roadmap/scripts/worktree_paths.py +++ b/skills/gsd-path-roadmap/scripts/worktree_paths.py @@ -103,6 +103,33 @@ def _workspace(primary: Path, pin: bool) -> Path: return workspace +def release_workspace(primary: Path) -> bool: + """Forget the pinned placement receipt once no linked worktrees remain. + + Lets GSD_PATH_WORKTREE_ROOT (or a new default) apply again at the next + milestone's first allocation. Returns True when a receipt was removed; + any remaining linked worktree (including one only still registered) keeps + the pin. + """ + primary = primary.resolve() + common = Path(_git(primary, "rev-parse", "--git-common-dir")) + common = (primary / common).resolve() + receipt = common / "gsd-path" / "workspaces" / f"{_identity(primary)}.json" + if not receipt.is_file(): + return False + output, nul_separated = _worktree_list_porcelain(primary) + records = output.split("\0\0") if nul_separated else [ + block for block in output.strip().split("\n\n") if block.strip() + ] + for record in records: + fields = record.split("\0") if nul_separated else record.splitlines() + for field in fields: + if field.startswith("worktree ") and Path(field[len("worktree "):]).resolve() != primary: + return False + receipt.unlink() + return True + + def worktree_path(primary: Path, kind: str, name: str, *, pin: bool = False) -> Path: primary = primary.resolve() if kind not in {"task", "verify", "integrate", "bound"} or not name or Path(name).name != name or name in {".", ".."}: diff --git a/skills/gsd-path-ship/scripts/check_handoffs.py b/skills/gsd-path-ship/scripts/check_handoffs.py index f86fa4306..743f7cdf9 100644 --- a/skills/gsd-path-ship/scripts/check_handoffs.py +++ b/skills/gsd-path-ship/scripts/check_handoffs.py @@ -1386,6 +1386,20 @@ def _task_deps(task_text: str, task_id: str) -> List[str]: return _inline_ids(rendered, "T", f"{task_id} deps") +def _dependency_reachable(graph: Dict[str, List[str]], start: str, target: str) -> bool: + seen: Set[str] = set() + pending = list(graph[start]) + while pending: + task_id = pending.pop() + if task_id in seen: + continue + if task_id == target: + return True + seen.add(task_id) + pending.extend(graph[task_id]) + return False + + def _canonical_repo_path(value: str, label: str) -> str: cleaned = value.strip().strip("`\"'") path = PurePosixPath(cleaned) @@ -1509,6 +1523,12 @@ def _validate_task_graph( and _task_scalar(tasks[right], right, "status") == "done" ): continue + # Dependency-ordered tasks serialize safely; only unordered + # same-wave overlap races on a shared file. + if _dependency_reachable(graph, left, right) or _dependency_reachable( + graph, right, left + ): + continue raise HandoffError( f"same-wave file overlap between {left} and {right}: {', '.join(overlap)}" ) diff --git a/skills/gsd-path-ship/scripts/integration.py b/skills/gsd-path-ship/scripts/integration.py index 0dd2a356b..c1b0971d5 100644 --- a/skills/gsd-path-ship/scripts/integration.py +++ b/skills/gsd-path-ship/scripts/integration.py @@ -9,6 +9,7 @@ import json import re +import contextlib from pathlib import Path, PurePosixPath from typing import Optional @@ -1939,6 +1940,10 @@ def retire_member(coordinator: Path, member: str, archive_path: str, reviewed_he for kind, ref in (("push", bound_ref), ("push", "refs/heads/main"), ("push", f"refs/tags/{tag_name}"), ("delete", bound_ref)): members.clear_authorization(checkout, project, kind, ref) + # The last sidecar retires here at milestone end; isolation's retires also + # release the pin whenever the workspace is idle, so a new root can apply. + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(checkout) return {"member": member, "retired": bound} diff --git a/skills/gsd-path-ship/scripts/isolation.py b/skills/gsd-path-ship/scripts/isolation.py index 50a9ef684..3805f809a 100644 --- a/skills/gsd-path-ship/scripts/isolation.py +++ b/skills/gsd-path-ship/scripts/isolation.py @@ -15,6 +15,7 @@ import io import json import shlex +import shutil import os import re import stat @@ -171,6 +172,12 @@ def sidecar_root(primary: Path, kind: str, name: str, *, pin: bool = False) -> P raise IsolationError(str(error)) from error +def release_workspace_quietly(primary: Path) -> None: + """Best-effort placement hygiene after a sidecar retires; never blocks landing.""" + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(primary) + + def validate_task_id(task_id: str) -> str: if not TASK_ID_RE.fullmatch(task_id): raise IsolationError(f"invalid task id: {task_id}") @@ -477,6 +484,70 @@ def _same_directory(path: Path, expected: os.stat_result) -> bool: return stat.S_ISDIR(current.st_mode) and os.path.samestat(current, expected) +def _long_path(absolute: str) -> str: + if absolute.startswith("\\\\?\\"): + return absolute + if absolute.startswith("\\\\"): + return "\\\\?\\UNC\\" + absolute[2:] + return "\\\\?\\" + absolute + + +def _force_rmtree(path: Path) -> None: + """Delete a sidecar tree directly, tolerating read-only files and long paths.""" + def _writable(function, target, _error): + os.chmod(target, stat.S_IWRITE) + function(target) + + # The \\?\ prefix sidesteps Windows MAX_PATH on deep sidecar trees. + target = Path(_long_path(os.path.abspath(path))) if os.name == "nt" else path + try: + shutil.rmtree(target, onerror=_writable) + except OSError as error: + raise IsolationError(f"could not delete {path}: {error}") from error + + +def _remove_worktree(primary: Path, destination: Path, force: bool) -> None: + """Remove a sidecar worktree, tolerating populated submodules and locked trees. + + git refuses to remove worktrees whose submodules were ever initialized, and + forced removal can still fail deleting deep trees (Windows "Directory not + empty"). A dirty sidecar keeps git's refusal unless `force`, so forced + retries only run for a provably clean tree. The direct-delete fallback runs + only for git's tree-deletion failures, so any other git error leaves the + sidecar intact for diagnosis. + """ + tree_delete_failure = ("failed to delete", "directory not empty", "permission denied") + + def failure(*flags: str) -> str: + result = run_git(primary, "worktree", "remove", *flags, str(destination)) + return (result.stderr or result.stdout).strip() if result.returncode else "" + + detail = failure("--force") if force else failure() + if not detail: + return + if not force and destination.is_dir(): + status = run_git( + destination, "status", "--porcelain", "--untracked-files=all" + ) + if status.returncode != 0 or status.stdout.strip(): + raise IsolationError(detail or "git worktree remove failed") + run_git(destination, "submodule", "deinit", "--all", "--force") + detail = failure("--force") + if not detail: + return + if destination.is_dir() and any( + marker in detail.lower() for marker in tree_delete_failure + ): + _force_rmtree(destination) + detail = failure("--force") + if detail and not destination.is_dir(): + pruned = run_git(primary, "worktree", "prune") + if pruned.returncode == 0: + detail = "" + if detail or destination.is_dir() or destination.resolve() in _registered_worktrees(primary): + raise IsolationError(detail or "git worktree remove failed") + + def _cleanup_reserved_worktree( primary: Path, branch: str, @@ -762,9 +833,7 @@ def retire_member_task( if destination.resolve() in registered: if registered[destination.resolve()] != ref: raise IsolationError(f"member task sidecar is on another branch: {destination}") - removed = run_git(checkout, "worktree", "remove", *(("--force",) if force else ()), str(destination)) - if removed.returncode != 0: - raise IsolationError((removed.stderr or removed.stdout).strip() or "git worktree remove failed") + _remove_worktree(checkout, destination, force) authorization = run_git(checkout, "update-ref", "-d", member_task_authorization_ref(project, task_id)) if authorization.returncode != 0: raise IsolationError((authorization.stderr or authorization.stdout).strip() or "git update-ref failed") @@ -772,6 +841,7 @@ def retire_member_task( deleted = run_git(checkout, "update-ref", "-d", ref, tip.stdout.strip()) if deleted.returncode != 0: raise IsolationError((deleted.stderr or deleted.stdout).strip() or "git update-ref failed") + release_workspace_quietly(checkout) return copy_rejected @@ -1392,8 +1462,9 @@ def retire_member_verify(created: Dict[str, Dict[str, str]]) -> None: """Remove the member siblings of a coordinator verify sidecar; they hold no output.""" for record in reversed(list(created.values())): checkout, destination, branch = _member_verify_sidecar(record) - git_output(checkout, "worktree", "remove", "--force", str(destination)) + _remove_worktree(checkout, destination, True) git_output(checkout, "branch", "-D", branch) + release_workspace_quietly(checkout) def check_member_verify(created: Dict[str, Dict[str, str]]) -> None: @@ -1425,6 +1496,89 @@ def clean_verify(primary: Path, worktree: Path, base: str, branch: str) -> Dict[ return {"worktree": str(worktree), "branch": branch, "base": expected, "removed": removed} +def ensure_verify_sidecar(primary: Path, base: str, branch: str) -> Dict[str, object]: + """Return the named verify sidecar, recreating it when it is missing. + + Finish needs the sidecar prepare-task created. A deleted sidecar must be + rebuildable from the recorded base without a clean primary: the serial + task's own product changes are exactly the tree finish reproduces next, + so this must not reuse isolate_verify's clean-tree gate. + """ + primary = require_directory(primary, "primary worktree") + if worktree_root(primary) != primary: + raise IsolationError(f"primary is not its Git root: {primary}") + resolved_base = require_commit(primary, require_full_sha(base)) + if not branch.startswith(VERIFY_BRANCH_PREFIX): + raise IsolationError("verify sidecar reproduction requires a verify branch") + destination = _sidecar_path_for_branch(primary, branch) + ref = f"refs/heads/{branch}" + registered = _registered_worktrees(primary) + if destination.is_symlink() or (destination.exists() and not destination.is_dir()): + raise IsolationError(f"worktree path already exists: {destination}") + if destination.is_dir(): + try: + # A leftover empty directory (a half-deleted sidecar) clears; + # anything non-empty must be the sidecar itself. + destination.rmdir() + except OSError: + try: + same = ( + worktree_root(destination) == destination + and destination.resolve() in registered + and registered[destination.resolve()] == ref + and common_git_dir(destination) == common_git_dir(primary) + ) + except IsolationError: + same = False + if not same: + raise IsolationError( + f"verify sidecar ownership changed: {destination}" + ) from None + if ( + require_attached(destination) != branch + or current_sha(destination) != resolved_base + ): + raise IsolationError(f"verify sidecar HEAD changed: {destination}") + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + if destination.resolve() in registered: + cleared = run_git(primary, "worktree", "remove", str(destination)) + if cleared.returncode != 0: + run_git(primary, "worktree", "prune") + if destination.resolve() in _registered_worktrees(primary): + raise IsolationError( + (cleared.stderr or cleared.stdout).strip() + or "could not clear stale worktree registration" + ) + existing = run_git(primary, "show-ref", "--verify", "--quiet", ref) + if existing.returncode == 0: + deleted = run_git(primary, "update-ref", "-d", ref) + if deleted.returncode != 0: + raise IsolationError( + (deleted.stderr or deleted.stdout).strip() or "git update-ref failed" + ) + create_named_worktree(primary, branch, destination, resolved_base) + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + + +def reproduce_verify_sidecar(sidecar_path: Path, tree: str) -> None: + """Copy the primary's current task tree into the verify sidecar (clean_verify's inverse).""" + git_output(sidecar_path, "read-tree", "-u", "--reset", tree) + if git_output(sidecar_path, "write-tree") != tree: + raise IsolationError("sidecar reproduction differs from the primary task diff") + + def _split_paths(block: str) -> Set[str]: return {line.strip() for line in block.splitlines() if line.strip()} @@ -1537,14 +1691,24 @@ def _commit_pending( def validate_allowed_changes( - repo: Path, base: str, pending: Set[str], allowed: Set[str] + repo: Path, + base: str, + pending: Set[str], + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> None: - if not pending: + # Tolerated paths (primary bookkeeping) may stay uncommitted; the next + # `.project` checkpoint commits them. + landing = pending - tolerated + if not landing: raise IsolationError("no changes to land") - since_base = committed_paths_since(repo, base) | pending + since_base = committed_paths_since(repo, base) | landing unexpected = sorted(path for path in since_base if path not in allowed) if unexpected: - raise IsolationError("unexpected paths: " + ", ".join(unexpected)) + raise IsolationError( + "unexpected paths: " + ", ".join(unexpected) + + "; landing allows only the task file and its declared files" + ) def _clean_blob_oid(repo: Path, path: str, text: str, *, write: bool) -> str: arguments = ["hash-object"] if write: @@ -1564,21 +1728,23 @@ def commit_allowed_changes( subject: str, allowed: Set[str], body: str, + tolerated: Set[str] = frozenset(), ) -> tuple[str, str, Dict[str, Optional[tuple[str, str]]]]: require_attached(repo) pending = uncommitted_paths(repo) - validate_allowed_changes(repo, base, pending, allowed) + validate_allowed_changes(repo, base, pending, allowed, tolerated) + staging = pending - tolerated reset = run_git(repo, "reset", "-q", "HEAD") if reset.returncode != 0: raise IsolationError("could not clear the index before landing") - for path in sorted(pending): + for path in sorted(staging): added = run_git(repo, "add", "-A", "--", path) if added.returncode != 0: raise IsolationError(f"could not stage {path}") staged = _split_paths( git_output(repo, "diff", "--cached", "--name-only", "--relative") ) - if staged != pending: + if staged != staging: raise IsolationError("staged paths do not match the uncommitted change set") verified_tree = git_output(repo, "write-tree") verified_delta = _tree_delta(repo, base, verified_tree) @@ -1724,6 +1890,17 @@ def activate_task( if task_branch is None: if branch.startswith(TASK_BRANCH_PREFIX): raise IsolationError("parallel task activation requires --task-branch") + live = [ + path + for path, ref in _registered_worktrees(worktree).items() + if ref == f"refs/heads/{task_branch_name(task_id)}" + and path != worktree + ] + if live: + raise IsolationError( + "task has an active parallel worktree; serial activation in " + f"another worktree is refused (activate {live[0]} with --task-branch)" + ) elif task_branch != task_branch_name(task_id) or branch != task_branch: raise IsolationError("task activation branch does not match the task worktree") normalized_task = relative_posix(task_file) @@ -1991,7 +2168,12 @@ def _restore_rejected_commit( def _stamp_and_commit( - worktree: Path, base: str, subject: str, task_file: str, allowed: Set[str] + worktree: Path, + base: str, + subject: str, + task_file: str, + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> tuple[str, Dict[str, Optional[tuple[str, str]]]]: task_path = worktree / task_file task_text, task_mode = _read_task_text(task_path) @@ -2012,14 +2194,14 @@ def _stamp_and_commit( pending = uncommitted_paths(worktree) if stamped != task_bytes: pending.add(task_file) - validate_allowed_changes(worktree, base, pending, allowed) + validate_allowed_changes(worktree, base, pending, allowed, tolerated) original_head = current_sha(worktree) index_tree = git_output(worktree, "write-tree") try: _replace_regular_file(task_path, stamped, task_mode) - body = task_commit_body(task_file, uncommitted_paths(worktree), base) + body = task_commit_body(task_file, uncommitted_paths(worktree) - tolerated, base) commit, verified_tree, verified_delta = commit_allowed_changes( - worktree, base, subject, allowed, body + worktree, base, subject, allowed, body, tolerated ) proof_error = _landing_commit_proof_error( worktree, @@ -2097,6 +2279,7 @@ def land( subject, relative_posix(task_file), allowed, + BOOKKEEPING_PATHS, ) return { "bound_branch": bound, @@ -2391,7 +2574,7 @@ def _landing_retry_error( if task_file not in pending: return "interrupted landing does not retain the task change" try: - validate_allowed_changes(repo, base, pending, contract_paths) + validate_allowed_changes(repo, base, pending, contract_paths, BOOKKEEPING_PATHS) except IsolationError as validation_error: return str(validation_error) return None @@ -4430,6 +4613,7 @@ def retire( or f"could not delete {branch}" ) branch_retired = True + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": branch, @@ -4463,22 +4647,13 @@ def retire( raise IsolationError( "worktree is dirty; pass --force only from the retry-retirement path" ) - remove = run_git( - primary, - "worktree", - "remove", - *(("--force",) if force else ()), - str(resolved_worktree), - ) - if remove.returncode != 0: - raise IsolationError( - (remove.stderr or remove.stdout).strip() or "git worktree remove failed" - ) + _remove_worktree(primary, resolved_worktree, force) deleted = run_git(primary, "branch", "-D", retire_branch) if deleted.returncode != 0: raise IsolationError( (deleted.stderr or deleted.stdout).strip() or f"could not delete {retire_branch}" ) + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": retire_branch, diff --git a/skills/gsd-path-ship/scripts/members.py b/skills/gsd-path-ship/scripts/members.py index d427c6ee4..7f516c846 100644 --- a/skills/gsd-path-ship/scripts/members.py +++ b/skills/gsd-path-ship/scripts/members.py @@ -255,6 +255,7 @@ def check_member( coordinator_name: str, checkout: Path, recorded_remote: Optional[str] = None, + refuse_reserved_refs: bool = True, ) -> str: """Refuse a checkout that cannot join; return its origin URL.""" if checkout.is_symlink() or not checkout.is_dir(): @@ -294,13 +295,14 @@ def check_member( raise MembersError(f"member STATE.md is unreadable: {error}") from error if state.milestone is not None and (state.phase, state.status) != ("shipped", "done"): raise MembersError(f"member has an active milestone: {state.milestone}") - prefixes = member_ref_prefixes(coordinator_name) + tuple( - "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS - ) - refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() - colliding = [ref for ref in refs if ref.startswith(prefixes)] - if colliding: - raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) + if refuse_reserved_refs: + prefixes = member_ref_prefixes(coordinator_name) + tuple( + "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS + ) + refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() + colliding = [ref for ref in refs if ref.startswith(prefixes)] + if colliding: + raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) return remote @@ -620,7 +622,7 @@ def validate_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) try: role = member_role(checkout) except MembersError as error: @@ -639,7 +641,7 @@ def repair_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) _refuse_foreign_marker(checkout.resolve(), root, state.project, member["name"]) for member in members: checkout = Path(member["checkout"]).resolve() diff --git a/skills/gsd-path-ship/scripts/worktree_paths.py b/skills/gsd-path-ship/scripts/worktree_paths.py index 19653ca92..b2cfda32f 100644 --- a/skills/gsd-path-ship/scripts/worktree_paths.py +++ b/skills/gsd-path-ship/scripts/worktree_paths.py @@ -103,6 +103,33 @@ def _workspace(primary: Path, pin: bool) -> Path: return workspace +def release_workspace(primary: Path) -> bool: + """Forget the pinned placement receipt once no linked worktrees remain. + + Lets GSD_PATH_WORKTREE_ROOT (or a new default) apply again at the next + milestone's first allocation. Returns True when a receipt was removed; + any remaining linked worktree (including one only still registered) keeps + the pin. + """ + primary = primary.resolve() + common = Path(_git(primary, "rev-parse", "--git-common-dir")) + common = (primary / common).resolve() + receipt = common / "gsd-path" / "workspaces" / f"{_identity(primary)}.json" + if not receipt.is_file(): + return False + output, nul_separated = _worktree_list_porcelain(primary) + records = output.split("\0\0") if nul_separated else [ + block for block in output.strip().split("\n\n") if block.strip() + ] + for record in records: + fields = record.split("\0") if nul_separated else record.splitlines() + for field in fields: + if field.startswith("worktree ") and Path(field[len("worktree "):]).resolve() != primary: + return False + receipt.unlink() + return True + + def worktree_path(primary: Path, kind: str, name: str, *, pin: bool = False) -> Path: primary = primary.resolve() if kind not in {"task", "verify", "integrate", "bound"} or not name or Path(name).name != name or name in {".", ".."}: diff --git a/skills/gsd-path-undo/scripts/worktree_paths.py b/skills/gsd-path-undo/scripts/worktree_paths.py index 19653ca92..b2cfda32f 100644 --- a/skills/gsd-path-undo/scripts/worktree_paths.py +++ b/skills/gsd-path-undo/scripts/worktree_paths.py @@ -103,6 +103,33 @@ def _workspace(primary: Path, pin: bool) -> Path: return workspace +def release_workspace(primary: Path) -> bool: + """Forget the pinned placement receipt once no linked worktrees remain. + + Lets GSD_PATH_WORKTREE_ROOT (or a new default) apply again at the next + milestone's first allocation. Returns True when a receipt was removed; + any remaining linked worktree (including one only still registered) keeps + the pin. + """ + primary = primary.resolve() + common = Path(_git(primary, "rev-parse", "--git-common-dir")) + common = (primary / common).resolve() + receipt = common / "gsd-path" / "workspaces" / f"{_identity(primary)}.json" + if not receipt.is_file(): + return False + output, nul_separated = _worktree_list_porcelain(primary) + records = output.split("\0\0") if nul_separated else [ + block for block in output.strip().split("\n\n") if block.strip() + ] + for record in records: + fields = record.split("\0") if nul_separated else record.splitlines() + for field in fields: + if field.startswith("worktree ") and Path(field[len("worktree "):]).resolve() != primary: + return False + receipt.unlink() + return True + + def worktree_path(primary: Path, kind: str, name: str, *, pin: bool = False) -> Path: primary = primary.resolve() if kind not in {"task", "verify", "integrate", "bound"} or not name or Path(name).name != name or name in {".", ".."}: diff --git a/skills/gsd-path/PLAN.md b/skills/gsd-path/PLAN.md index 654eacd1a..a87ea5149 100644 --- a/skills/gsd-path/PLAN.md +++ b/skills/gsd-path/PLAN.md @@ -115,9 +115,10 @@ as stated in Lookahead mode. In patch mode, tasks already `done` keep their landed metadata; only the appended patch tasks must start clean. - Require every dependency id to exist, forbid later-wave dependencies, - detect cycles, and forbid file overlap between planned tasks in the same - wave. Same-wave dependency chains are allowed only when their files do - not overlap; the build executes them in dependency layers. + detect cycles, and forbid unordered file overlap between planned tasks + in the same wave. Same-wave tasks may share files only when one + transitively depends on the other through `deps`; the build executes + them in dependency layers. - Reject a declared dependency that carries neither data nor a prerequisite effect. A `deps` edge is valid only when the dependent consumes a symbol, signature, schema, endpoint, file format, or path named in that diff --git a/skills/gsd-path/references/planner.md b/skills/gsd-path/references/planner.md index 5f4421976..e98a5c613 100644 --- a/skills/gsd-path/references/planner.md +++ b/skills/gsd-path/references/planner.md @@ -38,8 +38,9 @@ the how. security-critical; `verify-only` suits polish and low-risk feature waves whose Verify commands meaningfully cover the criteria. -Every dependency must be in an earlier wave or in the same wave with no file -overlap. Produce an acyclic graph; same-wave chains execute in layers. +Every dependency must be in an earlier wave or in the same wave. Same-wave +tasks may share files only when one transitively depends on the other. Produce +an acyclic graph; same-wave chains execute in layers. Declare a dependency only when something real crosses it: data — the dependent consumes a symbol, signature, schema, endpoint, file format, or path named in @@ -81,7 +82,8 @@ plan order above, which still keeps plan-invalidating risk in wave 1. held-out. Put each `judgment` prohibition's Must not and Detail in the owning task's Approach. - Make `files` exhaustive, including imports, routes, generated artifacts, - tests, and wiring. Require disjoint files for same-wave tasks. + tests, and wiring. Require disjoint files for same-wave tasks that are not + dependency-ordered. - Write an Interface contract in every task: `None` for independent tasks; for any tasks that exchange a symbol, signature, schema, endpoint, file format, or path, the exact shapes both sides code against — identical text diff --git a/skills/gsd-path/scripts/check_handoffs.py b/skills/gsd-path/scripts/check_handoffs.py index f86fa4306..743f7cdf9 100644 --- a/skills/gsd-path/scripts/check_handoffs.py +++ b/skills/gsd-path/scripts/check_handoffs.py @@ -1386,6 +1386,20 @@ def _task_deps(task_text: str, task_id: str) -> List[str]: return _inline_ids(rendered, "T", f"{task_id} deps") +def _dependency_reachable(graph: Dict[str, List[str]], start: str, target: str) -> bool: + seen: Set[str] = set() + pending = list(graph[start]) + while pending: + task_id = pending.pop() + if task_id in seen: + continue + if task_id == target: + return True + seen.add(task_id) + pending.extend(graph[task_id]) + return False + + def _canonical_repo_path(value: str, label: str) -> str: cleaned = value.strip().strip("`\"'") path = PurePosixPath(cleaned) @@ -1509,6 +1523,12 @@ def _validate_task_graph( and _task_scalar(tasks[right], right, "status") == "done" ): continue + # Dependency-ordered tasks serialize safely; only unordered + # same-wave overlap races on a shared file. + if _dependency_reachable(graph, left, right) or _dependency_reachable( + graph, right, left + ): + continue raise HandoffError( f"same-wave file overlap between {left} and {right}: {', '.join(overlap)}" ) diff --git a/skills/gsd-path/scripts/dispatch_driver.py b/skills/gsd-path/scripts/dispatch_driver.py index 010358d9d..c18b1e008 100644 --- a/skills/gsd-path/scripts/dispatch_driver.py +++ b/skills/gsd-path/scripts/dispatch_driver.py @@ -702,14 +702,14 @@ def finish_task(primary: Path, state: Dict[str, object]) -> Dict[str, object]: if not command: raise DriverStop("task has no Verify command", task=task_id) if state["mode"] == "serial": - # ponytail: sidecar reproduction is git plumbing that belongs in isolation.py as the - # inverse of clean_verify; lift it there when a second caller appears. sidecar = state["sidecar"] - sidecar_path = Path(str(sidecar["worktree"])) + ensured = isolation.ensure_verify_sidecar(primary, base, str(sidecar["branch"])) + sidecar_path = Path(str(ensured["worktree"])) tree = snapshot_tree(primary) - isolation.git_output(sidecar_path, "read-tree", "-u", "--reset", tree) - if isolation.git_output(sidecar_path, "write-tree") != tree: - raise DriverStop("sidecar reproduction differs from the primary task diff", task=task_id) + try: + isolation.reproduce_verify_sidecar(sidecar_path, tree) + except isolation.IsolationError as error: + raise DriverStop(str(error), task=task_id) from error execution = run_verify(command, sidecar_path) location = f"sidecar {sidecar['branch']}" isolation.clean_verify(primary, sidecar_path, base, str(sidecar["branch"])) @@ -2036,7 +2036,7 @@ def fix_tasks(primary: Path, options: argparse.Namespace) -> Dict[str, object]: texts = contracts._task_texts(primary, options.project_dir) groups = {group["locator"]: group for group in findings["groups"]} today = now()[:10] - # Same-wave file overlap is forbidden, so repairs land in a newly appended wave. + # Unordered same-wave file overlap is rejected, so repairs land in a newly appended wave. plan_path = project / "plan/PLAN.md" plan_text = plan_path.read_text(encoding="utf-8") depths, waves = contracts._plan_waves(plan_text) diff --git a/skills/gsd-path/scripts/integration.py b/skills/gsd-path/scripts/integration.py index 0dd2a356b..c1b0971d5 100644 --- a/skills/gsd-path/scripts/integration.py +++ b/skills/gsd-path/scripts/integration.py @@ -9,6 +9,7 @@ import json import re +import contextlib from pathlib import Path, PurePosixPath from typing import Optional @@ -1939,6 +1940,10 @@ def retire_member(coordinator: Path, member: str, archive_path: str, reviewed_he for kind, ref in (("push", bound_ref), ("push", "refs/heads/main"), ("push", f"refs/tags/{tag_name}"), ("delete", bound_ref)): members.clear_authorization(checkout, project, kind, ref) + # The last sidecar retires here at milestone end; isolation's retires also + # release the pin whenever the workspace is idle, so a new root can apply. + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(checkout) return {"member": member, "retired": bound} diff --git a/skills/gsd-path/scripts/isolation.py b/skills/gsd-path/scripts/isolation.py index 50a9ef684..3805f809a 100644 --- a/skills/gsd-path/scripts/isolation.py +++ b/skills/gsd-path/scripts/isolation.py @@ -15,6 +15,7 @@ import io import json import shlex +import shutil import os import re import stat @@ -171,6 +172,12 @@ def sidecar_root(primary: Path, kind: str, name: str, *, pin: bool = False) -> P raise IsolationError(str(error)) from error +def release_workspace_quietly(primary: Path) -> None: + """Best-effort placement hygiene after a sidecar retires; never blocks landing.""" + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(primary) + + def validate_task_id(task_id: str) -> str: if not TASK_ID_RE.fullmatch(task_id): raise IsolationError(f"invalid task id: {task_id}") @@ -477,6 +484,70 @@ def _same_directory(path: Path, expected: os.stat_result) -> bool: return stat.S_ISDIR(current.st_mode) and os.path.samestat(current, expected) +def _long_path(absolute: str) -> str: + if absolute.startswith("\\\\?\\"): + return absolute + if absolute.startswith("\\\\"): + return "\\\\?\\UNC\\" + absolute[2:] + return "\\\\?\\" + absolute + + +def _force_rmtree(path: Path) -> None: + """Delete a sidecar tree directly, tolerating read-only files and long paths.""" + def _writable(function, target, _error): + os.chmod(target, stat.S_IWRITE) + function(target) + + # The \\?\ prefix sidesteps Windows MAX_PATH on deep sidecar trees. + target = Path(_long_path(os.path.abspath(path))) if os.name == "nt" else path + try: + shutil.rmtree(target, onerror=_writable) + except OSError as error: + raise IsolationError(f"could not delete {path}: {error}") from error + + +def _remove_worktree(primary: Path, destination: Path, force: bool) -> None: + """Remove a sidecar worktree, tolerating populated submodules and locked trees. + + git refuses to remove worktrees whose submodules were ever initialized, and + forced removal can still fail deleting deep trees (Windows "Directory not + empty"). A dirty sidecar keeps git's refusal unless `force`, so forced + retries only run for a provably clean tree. The direct-delete fallback runs + only for git's tree-deletion failures, so any other git error leaves the + sidecar intact for diagnosis. + """ + tree_delete_failure = ("failed to delete", "directory not empty", "permission denied") + + def failure(*flags: str) -> str: + result = run_git(primary, "worktree", "remove", *flags, str(destination)) + return (result.stderr or result.stdout).strip() if result.returncode else "" + + detail = failure("--force") if force else failure() + if not detail: + return + if not force and destination.is_dir(): + status = run_git( + destination, "status", "--porcelain", "--untracked-files=all" + ) + if status.returncode != 0 or status.stdout.strip(): + raise IsolationError(detail or "git worktree remove failed") + run_git(destination, "submodule", "deinit", "--all", "--force") + detail = failure("--force") + if not detail: + return + if destination.is_dir() and any( + marker in detail.lower() for marker in tree_delete_failure + ): + _force_rmtree(destination) + detail = failure("--force") + if detail and not destination.is_dir(): + pruned = run_git(primary, "worktree", "prune") + if pruned.returncode == 0: + detail = "" + if detail or destination.is_dir() or destination.resolve() in _registered_worktrees(primary): + raise IsolationError(detail or "git worktree remove failed") + + def _cleanup_reserved_worktree( primary: Path, branch: str, @@ -762,9 +833,7 @@ def retire_member_task( if destination.resolve() in registered: if registered[destination.resolve()] != ref: raise IsolationError(f"member task sidecar is on another branch: {destination}") - removed = run_git(checkout, "worktree", "remove", *(("--force",) if force else ()), str(destination)) - if removed.returncode != 0: - raise IsolationError((removed.stderr or removed.stdout).strip() or "git worktree remove failed") + _remove_worktree(checkout, destination, force) authorization = run_git(checkout, "update-ref", "-d", member_task_authorization_ref(project, task_id)) if authorization.returncode != 0: raise IsolationError((authorization.stderr or authorization.stdout).strip() or "git update-ref failed") @@ -772,6 +841,7 @@ def retire_member_task( deleted = run_git(checkout, "update-ref", "-d", ref, tip.stdout.strip()) if deleted.returncode != 0: raise IsolationError((deleted.stderr or deleted.stdout).strip() or "git update-ref failed") + release_workspace_quietly(checkout) return copy_rejected @@ -1392,8 +1462,9 @@ def retire_member_verify(created: Dict[str, Dict[str, str]]) -> None: """Remove the member siblings of a coordinator verify sidecar; they hold no output.""" for record in reversed(list(created.values())): checkout, destination, branch = _member_verify_sidecar(record) - git_output(checkout, "worktree", "remove", "--force", str(destination)) + _remove_worktree(checkout, destination, True) git_output(checkout, "branch", "-D", branch) + release_workspace_quietly(checkout) def check_member_verify(created: Dict[str, Dict[str, str]]) -> None: @@ -1425,6 +1496,89 @@ def clean_verify(primary: Path, worktree: Path, base: str, branch: str) -> Dict[ return {"worktree": str(worktree), "branch": branch, "base": expected, "removed": removed} +def ensure_verify_sidecar(primary: Path, base: str, branch: str) -> Dict[str, object]: + """Return the named verify sidecar, recreating it when it is missing. + + Finish needs the sidecar prepare-task created. A deleted sidecar must be + rebuildable from the recorded base without a clean primary: the serial + task's own product changes are exactly the tree finish reproduces next, + so this must not reuse isolate_verify's clean-tree gate. + """ + primary = require_directory(primary, "primary worktree") + if worktree_root(primary) != primary: + raise IsolationError(f"primary is not its Git root: {primary}") + resolved_base = require_commit(primary, require_full_sha(base)) + if not branch.startswith(VERIFY_BRANCH_PREFIX): + raise IsolationError("verify sidecar reproduction requires a verify branch") + destination = _sidecar_path_for_branch(primary, branch) + ref = f"refs/heads/{branch}" + registered = _registered_worktrees(primary) + if destination.is_symlink() or (destination.exists() and not destination.is_dir()): + raise IsolationError(f"worktree path already exists: {destination}") + if destination.is_dir(): + try: + # A leftover empty directory (a half-deleted sidecar) clears; + # anything non-empty must be the sidecar itself. + destination.rmdir() + except OSError: + try: + same = ( + worktree_root(destination) == destination + and destination.resolve() in registered + and registered[destination.resolve()] == ref + and common_git_dir(destination) == common_git_dir(primary) + ) + except IsolationError: + same = False + if not same: + raise IsolationError( + f"verify sidecar ownership changed: {destination}" + ) from None + if ( + require_attached(destination) != branch + or current_sha(destination) != resolved_base + ): + raise IsolationError(f"verify sidecar HEAD changed: {destination}") + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + if destination.resolve() in registered: + cleared = run_git(primary, "worktree", "remove", str(destination)) + if cleared.returncode != 0: + run_git(primary, "worktree", "prune") + if destination.resolve() in _registered_worktrees(primary): + raise IsolationError( + (cleared.stderr or cleared.stdout).strip() + or "could not clear stale worktree registration" + ) + existing = run_git(primary, "show-ref", "--verify", "--quiet", ref) + if existing.returncode == 0: + deleted = run_git(primary, "update-ref", "-d", ref) + if deleted.returncode != 0: + raise IsolationError( + (deleted.stderr or deleted.stdout).strip() or "git update-ref failed" + ) + create_named_worktree(primary, branch, destination, resolved_base) + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + + +def reproduce_verify_sidecar(sidecar_path: Path, tree: str) -> None: + """Copy the primary's current task tree into the verify sidecar (clean_verify's inverse).""" + git_output(sidecar_path, "read-tree", "-u", "--reset", tree) + if git_output(sidecar_path, "write-tree") != tree: + raise IsolationError("sidecar reproduction differs from the primary task diff") + + def _split_paths(block: str) -> Set[str]: return {line.strip() for line in block.splitlines() if line.strip()} @@ -1537,14 +1691,24 @@ def _commit_pending( def validate_allowed_changes( - repo: Path, base: str, pending: Set[str], allowed: Set[str] + repo: Path, + base: str, + pending: Set[str], + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> None: - if not pending: + # Tolerated paths (primary bookkeeping) may stay uncommitted; the next + # `.project` checkpoint commits them. + landing = pending - tolerated + if not landing: raise IsolationError("no changes to land") - since_base = committed_paths_since(repo, base) | pending + since_base = committed_paths_since(repo, base) | landing unexpected = sorted(path for path in since_base if path not in allowed) if unexpected: - raise IsolationError("unexpected paths: " + ", ".join(unexpected)) + raise IsolationError( + "unexpected paths: " + ", ".join(unexpected) + + "; landing allows only the task file and its declared files" + ) def _clean_blob_oid(repo: Path, path: str, text: str, *, write: bool) -> str: arguments = ["hash-object"] if write: @@ -1564,21 +1728,23 @@ def commit_allowed_changes( subject: str, allowed: Set[str], body: str, + tolerated: Set[str] = frozenset(), ) -> tuple[str, str, Dict[str, Optional[tuple[str, str]]]]: require_attached(repo) pending = uncommitted_paths(repo) - validate_allowed_changes(repo, base, pending, allowed) + validate_allowed_changes(repo, base, pending, allowed, tolerated) + staging = pending - tolerated reset = run_git(repo, "reset", "-q", "HEAD") if reset.returncode != 0: raise IsolationError("could not clear the index before landing") - for path in sorted(pending): + for path in sorted(staging): added = run_git(repo, "add", "-A", "--", path) if added.returncode != 0: raise IsolationError(f"could not stage {path}") staged = _split_paths( git_output(repo, "diff", "--cached", "--name-only", "--relative") ) - if staged != pending: + if staged != staging: raise IsolationError("staged paths do not match the uncommitted change set") verified_tree = git_output(repo, "write-tree") verified_delta = _tree_delta(repo, base, verified_tree) @@ -1724,6 +1890,17 @@ def activate_task( if task_branch is None: if branch.startswith(TASK_BRANCH_PREFIX): raise IsolationError("parallel task activation requires --task-branch") + live = [ + path + for path, ref in _registered_worktrees(worktree).items() + if ref == f"refs/heads/{task_branch_name(task_id)}" + and path != worktree + ] + if live: + raise IsolationError( + "task has an active parallel worktree; serial activation in " + f"another worktree is refused (activate {live[0]} with --task-branch)" + ) elif task_branch != task_branch_name(task_id) or branch != task_branch: raise IsolationError("task activation branch does not match the task worktree") normalized_task = relative_posix(task_file) @@ -1991,7 +2168,12 @@ def _restore_rejected_commit( def _stamp_and_commit( - worktree: Path, base: str, subject: str, task_file: str, allowed: Set[str] + worktree: Path, + base: str, + subject: str, + task_file: str, + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> tuple[str, Dict[str, Optional[tuple[str, str]]]]: task_path = worktree / task_file task_text, task_mode = _read_task_text(task_path) @@ -2012,14 +2194,14 @@ def _stamp_and_commit( pending = uncommitted_paths(worktree) if stamped != task_bytes: pending.add(task_file) - validate_allowed_changes(worktree, base, pending, allowed) + validate_allowed_changes(worktree, base, pending, allowed, tolerated) original_head = current_sha(worktree) index_tree = git_output(worktree, "write-tree") try: _replace_regular_file(task_path, stamped, task_mode) - body = task_commit_body(task_file, uncommitted_paths(worktree), base) + body = task_commit_body(task_file, uncommitted_paths(worktree) - tolerated, base) commit, verified_tree, verified_delta = commit_allowed_changes( - worktree, base, subject, allowed, body + worktree, base, subject, allowed, body, tolerated ) proof_error = _landing_commit_proof_error( worktree, @@ -2097,6 +2279,7 @@ def land( subject, relative_posix(task_file), allowed, + BOOKKEEPING_PATHS, ) return { "bound_branch": bound, @@ -2391,7 +2574,7 @@ def _landing_retry_error( if task_file not in pending: return "interrupted landing does not retain the task change" try: - validate_allowed_changes(repo, base, pending, contract_paths) + validate_allowed_changes(repo, base, pending, contract_paths, BOOKKEEPING_PATHS) except IsolationError as validation_error: return str(validation_error) return None @@ -4430,6 +4613,7 @@ def retire( or f"could not delete {branch}" ) branch_retired = True + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": branch, @@ -4463,22 +4647,13 @@ def retire( raise IsolationError( "worktree is dirty; pass --force only from the retry-retirement path" ) - remove = run_git( - primary, - "worktree", - "remove", - *(("--force",) if force else ()), - str(resolved_worktree), - ) - if remove.returncode != 0: - raise IsolationError( - (remove.stderr or remove.stdout).strip() or "git worktree remove failed" - ) + _remove_worktree(primary, resolved_worktree, force) deleted = run_git(primary, "branch", "-D", retire_branch) if deleted.returncode != 0: raise IsolationError( (deleted.stderr or deleted.stdout).strip() or f"could not delete {retire_branch}" ) + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": retire_branch, diff --git a/skills/gsd-path/scripts/members.py b/skills/gsd-path/scripts/members.py index d427c6ee4..7f516c846 100644 --- a/skills/gsd-path/scripts/members.py +++ b/skills/gsd-path/scripts/members.py @@ -255,6 +255,7 @@ def check_member( coordinator_name: str, checkout: Path, recorded_remote: Optional[str] = None, + refuse_reserved_refs: bool = True, ) -> str: """Refuse a checkout that cannot join; return its origin URL.""" if checkout.is_symlink() or not checkout.is_dir(): @@ -294,13 +295,14 @@ def check_member( raise MembersError(f"member STATE.md is unreadable: {error}") from error if state.milestone is not None and (state.phase, state.status) != ("shipped", "done"): raise MembersError(f"member has an active milestone: {state.milestone}") - prefixes = member_ref_prefixes(coordinator_name) + tuple( - "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS - ) - refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() - colliding = [ref for ref in refs if ref.startswith(prefixes)] - if colliding: - raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) + if refuse_reserved_refs: + prefixes = member_ref_prefixes(coordinator_name) + tuple( + "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS + ) + refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() + colliding = [ref for ref in refs if ref.startswith(prefixes)] + if colliding: + raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) return remote @@ -620,7 +622,7 @@ def validate_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) try: role = member_role(checkout) except MembersError as error: @@ -639,7 +641,7 @@ def repair_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) _refuse_foreign_marker(checkout.resolve(), root, state.project, member["name"]) for member in members: checkout = Path(member["checkout"]).resolve() diff --git a/skills/gsd-path/scripts/worktree_paths.py b/skills/gsd-path/scripts/worktree_paths.py index 19653ca92..b2cfda32f 100644 --- a/skills/gsd-path/scripts/worktree_paths.py +++ b/skills/gsd-path/scripts/worktree_paths.py @@ -103,6 +103,33 @@ def _workspace(primary: Path, pin: bool) -> Path: return workspace +def release_workspace(primary: Path) -> bool: + """Forget the pinned placement receipt once no linked worktrees remain. + + Lets GSD_PATH_WORKTREE_ROOT (or a new default) apply again at the next + milestone's first allocation. Returns True when a receipt was removed; + any remaining linked worktree (including one only still registered) keeps + the pin. + """ + primary = primary.resolve() + common = Path(_git(primary, "rev-parse", "--git-common-dir")) + common = (primary / common).resolve() + receipt = common / "gsd-path" / "workspaces" / f"{_identity(primary)}.json" + if not receipt.is_file(): + return False + output, nul_separated = _worktree_list_porcelain(primary) + records = output.split("\0\0") if nul_separated else [ + block for block in output.strip().split("\n\n") if block.strip() + ] + for record in records: + fields = record.split("\0") if nul_separated else record.splitlines() + for field in fields: + if field.startswith("worktree ") and Path(field[len("worktree "):]).resolve() != primary: + return False + receipt.unlink() + return True + + def worktree_path(primary: Path, kind: str, name: str, *, pin: bool = False) -> Path: primary = primary.resolve() if kind not in {"task", "verify", "integrate", "bound"} or not name or Path(name).name != name or name in {".", ".."}: diff --git a/skills/gsd-path/templates/plan.md b/skills/gsd-path/templates/plan.md index a2c8ea8ec..2b49a23f4 100644 --- a/skills/gsd-path/templates/plan.md +++ b/skills/gsd-path/templates/plan.md @@ -47,8 +47,8 @@ Goal: Review depth: ## Surface contract diff --git a/skills/path/PLAN.md b/skills/path/PLAN.md index 654eacd1a..a87ea5149 100644 --- a/skills/path/PLAN.md +++ b/skills/path/PLAN.md @@ -115,9 +115,10 @@ as stated in Lookahead mode. In patch mode, tasks already `done` keep their landed metadata; only the appended patch tasks must start clean. - Require every dependency id to exist, forbid later-wave dependencies, - detect cycles, and forbid file overlap between planned tasks in the same - wave. Same-wave dependency chains are allowed only when their files do - not overlap; the build executes them in dependency layers. + detect cycles, and forbid unordered file overlap between planned tasks + in the same wave. Same-wave tasks may share files only when one + transitively depends on the other through `deps`; the build executes + them in dependency layers. - Reject a declared dependency that carries neither data nor a prerequisite effect. A `deps` edge is valid only when the dependent consumes a symbol, signature, schema, endpoint, file format, or path named in that diff --git a/skills/path/references/planner.md b/skills/path/references/planner.md index 5f4421976..e98a5c613 100644 --- a/skills/path/references/planner.md +++ b/skills/path/references/planner.md @@ -38,8 +38,9 @@ the how. security-critical; `verify-only` suits polish and low-risk feature waves whose Verify commands meaningfully cover the criteria. -Every dependency must be in an earlier wave or in the same wave with no file -overlap. Produce an acyclic graph; same-wave chains execute in layers. +Every dependency must be in an earlier wave or in the same wave. Same-wave +tasks may share files only when one transitively depends on the other. Produce +an acyclic graph; same-wave chains execute in layers. Declare a dependency only when something real crosses it: data — the dependent consumes a symbol, signature, schema, endpoint, file format, or path named in @@ -81,7 +82,8 @@ plan order above, which still keeps plan-invalidating risk in wave 1. held-out. Put each `judgment` prohibition's Must not and Detail in the owning task's Approach. - Make `files` exhaustive, including imports, routes, generated artifacts, - tests, and wiring. Require disjoint files for same-wave tasks. + tests, and wiring. Require disjoint files for same-wave tasks that are not + dependency-ordered. - Write an Interface contract in every task: `None` for independent tasks; for any tasks that exchange a symbol, signature, schema, endpoint, file format, or path, the exact shapes both sides code against — identical text diff --git a/skills/path/scripts/check_handoffs.py b/skills/path/scripts/check_handoffs.py index f86fa4306..743f7cdf9 100644 --- a/skills/path/scripts/check_handoffs.py +++ b/skills/path/scripts/check_handoffs.py @@ -1386,6 +1386,20 @@ def _task_deps(task_text: str, task_id: str) -> List[str]: return _inline_ids(rendered, "T", f"{task_id} deps") +def _dependency_reachable(graph: Dict[str, List[str]], start: str, target: str) -> bool: + seen: Set[str] = set() + pending = list(graph[start]) + while pending: + task_id = pending.pop() + if task_id in seen: + continue + if task_id == target: + return True + seen.add(task_id) + pending.extend(graph[task_id]) + return False + + def _canonical_repo_path(value: str, label: str) -> str: cleaned = value.strip().strip("`\"'") path = PurePosixPath(cleaned) @@ -1509,6 +1523,12 @@ def _validate_task_graph( and _task_scalar(tasks[right], right, "status") == "done" ): continue + # Dependency-ordered tasks serialize safely; only unordered + # same-wave overlap races on a shared file. + if _dependency_reachable(graph, left, right) or _dependency_reachable( + graph, right, left + ): + continue raise HandoffError( f"same-wave file overlap between {left} and {right}: {', '.join(overlap)}" ) diff --git a/skills/path/scripts/dispatch_driver.py b/skills/path/scripts/dispatch_driver.py index 010358d9d..c18b1e008 100644 --- a/skills/path/scripts/dispatch_driver.py +++ b/skills/path/scripts/dispatch_driver.py @@ -702,14 +702,14 @@ def finish_task(primary: Path, state: Dict[str, object]) -> Dict[str, object]: if not command: raise DriverStop("task has no Verify command", task=task_id) if state["mode"] == "serial": - # ponytail: sidecar reproduction is git plumbing that belongs in isolation.py as the - # inverse of clean_verify; lift it there when a second caller appears. sidecar = state["sidecar"] - sidecar_path = Path(str(sidecar["worktree"])) + ensured = isolation.ensure_verify_sidecar(primary, base, str(sidecar["branch"])) + sidecar_path = Path(str(ensured["worktree"])) tree = snapshot_tree(primary) - isolation.git_output(sidecar_path, "read-tree", "-u", "--reset", tree) - if isolation.git_output(sidecar_path, "write-tree") != tree: - raise DriverStop("sidecar reproduction differs from the primary task diff", task=task_id) + try: + isolation.reproduce_verify_sidecar(sidecar_path, tree) + except isolation.IsolationError as error: + raise DriverStop(str(error), task=task_id) from error execution = run_verify(command, sidecar_path) location = f"sidecar {sidecar['branch']}" isolation.clean_verify(primary, sidecar_path, base, str(sidecar["branch"])) @@ -2036,7 +2036,7 @@ def fix_tasks(primary: Path, options: argparse.Namespace) -> Dict[str, object]: texts = contracts._task_texts(primary, options.project_dir) groups = {group["locator"]: group for group in findings["groups"]} today = now()[:10] - # Same-wave file overlap is forbidden, so repairs land in a newly appended wave. + # Unordered same-wave file overlap is rejected, so repairs land in a newly appended wave. plan_path = project / "plan/PLAN.md" plan_text = plan_path.read_text(encoding="utf-8") depths, waves = contracts._plan_waves(plan_text) diff --git a/skills/path/scripts/integration.py b/skills/path/scripts/integration.py index 0dd2a356b..c1b0971d5 100644 --- a/skills/path/scripts/integration.py +++ b/skills/path/scripts/integration.py @@ -9,6 +9,7 @@ import json import re +import contextlib from pathlib import Path, PurePosixPath from typing import Optional @@ -1939,6 +1940,10 @@ def retire_member(coordinator: Path, member: str, archive_path: str, reviewed_he for kind, ref in (("push", bound_ref), ("push", "refs/heads/main"), ("push", f"refs/tags/{tag_name}"), ("delete", bound_ref)): members.clear_authorization(checkout, project, kind, ref) + # The last sidecar retires here at milestone end; isolation's retires also + # release the pin whenever the workspace is idle, so a new root can apply. + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(checkout) return {"member": member, "retired": bound} diff --git a/skills/path/scripts/isolation.py b/skills/path/scripts/isolation.py index 50a9ef684..3805f809a 100644 --- a/skills/path/scripts/isolation.py +++ b/skills/path/scripts/isolation.py @@ -15,6 +15,7 @@ import io import json import shlex +import shutil import os import re import stat @@ -171,6 +172,12 @@ def sidecar_root(primary: Path, kind: str, name: str, *, pin: bool = False) -> P raise IsolationError(str(error)) from error +def release_workspace_quietly(primary: Path) -> None: + """Best-effort placement hygiene after a sidecar retires; never blocks landing.""" + with contextlib.suppress(ValueError, OSError): + worktree_paths.release_workspace(primary) + + def validate_task_id(task_id: str) -> str: if not TASK_ID_RE.fullmatch(task_id): raise IsolationError(f"invalid task id: {task_id}") @@ -477,6 +484,70 @@ def _same_directory(path: Path, expected: os.stat_result) -> bool: return stat.S_ISDIR(current.st_mode) and os.path.samestat(current, expected) +def _long_path(absolute: str) -> str: + if absolute.startswith("\\\\?\\"): + return absolute + if absolute.startswith("\\\\"): + return "\\\\?\\UNC\\" + absolute[2:] + return "\\\\?\\" + absolute + + +def _force_rmtree(path: Path) -> None: + """Delete a sidecar tree directly, tolerating read-only files and long paths.""" + def _writable(function, target, _error): + os.chmod(target, stat.S_IWRITE) + function(target) + + # The \\?\ prefix sidesteps Windows MAX_PATH on deep sidecar trees. + target = Path(_long_path(os.path.abspath(path))) if os.name == "nt" else path + try: + shutil.rmtree(target, onerror=_writable) + except OSError as error: + raise IsolationError(f"could not delete {path}: {error}") from error + + +def _remove_worktree(primary: Path, destination: Path, force: bool) -> None: + """Remove a sidecar worktree, tolerating populated submodules and locked trees. + + git refuses to remove worktrees whose submodules were ever initialized, and + forced removal can still fail deleting deep trees (Windows "Directory not + empty"). A dirty sidecar keeps git's refusal unless `force`, so forced + retries only run for a provably clean tree. The direct-delete fallback runs + only for git's tree-deletion failures, so any other git error leaves the + sidecar intact for diagnosis. + """ + tree_delete_failure = ("failed to delete", "directory not empty", "permission denied") + + def failure(*flags: str) -> str: + result = run_git(primary, "worktree", "remove", *flags, str(destination)) + return (result.stderr or result.stdout).strip() if result.returncode else "" + + detail = failure("--force") if force else failure() + if not detail: + return + if not force and destination.is_dir(): + status = run_git( + destination, "status", "--porcelain", "--untracked-files=all" + ) + if status.returncode != 0 or status.stdout.strip(): + raise IsolationError(detail or "git worktree remove failed") + run_git(destination, "submodule", "deinit", "--all", "--force") + detail = failure("--force") + if not detail: + return + if destination.is_dir() and any( + marker in detail.lower() for marker in tree_delete_failure + ): + _force_rmtree(destination) + detail = failure("--force") + if detail and not destination.is_dir(): + pruned = run_git(primary, "worktree", "prune") + if pruned.returncode == 0: + detail = "" + if detail or destination.is_dir() or destination.resolve() in _registered_worktrees(primary): + raise IsolationError(detail or "git worktree remove failed") + + def _cleanup_reserved_worktree( primary: Path, branch: str, @@ -762,9 +833,7 @@ def retire_member_task( if destination.resolve() in registered: if registered[destination.resolve()] != ref: raise IsolationError(f"member task sidecar is on another branch: {destination}") - removed = run_git(checkout, "worktree", "remove", *(("--force",) if force else ()), str(destination)) - if removed.returncode != 0: - raise IsolationError((removed.stderr or removed.stdout).strip() or "git worktree remove failed") + _remove_worktree(checkout, destination, force) authorization = run_git(checkout, "update-ref", "-d", member_task_authorization_ref(project, task_id)) if authorization.returncode != 0: raise IsolationError((authorization.stderr or authorization.stdout).strip() or "git update-ref failed") @@ -772,6 +841,7 @@ def retire_member_task( deleted = run_git(checkout, "update-ref", "-d", ref, tip.stdout.strip()) if deleted.returncode != 0: raise IsolationError((deleted.stderr or deleted.stdout).strip() or "git update-ref failed") + release_workspace_quietly(checkout) return copy_rejected @@ -1392,8 +1462,9 @@ def retire_member_verify(created: Dict[str, Dict[str, str]]) -> None: """Remove the member siblings of a coordinator verify sidecar; they hold no output.""" for record in reversed(list(created.values())): checkout, destination, branch = _member_verify_sidecar(record) - git_output(checkout, "worktree", "remove", "--force", str(destination)) + _remove_worktree(checkout, destination, True) git_output(checkout, "branch", "-D", branch) + release_workspace_quietly(checkout) def check_member_verify(created: Dict[str, Dict[str, str]]) -> None: @@ -1425,6 +1496,89 @@ def clean_verify(primary: Path, worktree: Path, base: str, branch: str) -> Dict[ return {"worktree": str(worktree), "branch": branch, "base": expected, "removed": removed} +def ensure_verify_sidecar(primary: Path, base: str, branch: str) -> Dict[str, object]: + """Return the named verify sidecar, recreating it when it is missing. + + Finish needs the sidecar prepare-task created. A deleted sidecar must be + rebuildable from the recorded base without a clean primary: the serial + task's own product changes are exactly the tree finish reproduces next, + so this must not reuse isolate_verify's clean-tree gate. + """ + primary = require_directory(primary, "primary worktree") + if worktree_root(primary) != primary: + raise IsolationError(f"primary is not its Git root: {primary}") + resolved_base = require_commit(primary, require_full_sha(base)) + if not branch.startswith(VERIFY_BRANCH_PREFIX): + raise IsolationError("verify sidecar reproduction requires a verify branch") + destination = _sidecar_path_for_branch(primary, branch) + ref = f"refs/heads/{branch}" + registered = _registered_worktrees(primary) + if destination.is_symlink() or (destination.exists() and not destination.is_dir()): + raise IsolationError(f"worktree path already exists: {destination}") + if destination.is_dir(): + try: + # A leftover empty directory (a half-deleted sidecar) clears; + # anything non-empty must be the sidecar itself. + destination.rmdir() + except OSError: + try: + same = ( + worktree_root(destination) == destination + and destination.resolve() in registered + and registered[destination.resolve()] == ref + and common_git_dir(destination) == common_git_dir(primary) + ) + except IsolationError: + same = False + if not same: + raise IsolationError( + f"verify sidecar ownership changed: {destination}" + ) from None + if ( + require_attached(destination) != branch + or current_sha(destination) != resolved_base + ): + raise IsolationError(f"verify sidecar HEAD changed: {destination}") + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + if destination.resolve() in registered: + cleared = run_git(primary, "worktree", "remove", str(destination)) + if cleared.returncode != 0: + run_git(primary, "worktree", "prune") + if destination.resolve() in _registered_worktrees(primary): + raise IsolationError( + (cleared.stderr or cleared.stdout).strip() + or "could not clear stale worktree registration" + ) + existing = run_git(primary, "show-ref", "--verify", "--quiet", ref) + if existing.returncode == 0: + deleted = run_git(primary, "update-ref", "-d", ref) + if deleted.returncode != 0: + raise IsolationError( + (deleted.stderr or deleted.stdout).strip() or "git update-ref failed" + ) + create_named_worktree(primary, branch, destination, resolved_base) + return { + "base": resolved_base, + "branch": branch, + "kind": "verify", + "mode": "sidecar", + "worktree": str(destination), + } + + +def reproduce_verify_sidecar(sidecar_path: Path, tree: str) -> None: + """Copy the primary's current task tree into the verify sidecar (clean_verify's inverse).""" + git_output(sidecar_path, "read-tree", "-u", "--reset", tree) + if git_output(sidecar_path, "write-tree") != tree: + raise IsolationError("sidecar reproduction differs from the primary task diff") + + def _split_paths(block: str) -> Set[str]: return {line.strip() for line in block.splitlines() if line.strip()} @@ -1537,14 +1691,24 @@ def _commit_pending( def validate_allowed_changes( - repo: Path, base: str, pending: Set[str], allowed: Set[str] + repo: Path, + base: str, + pending: Set[str], + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> None: - if not pending: + # Tolerated paths (primary bookkeeping) may stay uncommitted; the next + # `.project` checkpoint commits them. + landing = pending - tolerated + if not landing: raise IsolationError("no changes to land") - since_base = committed_paths_since(repo, base) | pending + since_base = committed_paths_since(repo, base) | landing unexpected = sorted(path for path in since_base if path not in allowed) if unexpected: - raise IsolationError("unexpected paths: " + ", ".join(unexpected)) + raise IsolationError( + "unexpected paths: " + ", ".join(unexpected) + + "; landing allows only the task file and its declared files" + ) def _clean_blob_oid(repo: Path, path: str, text: str, *, write: bool) -> str: arguments = ["hash-object"] if write: @@ -1564,21 +1728,23 @@ def commit_allowed_changes( subject: str, allowed: Set[str], body: str, + tolerated: Set[str] = frozenset(), ) -> tuple[str, str, Dict[str, Optional[tuple[str, str]]]]: require_attached(repo) pending = uncommitted_paths(repo) - validate_allowed_changes(repo, base, pending, allowed) + validate_allowed_changes(repo, base, pending, allowed, tolerated) + staging = pending - tolerated reset = run_git(repo, "reset", "-q", "HEAD") if reset.returncode != 0: raise IsolationError("could not clear the index before landing") - for path in sorted(pending): + for path in sorted(staging): added = run_git(repo, "add", "-A", "--", path) if added.returncode != 0: raise IsolationError(f"could not stage {path}") staged = _split_paths( git_output(repo, "diff", "--cached", "--name-only", "--relative") ) - if staged != pending: + if staged != staging: raise IsolationError("staged paths do not match the uncommitted change set") verified_tree = git_output(repo, "write-tree") verified_delta = _tree_delta(repo, base, verified_tree) @@ -1724,6 +1890,17 @@ def activate_task( if task_branch is None: if branch.startswith(TASK_BRANCH_PREFIX): raise IsolationError("parallel task activation requires --task-branch") + live = [ + path + for path, ref in _registered_worktrees(worktree).items() + if ref == f"refs/heads/{task_branch_name(task_id)}" + and path != worktree + ] + if live: + raise IsolationError( + "task has an active parallel worktree; serial activation in " + f"another worktree is refused (activate {live[0]} with --task-branch)" + ) elif task_branch != task_branch_name(task_id) or branch != task_branch: raise IsolationError("task activation branch does not match the task worktree") normalized_task = relative_posix(task_file) @@ -1991,7 +2168,12 @@ def _restore_rejected_commit( def _stamp_and_commit( - worktree: Path, base: str, subject: str, task_file: str, allowed: Set[str] + worktree: Path, + base: str, + subject: str, + task_file: str, + allowed: Set[str], + tolerated: Set[str] = frozenset(), ) -> tuple[str, Dict[str, Optional[tuple[str, str]]]]: task_path = worktree / task_file task_text, task_mode = _read_task_text(task_path) @@ -2012,14 +2194,14 @@ def _stamp_and_commit( pending = uncommitted_paths(worktree) if stamped != task_bytes: pending.add(task_file) - validate_allowed_changes(worktree, base, pending, allowed) + validate_allowed_changes(worktree, base, pending, allowed, tolerated) original_head = current_sha(worktree) index_tree = git_output(worktree, "write-tree") try: _replace_regular_file(task_path, stamped, task_mode) - body = task_commit_body(task_file, uncommitted_paths(worktree), base) + body = task_commit_body(task_file, uncommitted_paths(worktree) - tolerated, base) commit, verified_tree, verified_delta = commit_allowed_changes( - worktree, base, subject, allowed, body + worktree, base, subject, allowed, body, tolerated ) proof_error = _landing_commit_proof_error( worktree, @@ -2097,6 +2279,7 @@ def land( subject, relative_posix(task_file), allowed, + BOOKKEEPING_PATHS, ) return { "bound_branch": bound, @@ -2391,7 +2574,7 @@ def _landing_retry_error( if task_file not in pending: return "interrupted landing does not retain the task change" try: - validate_allowed_changes(repo, base, pending, contract_paths) + validate_allowed_changes(repo, base, pending, contract_paths, BOOKKEEPING_PATHS) except IsolationError as validation_error: return str(validation_error) return None @@ -4430,6 +4613,7 @@ def retire( or f"could not delete {branch}" ) branch_retired = True + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": branch, @@ -4463,22 +4647,13 @@ def retire( raise IsolationError( "worktree is dirty; pass --force only from the retry-retirement path" ) - remove = run_git( - primary, - "worktree", - "remove", - *(("--force",) if force else ()), - str(resolved_worktree), - ) - if remove.returncode != 0: - raise IsolationError( - (remove.stderr or remove.stdout).strip() or "git worktree remove failed" - ) + _remove_worktree(primary, resolved_worktree, force) deleted = run_git(primary, "branch", "-D", retire_branch) if deleted.returncode != 0: raise IsolationError( (deleted.stderr or deleted.stdout).strip() or f"could not delete {retire_branch}" ) + release_workspace_quietly(primary) return { "bound_branch": bound, "branch": retire_branch, diff --git a/skills/path/scripts/members.py b/skills/path/scripts/members.py index d427c6ee4..7f516c846 100644 --- a/skills/path/scripts/members.py +++ b/skills/path/scripts/members.py @@ -255,6 +255,7 @@ def check_member( coordinator_name: str, checkout: Path, recorded_remote: Optional[str] = None, + refuse_reserved_refs: bool = True, ) -> str: """Refuse a checkout that cannot join; return its origin URL.""" if checkout.is_symlink() or not checkout.is_dir(): @@ -294,13 +295,14 @@ def check_member( raise MembersError(f"member STATE.md is unreadable: {error}") from error if state.milestone is not None and (state.phase, state.status) != ("shipped", "done"): raise MembersError(f"member has an active milestone: {state.milestone}") - prefixes = member_ref_prefixes(coordinator_name) + tuple( - "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS - ) - refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() - colliding = [ref for ref in refs if ref.startswith(prefixes)] - if colliding: - raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) + if refuse_reserved_refs: + prefixes = member_ref_prefixes(coordinator_name) + tuple( + "refs/remotes/origin/" + branch.format(coordinator_name) for branch in MEMBER_BRANCH_FORMATS + ) + refs = _git(checkout, "for-each-ref", "--format=%(refname)").splitlines() + colliding = [ref for ref in refs if ref.startswith(prefixes)] + if colliding: + raise MembersError("member refs collide with coordinator names: " + ", ".join(colliding)) return remote @@ -620,7 +622,7 @@ def validate_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) try: role = member_role(checkout) except MembersError as error: @@ -639,7 +641,7 @@ def repair_members(repo: Path) -> list[dict[str, str]]: members = read_members(root) for member in members: checkout = Path(member["checkout"]) - check_member(root, state.project, checkout, member["remote"]) + check_member(root, state.project, checkout, member["remote"], refuse_reserved_refs=False) _refuse_foreign_marker(checkout.resolve(), root, state.project, member["name"]) for member in members: checkout = Path(member["checkout"]).resolve() diff --git a/skills/path/scripts/worktree_paths.py b/skills/path/scripts/worktree_paths.py index 19653ca92..b2cfda32f 100644 --- a/skills/path/scripts/worktree_paths.py +++ b/skills/path/scripts/worktree_paths.py @@ -103,6 +103,33 @@ def _workspace(primary: Path, pin: bool) -> Path: return workspace +def release_workspace(primary: Path) -> bool: + """Forget the pinned placement receipt once no linked worktrees remain. + + Lets GSD_PATH_WORKTREE_ROOT (or a new default) apply again at the next + milestone's first allocation. Returns True when a receipt was removed; + any remaining linked worktree (including one only still registered) keeps + the pin. + """ + primary = primary.resolve() + common = Path(_git(primary, "rev-parse", "--git-common-dir")) + common = (primary / common).resolve() + receipt = common / "gsd-path" / "workspaces" / f"{_identity(primary)}.json" + if not receipt.is_file(): + return False + output, nul_separated = _worktree_list_porcelain(primary) + records = output.split("\0\0") if nul_separated else [ + block for block in output.strip().split("\n\n") if block.strip() + ] + for record in records: + fields = record.split("\0") if nul_separated else record.splitlines() + for field in fields: + if field.startswith("worktree ") and Path(field[len("worktree "):]).resolve() != primary: + return False + receipt.unlink() + return True + + def worktree_path(primary: Path, kind: str, name: str, *, pin: bool = False) -> Path: primary = primary.resolve() if kind not in {"task", "verify", "integrate", "bound"} or not name or Path(name).name != name or name in {".", ".."}: diff --git a/skills/path/templates/plan.md b/skills/path/templates/plan.md index a2c8ea8ec..2b49a23f4 100644 --- a/skills/path/templates/plan.md +++ b/skills/path/templates/plan.md @@ -47,8 +47,8 @@ Goal: Review depth: ## Surface contract diff --git a/tests/test_daemon_probe.py b/tests/test_daemon_probe.py index bdb5f6afe..c6a1e4ff5 100644 --- a/tests/test_daemon_probe.py +++ b/tests/test_daemon_probe.py @@ -1,5 +1,7 @@ +import hashlib import json import os +import shutil import sys import tempfile import unittest @@ -10,6 +12,8 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "daemon")) +SOURCE = Path(__file__).resolve().parents[1] + from gsd_daemon import probe from gsd_daemon.model import ProjectStatus @@ -101,6 +105,18 @@ })) """ +LAUNCHER_RUNTIME_STUB = """import json +print(json.dumps({ + "schema": "gsd-path/status/v1", + "state": {"project": "demo", "milestone": "demo-ms", "phase": "build", + "status": "active", "branch": "gsd-path/M001"}, + "route": {}, + "git": {"branch": "gsd-path/M001", "head": "def456", "dirty": False}, + "pending_answers": [{"id": "Q001"}], + "next_skill": "gsd-path-ship", +})) +""" + def make_task(project_dir: Path, task_id: str, slug: str, wave: int, status: str, title: str = None, files=("src/a.py",)) -> Path: @@ -147,6 +163,31 @@ def make_project(root: Path, state: str = STATE_FULL, with_next: bool = True) -> return root +def make_declared_runtime(root: Path, home: Path) -> None: + """Pinned-runtime install shape: shared store + declaration + real launcher. + + The store's pipeline_state is a payload stub; everything between it and + the daemon (runtime.json resolution, manifest validation, the subprocess + hop) is the real installed launcher from scripts/status_runtime.py. + """ + stub = LAUNCHER_RUNTIME_STUB.encode("utf-8") + manifest = {"version": "test", + "files": {"pipeline_state.py": hashlib.sha256(stub).hexdigest()}} + digest = hashlib.sha256( + json.dumps(manifest, sort_keys=True, separators=(",", ":")).encode("utf-8")).hexdigest() + runtime = home / ".gsd-path" / "runtimes" / digest + runtime.mkdir(parents=True) + (runtime / "pipeline_state.py").write_bytes(stub) + (runtime / "manifest.json").write_bytes( + (json.dumps(manifest, indent=2, sort_keys=True) + "\n").encode("utf-8")) + gsd = root / ".gsd-path" + gsd.mkdir(parents=True, exist_ok=True) + (gsd / "runtime.json").write_bytes( + (json.dumps({"schema": "gsd-path/runtime/v1", "version": "test", "digest": digest}, + sort_keys=True) + "\n").encode("utf-8")) + shutil.copyfile(SOURCE / "scripts" / "status_runtime.py", gsd / "status_runtime.py") + + class ParseStateTests(unittest.TestCase): def setUp(self) -> None: self.tmp = tempfile.TemporaryDirectory() @@ -458,6 +499,58 @@ def test_round_trip(self) -> None: self.assertEqual(restored.to_dict(), status.to_dict()) +class DeclaredRuntimeTests(unittest.TestCase): + """The project launcher is the probe's first hop for pinned-runtime projects.""" + + def setUp(self) -> None: + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + base = Path(self.tmp.name).resolve() + self.root = base / "demo" + self.home = base / "home" + self.home.mkdir() + make_project(self.root) + environment = mock.patch.dict(os.environ, {"HOME": str(self.home), + "USERPROFILE": str(self.home)}) + environment.start() + self.addCleanup(environment.stop) + + def test_launcher_payload_enriches_projection(self) -> None: + make_declared_runtime(self.root, self.home) + status = probe.probe_project(self.root) + self.assertEqual(status.status_source, "runtime") + self.assertEqual(status.workflow, {"state": "active", "label": "In build"}) + self.assertEqual(status.project, "demo") + self.assertEqual(status.branch, "gsd-path/M001") + self.assertEqual(status.git["head"], "def456") + self.assertEqual(status.next_skill, "gsd-path-ship") + + def test_unavailable_runtime_store_fails_soft(self) -> None: + make_declared_runtime(self.root, self.home) + shutil.rmtree(self.home / ".gsd-path" / "runtimes") + status = probe.probe_project(self.root) + self.assertEqual(status.status_source, "parse-only") + self.assertEqual(status.workflow["state"], "unverified") + self.assertEqual(status.health, "amber") + + def test_failing_launcher_does_not_run_legacy_runtime(self) -> None: + gsd = self.root / ".gsd-path" + (gsd / "runtime").mkdir(parents=True) + (gsd / "runtime" / "pipeline_state.py").write_bytes(RUNTIME_STUB.encode("utf-8")) + (gsd / "status_runtime.py").write_bytes("raise SystemExit(2)\n".encode("utf-8")) + status = probe.probe_project(self.root) + self.assertEqual(status.status_source, "parse-only") + self.assertEqual(status.workflow["state"], "unverified") + + def test_failing_launcher_without_legacy_fails_soft(self) -> None: + gsd = self.root / ".gsd-path" + gsd.mkdir(parents=True) + (gsd / "status_runtime.py").write_bytes("raise SystemExit(2)\n".encode("utf-8")) + status = probe.probe_project(self.root) + self.assertEqual(status.status_source, "parse-only") + self.assertEqual(status.workflow["state"], "unverified") + + WAVE_REVIEW = """# Review — wave 1, cycle 1