daemon reload is partial: new project polls but cannot execute (sandbox project map frozen at bootstrap)

[wrenrichley]

Summary

foreman daemon reload refreshes the project list for polling but not for role execution. With the sandbox enabled, a newly-registered project is picked up, dispatched, and fails three times into NeedsHelp.

Reproduced end-to-end registering jeffrichley/bookwright on 2026-08-05:

Ticket 42767 (bookwright#103) — NeedsHelp
├── Queued #1 → clean → Planning
├── Planning #2 → failed @ execute: RoleSubprocessError("role=planner: project 'bookwright'
│                 absent from the sandbox project map; cannot prep its private clone")
├── Planning #3 → failed @ execute: (identical)
├── Planning #4 → failed @ retry_cap: state 'Planning' failed 3 consecutive times (cap=3)
└── NeedsHelp #5

Both halves are observable, which is what makes the diagnosis certain. The poller saw the project — the ticket reached Queued roughly 15 seconds after the foreman:plan label was applied. The dispatcher did not.

Cause

sandbox_projects is passed into SubprocessRoleDispatcher at construction:

# v4/bootstrap.py:258
dispatcher = SubprocessRoleDispatcher(
    ...
    sandbox_projects=sandbox_projects,
)

# v4/subprocess_dispatcher.py:369
self._sandbox_projects = sandbox_projects

and consumed per dispatch:

# v4/subprocess_dispatcher.py:467
project_cfg = self._sandbox_projects.get(project)
if project_cfg is None:
    raise RoleSubprocessError(f"role={role}: project {project!r} absent from the "
                              f"sandbox project map; cannot prep its private clone")

The SIGHUP handler does not rebuild the dispatcher, so this map is frozen at daemon start.

The documentation says otherwise

~/.foreman/projects.toml opens with:

# Foreman managed projects (host-mounted, hot-reloadable per issue #477).
# Edit and run `foreman daemon reload` to add/rename/remove a repo.

That was true when #477 landed. Sandboxing (#557/#556, enabled 2026-07-20) added a second consumer of the project list that hot-reload does not reach, and the instruction was never amended. An operator following the documented procedure gets a project that polls but cannot execute.

Note the failure is also expensive rather than immediate: it burns the full retry cap (3 attempts) before escalating, so the first symptom is a NeedsHelp ticket rather than a clear "reload didn't take" signal.

Suggested fix

Either:

1. Rebuild the sandbox project map on SIGHUP — matches the documented behaviour and is what an operator expects from "hot-reloadable."
2. Make reload refuse to half-apply — if the sandbox map cannot be refreshed, say so on the reload path (reload applied to poller only; restart required for sandbox) rather than succeeding silently and failing three dispatches later.

⇒ If neither is done, the projects.toml comment and any onboarding docs must stop promising reload is sufficient, because today they promise a capability that is half-present.

Workaround

Restart the daemon after adding a project. Wait for an idle board first — a restart mid-flight fails in-flight tickets.

Related: #585 (foreman init validates for .git, broken by the bare-mirror base) and #586 (roles get no project instructions, same mirror change). All three surfaced in one onboarding attempt. The common thread is that project onboarding is exercised rarely enough that changes elsewhere break it without anyone noticing.