Skip to content

Sync main to 39d52298 (GF-17/18/19 fixes) - #373

Open
aliasfoxkde wants to merge 36 commits into
mainfrom
sync/main-39d52298
Open

aliasfoxkde wants to merge 36 commits into
mainfrom
sync/main-39d52298

Conversation

@aliasfoxkde

Copy link
Copy Markdown
Owner

Fast-forward of local main carrying the validated WIP landing, warm-pool reclaim, docker socket expansion, and pre-clone job planning fixes. Green CI run 14730f3e at this SHA on the internal GitForge.

aliasfoxkde and others added 30 commits October 30, 2026 10:56
The deployed binary on 192.168.1.202 (69245e2 bundle, Oct 29) predates
the job-level environment field; its parser rejects the key (fail-closed
per 0857218), so every push event for 7f4e8fd failed with 'unknown job
key environment'. The top-level environment: {} remains — that one the
deployed parser requires and accepts.
Clippy (rust 1.99 in dsc-ci-rust:12) flags the manual str::repeat
pattern as manual_str_repeat/manual_repeat_n under -D warnings.

Co-Authored-By: Claude <noreply@anthropic.com>
A busy branch occupied its repository's FIFO trigger lane for the
whole of each ~10-minute pipeline, leaving other refs of the same
repo starved: five main-branch trigger requests sat pending for
hours while codex-branch pushes kept re-arming the lane, and the
republish redrive could not help because it re-queued into the
same lane. Lane keys now include the ref name, so ordering holds
within a ref while cross-ref starvation is impossible.
The deployed dsc-ci-rust:12 image predates the recipe's shellcheck,
python3, g++ and cargo-fuzz entries: its lint job dies in seconds with
"shellcheck: not found" and no self-hosted run of the lane-fix commit
can go green, which blocks the release cut that would deploy it.
Rebuilt from the tip's Cargo.lock per the documented recipe and bumped
every image: reference.

Co-Authored-By: Claude <noreply@anthropic.com>
harness_passphrase was a local in spawn_api() but referenced from the
login test body — E0425, caught by CI clippy on dsc-ci-rust:13. The
spawned-binary harness now returns the runtime-generated passphrase so
the test logs in with the credential it actually seeded.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
The job container's keepalive process was `sleep 3600`, so every job
longer than one hour had its container — and any running exec — torn
down at exactly 3600s (exit 137, OOMKilled=false), independent of the
job's configured timeout. Use a 24h keepalive, matching the maximum
job-deadline clamp; containers are still removed per-job in destroy().
…runner fleet

Co-Authored-By: Claude <noreply@anthropic.com>
The evidence grader read only `receipt["status"]`, but the scheduler's
runner-completion receipt carries `success` and `outcome` with no
`status` key. Every stranded runner-completion row was therefore
graded `failed` on fail-closed, including real successes: job
98eecf71 (aegis, 2026-11-03) held `success:true, exit_code:0` in
result_json, CI logged "succeeded", and the watchdog re-graded it
`failed` five seconds later, failing run 4e58bcb2.

Two coupled defects, both fixed here:

- The grader now classifies a receipt through `JobStatus::from_receipt`,
  which reads `status`, then `outcome`, then the `success` boolean.
  Precedence is strict and per-field: the first field present decides
  the verdict, and a present-but-unusable value yields `None` rather
  than falling through to a weaker field, so bad evidence cannot be
  laundered into a different verdict. The caller still fails closed.
- The scheduler's completion receipt now records the `outcome` class it
  graded the durable row with. A receipt carrying only the runner's
  boolean could not say "the backend failed, not the commit", so a
  stranded infrastructure failure was indistinguishable from a code
  failure once the durable status write was lost.

Regression: `test_reconcile_evidence_rows_grades_stranded_rows` gains a
runner-completion row (success:true -> succeeded) and an
infrastructure-failure row (success:false + outcome -> the class
preserved), and the repaired count moves 4 -> 6.

Workspace: 949 tests pass, 0 failed. fmt and clippy -D warnings clean.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The test spawned the real api binary without GIT_ROOT, so create_repo
fell back to a CWD-relative target/gitforge-repos. That only worked
where the container CWD was writable (swarmone-docker); on
fedora-docker ensure_root() fails and POST /api/repos returns 500.

Co-Authored-By: Claude <noreply@anthropic.com>
safe.directory matches exact paths, never prefixes, so the previous
/workspace entry left every repository created under the mount rejected:
steps and test fixtures routinely git-init sub-repos there. On
fedora-docker the workspace rides an sshfs mount mapping all files to
uid 1000 while the container runs as root, so git config in a freshly
created fixture failed with "not in a git directory" and dropped 10 ci
unit tests (run f2a1a2d9, 2026-11-04). Blanket trust is safe inside a
sandbox whose only host surface is the per-run workspace mount.

Co-Authored-By: Claude <noreply@anthropic.com>
git honors the blanket "safe.directory = *" wildcard only from system or
global configuration; command-scoped GIT_CONFIG_* entries are ignored for
it, so the previous env-based attempt still let git clone refuse
sub-repositories created under the workspace mount by steps and test
fixtures. Workspace execs now export GIT_CONFIG_GLOBAL aimed at a tiny
config the exec bootstrap materializes before running the step, and the
wildcard is honored everywhere including the upload-pack children git
clone spawns. Verified against the fedora-docker sshfs reproduction of
run f2a1a2d9's 10-test failure: with the bootstrap the previously failing
prepare_run_workspace and load_pipeline_from_commit suites pass.

Co-Authored-By: Claude <noreply@anthropic.com>
The repository storage root was read from GIT_ROOT inside the request
handlers, with a CWD-relative fallback when the variable was unset.
Two consequences:

- Every request depended on process-global state that another thread
  could change underneath it.
- The fallback ("target/gitforge-repos") cannot be created in the CI
  sandbox, so create_repo returned 500 where 201 is expected.

The root is now resolved once, at the startup edge, and carried into
the request path as data via a RepoStorageRoot extension. The
handlers never read the environment.

The existing but unused ApiServer::build(storage_path) seam is now
honoured, so tests can inject a scratch root instead of mutating
process-global state.

Fixes the api integration test failure seen in CI run
c4e3a7ef-ae25-4d70-9d53-359914b41429.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The served gitforge repo's main (31de5bc) had diverged from the
lineage carrying the four GitForge CI fixes. Merge it in so the
fixes land on main rather than only on a branch.

Resolved three conflicts:

- .gitforge.yml: keep the branch's dsc-ci-rust:16 image (the tag
  the green run e95db601 used) and restore main's two comment
  blocks, which the branch's version had dropped.
- services/api/tests/api_gateway_flow.rs: take main's `passphrase`
  field name; keep the branch's GIT_ROOT env line and its
  status-poll auth test.
- services/ci/src/main.rs: keep main's event_lane_key (per-repo,
  per-ref lanes) and the branch's pipeline_id field on the push
  payload.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The merge commit shipped .gitforge.yml with unresolved conflict markers
because the file was resolved in an earlier, aborted merge attempt and
not re-resolved when the merge was re-run. The trigger request for
9741f96 failed to parse the file ("deserializing from YAML containing
more than one document is not supported"), so CI never ran on main.

The file is now rebuilt from the branch version with main's two comment
blocks restored, and validated as single-document YAML with all eight
jobs on dsc-ci-rust:16.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The cutover picked its stop/start set from `systemctl list-unit-files`,
which enumerates unit FILES, not template INSTANCES. Every unit on the
NAS profile is an instance of gitforge@.service, so all four resolved to
"not installed": the cutover would flip the release symlink, stop nothing,
start nothing, and report success while the old binaries kept serving.
That is a silent no-op release, not a failed one.

Detection is now instance-aware: LoadState carries the unit's own state
(a retired unit reports "masked"), and list-units --all reports whether
the manager actually knows the instance. A concrete unit file (the fedora
user profile) is still detected through list-unit-files. The mask check
runs before the presence check, because a masked unit is retired rather
than absent and the manager does not list it.

The same blindness made gitforge-release-auto try to restart a masked
instance; it now reads LoadState for the retired-component check.

tests/remote-cutover-unit-detection-selftest.sh drives the real script
against a stubbed systemctl that reproduces both topologies, and the CI
lint job runs it.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Every service migrates at boot, and a release cutover starts the
stateless pair (api, git-server) together, so two processes can probe
the schema at the same moment and race on the same ADD COLUMN. The
pipeline_trigger_requests migration was the one ADD COLUMN that did not
tolerate the duplicate: the loser of the race aborted its boot with
"duplicate column name: pipeline_id" instead of converging, so a
cutover that added the column could drop a service for a restart
interval.

The tolerance now lives in one helper used by every ADD COLUMN
migration in the function, so the five inline copies of the same
check are gone and the rule is stated once: a boot must never fail
because a concurrent boot applied the same change first. The
superseded dedup index is dropped by whichever process reaches it, so
the widened replacement converges whichever order they finish in.

Two tests: the helper tolerates a duplicate when a second caller
re-adds the column, and three services migrating the same file at once
all succeed and leave the widened column behind. The second test fails
on the old behaviour (duplicate column name) and passes on the new one.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The CI workspace clones AFTER the push lands, so inside the workspace
origin/main already IS the pushed tip: a merge-base range diff is empty
by construction, and the tip commit alone describes only the last commit
of a multi-commit push. A classifier that diffs against the tip can skip
clippy/test/coverage on a code change — a false green. The platform
holds the missing datum, the pushed ref's pre-push tip, but never
exported it to jobs.

CI_PUSH_BASE_SHA now carries it into every job's environment, published
through the pipeline environment map so the DAG builder merges it into
each job definition, and durable on the run row so a restarted control
plane republishes the same base — the jobs table has no env column, so
the run row is the only carrier that survives a restart. A branch
creation has an all-zero old hash: nothing is published and the
consumer fails safe.

The same change repairs the "newest-SHA-wins" supersede, which has
never fired. Its INSERT dropped ref_name, and its filter is
AND ref_name = ?, so every push run stored NULL there — 10676 of 10676
rows in the live store. The filter also matched on pipeline_id, but the
push path mints a fresh PipelineId and activates a new version row per
push, so consecutive pushes never shared one: matching on the row id
could not reach the previous push's run either. The scope is now the
pipeline's logical identity — its name and trigger type within the
repo — which is what the design documents describe.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Two pipeline runs that mount the same host directory compile into one
build tree. A rebuild in one run replaces the binaries another run is
about to execute, and that run dies with ENOENT on the missing binary
(GF-11, measured 2026-11-05 on the DIL repository: two coverage jobs
lost their test binaries to a concurrent rebuild).

The runner now takes an exclusive advisory lock on every writable host
directory a job mounts and holds it for the job's lifetime. The lock is
the directory's own inode, so there is no lock file to place, no file
for the job's container to delete, and no second path convention to keep
in step with the mount: the runner locks exactly the path it hands to
the sandbox. Only the runner creates containers, so every writer of a
shared directory passes through the lock.

The wait is charged to the job's own budget. A directory that stays busy
for the whole wait refuses the job rather than running it against a
directory another job is writing to. A directory the runner cannot open
at all is dropped from the job's mounts, so the job rebuilds cold
instead of sharing a directory it cannot protect - the same degradation
the sandbox already applies to a mount it cannot create.

The cache root moves to gitforge-common so the runner (which locks the
directory) and the sandbox (which mounts it) cannot disagree about which
host directory a cache points at.

Tests: six unit tests cover the exclusion of a second holder, the
release on drop, the independence of different directories, the
writable-bind-mount filter, the refusal on contention, and the
degradation of an unopenable mount.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The CI engine re-enqueued a job that was already running, and the
scheduler's enqueue path demoted the running job's durable row to
`queued` with an unguarded UPDATE. That demotion broke every
lease-gated write at once: the runner's heartbeat was refused (409),
the row became dispatchable again, the scheduler dropped the mirror's
lease and then minted a fresh token, and the runner's completions were
rejected (`completion_persistence_failed`). The job was finally fenced
as `runner_lost_while_running` after three executions.

Measured on the live GitForge CI service, 2026-11-05: job 80e812aa of
run 3eb28707 - the first heartbeat was refused 18 seconds after the
job started, and three completions were rejected with the same cause.

The row's live status is the invariant every later write is fenced by,
so the fix is at the write:

- `JobQueries::mark_queued_if_dispatchable` moves a row to `queued` only
  while it is `pending`/`queued`, and the enqueue path uses it. A live
  row is left to the runner that holds it, and the job is not queued
  again.
- The enqueue of a job this scheduler already dispatched is a no-op, so
  the duplicate never reaches the row.
- The CI service consumes the scheduler's `JobAssigned` event and
  advances the engine's mirror out of `Queued`. `ready_jobs()` releases
  every job whose mirror still says `Queued`, so the dispatched job was
  re-released by every later completion of the run - that was the
  source of the duplicate. The event carries the run id for the lookup.
- A refused assignment no longer drops the mirror's lease while the job
  is still mirrored as assigned, and a lease the mirror lost is adopted
  from the durable row instead of being minted anew, so the token under
  the executing runner never rotates.
- The engine's assignment replay is idempotent, and a job that fails
  while it is assigned is a failure rather than an invalid transition.

Regressions (all three FAIL against the pre-fix body and pass after):
`a_duplicate_enqueue_does_not_break_a_running_jobs_lease` (the row
stays `running`, the lease does not rotate, the heartbeat and the
completion are accepted),
`the_lease_handed_to_a_runner_is_the_durable_token`, and
`a_dispatched_job_is_not_released_again`.

Co-Authored-By: Claude Code <noreply@anthropic.com>
GF-09: cache_root() resolved /var/cache/gitforge while docs/RUNBOOK.md
documents the default as ~/.cache/gitforge. The runner unit cannot
write /var/cache (ProtectSystem=strict, and no ReadWritePaths entry),
so every declared cache mount was dropped and the platform's own
pipeline cold-built on every run.

The decision is now a pure function of the environment it is given
(cache_root_with); cache_root() reads the environment once at the edge
and passes it down. The override still wins, the home directory's
.cache/gitforge applies next, and the system path is the fallback when
$HOME is unset.

Co-Authored-By: Claude Code <noreply@anthropic.com>
GF-10: build_volume_mounts failed the whole mount set on the first
unusable source, and both call sites swallowed that into a warning, so
the job ran with a contract the pipeline never declared and the reason
reached only the log. The runner's mount lock dropped an un-lockable
mount the same way.

The sandbox now fails the job, and the runner refuses a shared mount
it cannot lock instead of dropping it. Both reasons already travel to
the job's own result as an infrastructure failure, so the result schema
does not change: the missing mount is now visible where the job is,
rather than only in a runner log.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Measured 2026-11-05 on the NAS instance: eight `gitforce-job-*` containers
were still running, hours after the jobs that owned them had succeeded.
The runner journal showed the cause on every one of them - "timed out
cleaning up sandbox for job <id>" - and the release drain gate counts
live job containers, so the leak blocked every cutover for four hours.

Root cause: DockerSandbox::destroy() chowns the whole workspace tree
before it stops and removes the container. That walk is recursive over
whatever the job checked out - the DIL tree carries a vendored crate
directory with 10,730 files, on a network filesystem - so it can take
minutes. The executor wrapped pool.release() in a 30-second timeout, so
the timeout cancelled the teardown before the container was ever
stopped. The container then ran forever.

The background reconciler cannot cover this: its policy never removes a
RUNNING container, and a leaked job container is running by definition.

The fix is in the tool, in two parts:

  - The workspace ownership cleanup is a convenience, not a precondition
    of the teardown, so it now runs under its own 15-second budget
    (bounded_cleanup) and the container is always stopped and removed
    afterwards. The CI service already removes a workspace it cannot
    delete with a container-assisted cleanup, so bounding the chown
    creates no new leak class.
  - If the teardown still overruns its budget, the executor removes the
    container by its job label, forcefully (remove_job_containers), so a
    leaked container cannot outlive the attempt.

Evidence: cargo fmt, cargo clippy --all-targets -D warnings, 132 runner
tests and 98 sandbox tests green on Fedora; the real-Docker regression
(remove_job_containers removes only that job's containers) passes there
too.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The exclusive resource class was decided by scanning each job command's
words for `cargo`, `test` and `--workspace`, independently. A command
that merely MENTIONED those words landed in the class: the DIL coverage
job's comment about "the test binary" put a `cargo llvm-cov --workspace`
job into `workspace-cargo-test`, and the platform's own `cargo test
--workspace` job then excluded it from the only runner - which had a
free slot.

The class now reads the cargo SUBCOMMAND an invocation actually runs and
requires `--workspace` (or its `--all` alias) among that invocation's
arguments. Comments, echoed strings and other cargo subcommands are not
the class. The scan is lexical and conservative: it splits the shell
script into segments, strips `#` comments, honours quotes, and steps
over cargo's global options and a `+toolchain` override to reach the
subcommand.

The second half of the defect was silence. A job the class excluded was
skipped with a debug log and no durable reason, so `jobs.deferred_reason`
stayed NULL and the block was invisible in the job's own evidence. The
dispatch loop now distinguishes a runner whose free slot the class took
away from a runner that is simply busy, and records the new durable
reason `resource_class_conflict` for the former.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The subcommand-based classifier only looked at the first program
word of a shell segment, so a wrapper in front of the invocation
hid it: `flock <lockfile> cargo test --workspace` was read as a
non-workspace test. The old word scan caught that shape.

That is a false negative in the hazardous direction: two workspace
cargo tests building the same target tree on one runner is the
failure that killed the coverage jobs with ENOENT on the test
binary. A false positive only costs a deferral, and the deferral
now carries a durable reason.

Scan every word position as a possible cargo invocation. Measured
over the 473 distinct command strings in the live job table, the
new rule classifies 39 and the old word scan 45; the six
differences are all per-package `cargo test -p <crate>` scripts
and `cargo llvm-cov` commands, and no command is newly classified.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The release chain stamped a source_commit onto a bundle without ever
checking that the binaries it shipped were built from that commit. The
release binaries live in gitignored paths at the root of the source
root, so a bundle assembled from stale leftovers passed every gate:
present, executable, checksummed from whatever was copied. The promoted
GF-14 bundle shipped pre-fix binaries under a source_commit that named
the fix.

The claim now travels with the binaries:

- crates/gitforge-common/build.rs stamps the build commit, and whether
  the tree was clean, into every binary; the four services report it
  through --version;
- scripts/gitforge-release-stamps refuses binaries that do not report
  the expected commit;
- scripts/gitforge-release-preflight and -verify run that check, and
  gitforge-release-bundle runs it on the staged tree before the bundle
  is published, so a refused cut leaves nothing behind;
- scripts/gitforge-release-build builds and installs the release
  binaries and refuses to install one built from another commit.

tests/test_release_bundle.py pins the regression: a bundle of binaries
from another commit is refused by the bundler and by the verifier.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The bundle's RELEASE_METADATA.txt recorded the toolchain of whichever
host ran the cut and the time of the cut, so a bundle of Fedora-built
binaries claimed `toolchain=rustc 1.99.0 (b940084d7 2026-09-28)` and
`build_timestamp_utc=<the cut time>` while its own binaries were
compiled by rustc 1.99.1 (58cf8f9c3 2026-10-13). The metadata
described the machine that assembled the release, not the artifact the
release ships.

Stamp the HOW into the binary itself: the build script records the
toolchain and the build epoch, and the version line carries them as
`toolchain=` and `built=`. The bundler then reads those values back
out of the four staged binaries and writes them into the metadata, and
the verifier re-derives them from the shipped binaries and refuses a
bundle whose metadata names a different toolchain or a different build
time. Binaries that disagree with each other, or that cannot say how
they were built, are refused.

The cut time is kept as a separate `bundle_cut_utc=` field: it is a
fact about the cut, and it is no longer allowed to masquerade as the
build time.

GF-15.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The release channel's busy gate counted the runner's `gitforce-job-*`
containers. The runner keeps a warm pool of containers that have no job
row at all, so the count was permanently non-zero: every cutover was
deferred forever and the channel could never promote anything. The
condition was reported as an ordinary "deferred", which reads like
normal operation.

The busy signal now comes from the authoritative store through the new
scripts/gitforge-release-busy: a job is in flight when its row is
`assigned` or `running` AND its proof of life is still inside the fence
grace window - the same liveness term the scheduler's requeue_inflight
uses, so a row left behind by a dead runner cannot wedge the gate
either. Queued and pending rows are queued work and do not block. An
unreadable store fails closed: the pass refuses the cutover instead of
starting on an unknown busy state.

The sibling operator cutover (scripts/gitforge-remote-cutover) drains on
the same reader, so both agree on what "in flight" means, a stale row
no longer holds that wait for ever, and the sqlite3 CLI is no longer
required on the target host.

This also carries the release-auto restart plan mode and the
masked-unit detection fix (LoadState, which sees template instances)
out of the host checkout, so the deployed copy and this branch are one
source of truth.

Regression: tests/release-auto-busy-gate-selftest.sh, wired into the
lint job, pins the store-backed count, the stale-row case, the
fail-closed behaviour, the absence of the container filter, and the
cutover drain. tests/release-auto-restart-plan-selftest.sh pins the
masked-unit exclusion.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Merge the operator's working-tree changes from the deployed host checkout
onto current main. The host tree was stale relative to main (it lacked the
push-base work, the cutover and busy-gate selftests, the GF-10 cache-mount
comment fix and the clippy-TESTS debt note), so this is a three-way merge,
not a copy.

Kept from the WIP:
  - .gitforge.yml: a docker-integration job (dsc-ci-rust:16, needs: [test])
    that runs the #[ignore]-gated gitforge-sandbox/gitforge-runner lifecycle
    tests no gate had ever run; coverage now needs it. The job mounts the
    docker socket read-only via the {docker_socket} placeholder.
  - crates/gitforge-ci/src/pipeline.rs: a contract test asserting the
    committed pipeline keeps that socket mount.
  - .githooks/pre-push: advisory clean-branch check against @{u}/HEAD, and
    the fi->done fix in pre-push.d/20-clean-branch.
  - .github/workflows/*: dtolnay/rust-toolchain SHA bump.
  - docs/audits/OPENCODIFIER_QUALITY_2026-11-03.md (new),
    docs/planning/EXECUTION_PLAN_20261103.md.

Removed from the WIP:
  - .githooks/pre-push.d/10-complexity-gate and 30-coverage-gate: already
    unreferenced by pre-push, so dead code.
  - docs/planning/CI_STORAGE_EFFICIENCY_ARCHITECTURE_2026-10-31.md: zero
    references repo-wide.

Not carried over: the host's untracked cache/ tree (13G of build artifacts).

Co-Authored-By: Claude Code <noreply@anthropic.com>
aliasfoxkde and others added 6 commits November 6, 2026 06:54
The runner keeps a warm pool of spare containers so a job does not pay
container creation latency. The pool lives only in memory, so when the
runner process dies or restarts its containers keep running with no
owner: nothing in the tree can reach them. `remove_job_containers`
scopes by exact job-id label equality, and the general reconciler
retains every running container, so a pooled container is invisible to
both cleanup paths. They accumulate until the host runs out of
containers.

Give the pool an ownership contract:

- Pooled containers are stamped with owner, generation and creating-PID
  labels at creation. The owner separates runners on one host, the
  generation separates processes of the same runner, and the PID tells
  a dead creator from a live one.
- A pure decision function decides reclamation. Every ownership fact
  must be established: an unknown owner, an unknown age or another
  owner's container is retained, never removed.
- A reaper in the runner process reclaims the orphans a previous
  process left behind, bounded per pass and idempotent.
- The general reconciler keeps its hands off pooled containers
  (`RetainPool`), so the two mechanisms cannot disagree about a live
  container.
- The pool drains itself when the runner cancels its jobs, expires a
  spare that outlived its TTL instead of handing it out, and the TTL
  has a single definition shared by the pool and the reaper's policy.

The leak is fixed in the tool rather than worked around by a cleanup
procedure: the containers now carry the ownership facts a later
process needs to find them.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The pool module's rustdoc named [`POOL_SIZE`], a link that no longer
resolves, so `cargo doc` failed under the clippy job's
`RUSTDOCFLAGS="-D warnings"` gate. The pool holds a small number of
spare containers per image; the prose now says that instead of naming
a constant that is not there.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The docker-integration job declares `source: "{docker_socket}"`, and
the platform documents that placeholder as the daemon socket of the
daemon the runner drives. No code expanded it: VolumeSpec expanded
only {image} and {cache_root}, so the literal reached the daemon as a
bind source and the job died in the sandbox with "volume mount source
must be an absolute path: {docker_socket}" (CI run f2292b0e).

The placeholder is now expanded against one shared definition of the
daemon socket, so the sandbox, the reconciler and the mount all name
the same daemon:

- gitforge_common::docker_socket() resolves DOCKER_HOST when it names
  a unix socket, and falls back to /var/run/docker.sock.
- The sandbox and the reconciler connect to that socket instead of
  bollard's own default, so a reconciler cannot drift onto a daemon the
  sandbox does not supervise.
- A socket source is not a directory: the bind is not relabeled (`:z`
  would rewrite the label of the host's own socket file) and is not
  created, and the container runs with SELinux labeling disabled
  instead. Measured on the Fedora runner: without the opt the
  connection to the daemon socket is refused.
- The daemon socket is excluded from the mount lock: it is shared by
  every job on the runner and is not a lockable directory.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The warm pool's pre-pool leftovers are unreachable by every existing
reclamation rule: no pool label, still running, and a job id the
authoritative store has never heard of. Rule 4 misses them for want of
the label, rule 5 retains them for being running, and the pool reaper
reports them as not-pool. They therefore survive forever.

Give the reconciler a store-backed rule that reaches them. The store's
answer is injected as data (JobRowState), never read inside the
decision, and the rule is armed explicitly and fails closed at every
unknown: unarmed policy, unreadable store, unparsable id, unknown age,
a container inside the orphan TTL, and a missing resolver all retain.

The scheduler answers per id on POST /jobs/existence, in the runner's
credential scope so the runner's existing bearer authorizes it. A
durable job row is written at enqueue time, before dispatch, so "no
row" means "never was a job" - except for repository deletion, which
the orphan TTL bounds.

Co-Authored-By: Claude Code <noreply@anthropic.com>
gitforge-common declares serde_json (job_row.rs) and gitforge-cli has
used it since dc3deb0, but Cargo.lock omitted the edges, so every
cargo build rewrote the lock and dirtied release builds.

Co-Authored-By: Claude <noreply@anthropic.com>
A slow workspace clone (GF-19: the NAS spinner) stranded a run as
`running` with zero job rows because planning ran only after the
clone returned. persist_planned_jobs now runs before the clone and
skips already-persisted rows on the second pass; the dispatch-time
upsert corrects working_dir, so the pre-clone write is safe.
@coderabbitai

coderabbitai Bot commented Nov 6, 2026

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 65d0ac33-de4e-4f5e-a960-6a330b4230a8
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@aliasfoxkde
aliasfoxkde enabled auto-merge (rebase) November 6, 2026 22:17
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

🤖 AI Code Review Results

AI review skipped: provider 'anthropic' is not configured; deterministic gates remain authoritative.

✅ No critical issues found.

AI Code Review powered by GitForge

State(state): State<SchedulerServerState>,
Json(request): Json<JobExistenceRequest>,
) -> impl IntoResponse {
let mut parsed: Vec<JobId> = Vec::with_capacity(request.job_ids.len());

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants