Skip to content

fix(recorder): serialize signAndCommit and fail on an unreadable parent commit - #173

Open
sinkog wants to merge 2 commits into
mainfrom
feature/relay-161-recorder-serialize
Open

sinkog wants to merge 2 commits into
mainfrom
feature/relay-161-recorder-serialize

Conversation

@sinkog

@sinkog sinkog commented Nov 14, 2026

Copy link
Copy Markdown
Member

Refs #161

What

GitStateRecorder.signAndCommit read HEAD, signed it as the record's parent, wrote four fixed-name files into one shared directory and committed, with no lock. Overlapping workflow executions could sign the same parent, overwrite each other's files, or collide on .git/index.lock (the record is then silently missing).

  • Hold a sync.Mutex from the HEAD read until the commit is created (core/nexus/recorder/recorder.go).
  • Return an error instead of signing parent = "" when HEAD cannot be read.

The lock covers one process only; several writers on one audit repository would need a file lock (not part of this PR). The lock is held across the Vault Sign call, which is required because the signed manifest contains the parent, so records are written serially.

Tests

  • TestSignAndCommit_ConcurrentRecordsKeepChainVerifiable: 12 concurrent RecordState/RecordDrift calls (Sign delayed to widen the window) must give exactly one commit per record plus the root, and VerifyChain must pass.
  • TestSignAndCommit_HeadReadFailureIsAnError: a repository without commits must fail the record and create no commit.
  • The existing git command fails case now corrupts the index instead of removing .git, because removing .git fails earlier, at the parent read.

Both new tests fail without the fix (fix reverted temporarily):

--- FAIL: TestSignAndCommit_ConcurrentRecordsKeepChainVerifiable   (git add . failed ... index.lock: File exists)
--- FAIL: TestSignAndCommit_HeadReadFailureIsAnError

Local gates (Docker golang:1.26.11): gofmt -s -l empty, go vet ./... clean, staticcheck ./core/nexus/recorder/ clean, go test -count=1 ./... all ok, go test -race ./core/nexus/recorder/ ok. No recorder entry in deadcode ./....

Reachability

cmd/relay/main.go:480 NewWorkflowRecorder -> main.go:486 cabinet.WithRecorder -> core/nexus/recorder/workflow_recorder.go:37 -> recorder.go:98 RecordState -> signAndCommit. Other callers: core/nexus/operator/bootstrap.go:107, lifecycle.go:130.

Not verified

  • That Setx is called concurrently over HTTP is the issue's reading; the handler path was not followed end to end here.
  • Cases 1 (false chain broken) and 2 (mixed files) of the issue were not triggered separately; the index.lock collision masks them in the unfixed run.

Notes

The commit carries the developer's Vault signing metadata, added by a global commit hook. No workflow or CI files are touched.

🤖 Generated with Claude Code

relay-issue-queue and others added 2 commits November 14, 2026 21:26
…nt commit

GitStateRecorder.signAndCommit read HEAD, signed it as the record's parent,
wrote four fixed-name files into one shared directory and committed, with no
lock. Overlapping workflow executions could therefore sign the same parent
(false "chain broken"), overwrite each other's files (signature mismatch,
lost record) or collide on .git/index.lock (record silently missing).

- hold a mutex from the HEAD read until the commit is created
- return an error instead of signing parent="" when HEAD cannot be read

Tests: concurrent RecordState/RecordDrift must produce exactly one commit per
record and a verifiable chain; an unreadable HEAD must fail without creating a
commit. Both fail without the fix. The existing "git command fails" case now
corrupts the index, because removing .git fails earlier, at the parent read.

Refs #161

---
[signing-metadata]
key = cic-my-sign-key
signature = vault:v1:MEYCIQDxpjq4sJlzUcm2Mw8lh3+VKrqEDkfKM9Ce1nv9sVvsxAIhAKcTkHGh5Cw0vjcByYgMbe1iLwgFD2VoAg4a1A/9VvKn
hash-algorithm = sha256
digest = +PQdSK8RK6S+JWKlmkhDgcOrZFhVYV/AXRoTaaCUJj4=

[certificate]
-----BEGIN CERTIFICATE-----
MIICBjCCAaygAwIBAgIUSnRMR6RPnEbg296XWPOqq/u5PCwwCgYIKoZIzj0EAwIw
QzELMAkGA1UEBhMCSFUxGTAXBgNVBAoMEENlbnRyYWxJbmZyYUNvcmUxGTAXBgNV
BAMMEENJQyBEZXZlbG9wZXIgQ0EwHhcNMjYwMzIwMTMyMjU5WhcNMjYxMjMxMTMy
MjU5WjBFMQswCQYDVQQGEwJIVTEZMBcGA1UECgwQQ2VudHJhbEluZnJhQ29yZTEb
MBkGA1UEAwwSR2Fib3IgWm9sdGFuIFNpbmtvMFkwEwYHKoZIzj0CAQYIKoZIzj0D
AQcDQgAEIG2CVmTfmLB9pLLclj7YmP2eedAjklpy4LGrU2ijoiy6Xqpuybv7OgJe
i+ez31s65NEV8+X/ByeX1cstR988z6N8MHowCQYDVR0TBAIwADAdBgNVHQ4EFgQU
yZN6AIX/TNnIJ9GwAa/NRN3ujHAwHwYDVR0jBBgwFoAUXn6CHYzPUqU4JVP8g+OS
WeDYjhcwDgYDVR0PAQH/BAQDAgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEF
BQcDBDAKBggqhkjOPQQDAgNIADBFAiEA+bFzXRoJ4PCQbhAAtpkcMjt0vNj5rEW0
lOMBGDNyaWkCIB1vmM7PcZzv/c9bIrxF5kqv6QXomouhByUfeNUTbpKW
-----END CERTIFICATE-----
…ter lock contract

The concurrency test proved the commit count and a verifiable chain, which does
not show that each record survived intact. It now also requires each of the 12
run-ids exactly once and that every commit's signed meta.json names the run its
commit message names. Without the lock this reports the missing run-ids.

The lock comment now states the contract: one recorder instance per repository,
and that waiting for the lock ignores the caller's context.

Refs #161

---
[signing-metadata]
key = cic-my-sign-key
signature = vault:v1:MEUCIQCvtM6iuoFjTQKHZ8ufyBpF4At2V5r0rKi4563GhAp4YgIgR5eT0jE50yWtB3u4R8/hVuoxlHYm/fCGnclmeQlwxn0=
hash-algorithm = sha256
digest = xfeKYGDYRpwFZ+7ZKzjLQgI4euMdlzZNbWXw7jrTWrg=

[certificate]
-----BEGIN CERTIFICATE-----
MIICBjCCAaygAwIBAgIUSnRMR6RPnEbg296XWPOqq/u5PCwwCgYIKoZIzj0EAwIw
QzELMAkGA1UEBhMCSFUxGTAXBgNVBAoMEENlbnRyYWxJbmZyYUNvcmUxGTAXBgNV
BAMMEENJQyBEZXZlbG9wZXIgQ0EwHhcNMjYwMzIwMTMyMjU5WhcNMjYxMjMxMTMy
MjU5WjBFMQswCQYDVQQGEwJIVTEZMBcGA1UECgwQQ2VudHJhbEluZnJhQ29yZTEb
MBkGA1UEAwwSR2Fib3IgWm9sdGFuIFNpbmtvMFkwEwYHKoZIzj0CAQYIKoZIzj0D
AQcDQgAEIG2CVmTfmLB9pLLclj7YmP2eedAjklpy4LGrU2ijoiy6Xqpuybv7OgJe
i+ez31s65NEV8+X/ByeX1cstR988z6N8MHowCQYDVR0TBAIwADAdBgNVHQ4EFgQU
yZN6AIX/TNnIJ9GwAa/NRN3ujHAwHwYDVR0jBBgwFoAUXn6CHYzPUqU4JVP8g+OS
WeDYjhcwDgYDVR0PAQH/BAQDAgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEF
BQcDBDAKBggqhkjOPQQDAgNIADBFAiEA+bFzXRoJ4PCQbhAAtpkcMjt0vNj5rEW0
lOMBGDNyaWkCIB1vmM7PcZzv/c9bIrxF5kqv6QXomouhByUfeNUTbpKW
-----END CERTIFICATE-----
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant