Repository navigation
feat: connect S3 task archives with recoverable server lifecycle - #318
Conversation
|
Updated in 223ccf6 and synchronized with upstream 65045ef in 5f192ad.
The tested integration c0bbed6 combines #285/#318/#319 on upstream 65045ef and passed:
Standalone CI at 5f192ad failed only the Go jobs because the scoped branch cannot resolve This remains a dependent draft: standalone Go CI cannot resolve archivestore until #285 lands. The integrated result above is separate evidence; after the dependency merges, main must be incorporated and standalone CI rerun. |
5f192ad to
f69eb06
Compare
|
Synchronized this branch with upstream 705ac80 in 044fbdf, including the now-merged #285 backend. The server branch builds independently; the earlier missing-archivestore dependency is resolved.
There are currently no review comments on this PR requiring a code change. The backend's earlier review blockers were approved in #285. This PR is ready for review; UI/operator documentation remain in the separate dependent draft #319. |
|
Synced with upstream 9465dfa and fixed the startup integration in 64a53ea. Upstream now scrubs the archive-specific S3 credentials from the process environment. This branch previously constructed the archive client later, when starting the worker, so the dedicated credentials were already gone: authentication could fail or use an unintended ambient AWS identity. The client now captures its configuration before the scrub, while the secrets are still removed before subsequent child processes start. Added TestArchiveStoreLoadsCredentialsBeforeSecretScrub. It exercises New with WithSecretScrub, supplies different ambient credentials, verifies the actual signed S3 request uses the dedicated key/session token, and checks those variables are absent afterward. It failed against the prior startup order and passes with the fix. Also adapted the interruption-recovery test to upstream's new restore execution-identity parameter, preserving the new workspace/spill ownership handling.
#318 remains ready for review. #319 remains the separate UI/docs follow-up. |
The archive store falls back to the AWS SDK default chain when no ARTEX_ARCHIVE_S3_* override is set. Those AWS_* variables stayed in the environment after startup, so agent shells, custom tools and MCP stdio servers inherited them. Add them to secretEnv: they are scrubbed after the store is built and kept out of /proc/<pid>/environ by the seal. The SDK captures what it needs at config load. Tests cover static keys (both spellings), container credentials and web identity fetched after the scrub, and a real New with WithSecretScrub.
The archive worker goroutine had no recover. An S3-compatible endpoint that answers CreateMultipartUpload with 200 and no UploadId makes the SDK upload manager dereference a nil pointer, which ended the server. Run each claimed job under a recover: log the panic value and stack and return it as the job error, so the job is failed through the same FailTaskArchiveJob path as any other error. The job's deferred rollbacks have run by then, so the source data stays in place. FailTaskArchiveJob now only fails a job that is still in its active state. A panic recovered after the archive committed would otherwise turn a ready archive into archive_failed, which can be neither restored nor deleted.
writeRemoteArchiveJournal ended with Sync on two directory handles. Windows cannot flush a directory handle, so the journal write failed there and no remote archive or remote deletion could start. Move the directory sync to syncDir: the unix build keeps the fsync, the other builds return nil (the journal file itself is already synced).
The SDK reads AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE while LoadDefaultConfig builds the archive store and captures the path; the file itself is read lazily. Adding the name to secretEnv keeps that path out of agent child processes, the same way AWS_WEB_IDENTITY_TOKEN_FILE is already scrubbed, and does not affect the client.
Problem and behavior
Local task archives continue consuming server disk, including large APK-analysis workspaces. This connects the encrypted S3 backend to Archive, Restore and Delete while retaining local source data until verification and the database commit succeed.
The backend from #285 is merged and is included in this branch through upstream 9465dfa. This PR contains only server/database integration and recovery tests. UI and operator documentation remain in the dependent draft #319.
Validation
go test -race -p 1 ./server ./db -run 'TestArchive|TestRemoteArchive|TestTaskArchive|TestNewAppliesSecretScrub|TestScrubSecretEnv' -count=1passed.go test -p 1 ./server ./db -count=1passed.go build -p 1 ./...,go vet -p 1 ./...andgo mod tidy -diffpassed.Archive coverage includes 14 persisted interruption boundaries with repeated recovery, local/S3 committed and uncommitted states, partial-write crashes, rollback conflicts, cleanup failures, unknown-storage JSON, corrupt content, already-restored retries and protection of a successful local retry.
Tests use temporary PostgreSQL and HTTP S3 fixtures. No live-cloud compatibility claim is made.
Operational limits
Archive/restore still require temporary local disk, and full readback incurs object-store reads and transfer charges. Preserve database metadata, bucket objects, access and old KMS keys for restores. Bucket lifecycle/version-retention policies remain necessary for abrupt process death and failed version cleanup.