Skip to content

feat: connect S3 task archives with recoverable server lifecycle - #318

Merged
BaaRai merged 10 commits into
BaaRai:mainfrom
renaesop:codex/s3-archive-server
Nov 20, 2026
Merged

BaaRai merged 10 commits into
BaaRai:mainfrom
renaesop:codex/s3-archive-server

Conversation

@renaesop

@renaesop renaesop commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem and behavior

Local task archives continue consuming server disk, including large APK-analysis workspaces. This connects the encrypted S3 backend to Archive, Restore and Delete while retaining local source data until verification and the database commit succeed.

The backend from #285 is merged and is included in this branch through upstream 9465dfa. This PR contains only server/database integration and recovery tests. UI and operator documentation remain in the dependent draft #319.

  • Capture the archive credentials before startup environment scrubbing so the configured S3 identity remains available after the secrets are removed from child-process environments.
  • Confirm encryption, size and full SHA-256 before committing a remote location; remove local packages and staging after commit.
  • Verify downloads before restoring. Recognize committed restores so cleanup retries do not replay database rows.
  • Persist upload/delete cleanup journals, retaining committed objects and packages referenced by a later local-storage retry.
  • Recover staging using the committed location/checksum even when Restore or Delete is queued.
  • Roll source files back before removing interrupted compressed packages or .partial files. Preserve the staging journal on rollback/cleanup failure so recovery can retry.
  • Omit storage when no stored location exists; derive local/s3 for existing locations without a schema or archive-format migration.

Validation

  • CI at 64a53ea passed all jobs: Go (including the full race suite), stable Go, web, Python and shell.
  • The new startup credential regression failed before the fix and passed afterward.
  • With isolated temporary PostgreSQL, go test -race -p 1 ./server ./db -run 'TestArchive|TestRemoteArchive|TestTaskArchive|TestNewAppliesSecretScrub|TestScrubSecretEnv' -count=1 passed.
  • Full go test -p 1 ./server ./db -count=1 passed.
  • go build -p 1 ./..., go vet -p 1 ./... and go mod tidy -diff passed.
  • The temporary PostgreSQL instance was stopped after validation.

Archive coverage includes 14 persisted interruption boundaries with repeated recovery, local/S3 committed and uncommitted states, partial-write crashes, rollback conflicts, cleanup failures, unknown-storage JSON, corrupt content, already-restored retries and protection of a successful local retry.

Tests use temporary PostgreSQL and HTTP S3 fixtures. No live-cloud compatibility claim is made.

Operational limits

Archive/restore still require temporary local disk, and full readback incurs object-store reads and transfer charges. Preserve database metadata, bucket objects, access and old KMS keys for restores. Bucket lifecycle/version-retention policies remain necessary for abrupt process death and failed version cleanup.

@renaesop

Copy link
Copy Markdown
Contributor Author

Updated in 223ccf6 and synchronized with upstream 65045ef in 5f192ad.

  • Startup recovery now removes interrupted .partial compression output only after restoring staged source files.
  • If rollback or package cleanup fails, its journal survives for a safe retry. New PostgreSQL regressions cover a mid-compression crash, rollback conflict, cleanup failure and repeated recovery.
  • Archives without a stored location omit storage from JSON instead of reporting local. Existing local and S3 locations retain their correct labels; the database/lifecycle tests cover these cases.

The tested integration c0bbed6 combines #285/#318/#319 on upstream 65045ef and passed:

  • go test -race -p 1 ./archivestore ./server ./db -run 'TestArchive|TestS3|TestRemoteArchive|TestTaskArchive' -count=1
  • go test -p 1 ./server ./db -count=1 (full suites)
  • go build -p 1 ./..., go vet -p 1 ./..., go mod tidy -diff
  • Frontend tsc --noEmit --incremental false

Standalone CI at 5f192ad failed only the Go jobs because the scoped branch cannot resolve github.com/Autumn-27/artex/archivestore before #285 merges. Web/Python/shell passed. This dependency failure is kept explicit.

This remains a dependent draft: standalone Go CI cannot resolve archivestore until #285 lands. The integrated result above is separate evidence; after the dependency merges, main must be incorporated and standalone CI rerun.

@renaesop
renaesop force-pushed the codex/s3-archive-server branch from 5f192ad to f69eb06 Compare November 20, 2026 02:09
@renaesop
renaesop marked this pull request as ready for review November 20, 2026 02:33
@renaesop

Copy link
Copy Markdown
Contributor Author

Synchronized this branch with upstream 705ac80 in 044fbdf, including the now-merged #285 backend. The server branch builds independently; the earlier missing-archivestore dependency is resolved.

  • CI at 044fbdf passed all jobs: Go (including the full race suite), stable Go, web, Python and shell.
  • On shein-artex, using a private temporary PostgreSQL instance, go test -race -p 1 ./server ./db -run 'TestArchive|TestRemoteArchive|TestTaskArchive' -count=1 passed.
  • Full go test -p 1 ./server ./db -count=1 passed.
  • go build -p 1 ./..., go vet -p 1 ./... and go mod tidy -diff passed.
  • The temporary PostgreSQL instance was shut down after validation.

There are currently no review comments on this PR requiring a code change. The backend's earlier review blockers were approved in #285. This PR is ready for review; UI/operator documentation remain in the separate dependent draft #319.

@renaesop

Copy link
Copy Markdown
Contributor Author

Synced with upstream 9465dfa and fixed the startup integration in 64a53ea.

Upstream now scrubs the archive-specific S3 credentials from the process environment. This branch previously constructed the archive client later, when starting the worker, so the dedicated credentials were already gone: authentication could fail or use an unintended ambient AWS identity. The client now captures its configuration before the scrub, while the secrets are still removed before subsequent child processes start.

Added TestArchiveStoreLoadsCredentialsBeforeSecretScrub. It exercises New with WithSecretScrub, supplies different ambient credentials, verifies the actual signed S3 request uses the dedicated key/session token, and checks those variables are absent afterward. It failed against the prior startup order and passes with the fix.

Also adapted the interruption-recovery test to upstream's new restore execution-identity parameter, preserving the new workspace/spill ownership handling.

  • CI at 64a53ea passed all jobs: Go (including the full race suite), stable Go, web, Python and shell.
  • The new startup credential regression failed before the fix and passed afterward.
  • With isolated temporary PostgreSQL, go test -race -p 1 ./server ./db -run 'TestArchive|TestRemoteArchive|TestTaskArchive|TestNewAppliesSecretScrub|TestScrubSecretEnv' -count=1 passed.
  • Full go test -p 1 ./server ./db -count=1 passed.
  • go build -p 1 ./..., go vet -p 1 ./... and go mod tidy -diff passed.
  • The temporary PostgreSQL instance was stopped after validation.

#318 remains ready for review. #319 remains the separate UI/docs follow-up.

The archive store falls back to the AWS SDK default chain when no
ARTEX_ARCHIVE_S3_* override is set. Those AWS_* variables stayed in the
environment after startup, so agent shells, custom tools and MCP stdio
servers inherited them. Add them to secretEnv: they are scrubbed after
the store is built and kept out of /proc/<pid>/environ by the seal.

The SDK captures what it needs at config load. Tests cover static keys
(both spellings), container credentials and web identity fetched after
the scrub, and a real New with WithSecretScrub.
The archive worker goroutine had no recover. An S3-compatible endpoint
that answers CreateMultipartUpload with 200 and no UploadId makes the
SDK upload manager dereference a nil pointer, which ended the server.

Run each claimed job under a recover: log the panic value and stack and
return it as the job error, so the job is failed through the same
FailTaskArchiveJob path as any other error. The job's deferred rollbacks
have run by then, so the source data stays in place.

FailTaskArchiveJob now only fails a job that is still in its active
state. A panic recovered after the archive committed would otherwise
turn a ready archive into archive_failed, which can be neither restored
nor deleted.
writeRemoteArchiveJournal ended with Sync on two directory handles.
Windows cannot flush a directory handle, so the journal write failed
there and no remote archive or remote deletion could start.

Move the directory sync to syncDir: the unix build keeps the fsync, the
other builds return nil (the journal file itself is already synced).
The SDK reads AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE while LoadDefaultConfig
builds the archive store and captures the path; the file itself is read lazily.
Adding the name to secretEnv keeps that path out of agent child processes, the
same way AWS_WEB_IDENTITY_TOKEN_FILE is already scrubbed, and does not affect
the client.
@BaaRai
BaaRai merged commit 9057b36 into BaaRai:main Nov 20, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants