Skip to content

feat: add encrypted S3 archive storage backend - #285

Merged
BaaRai merged 6 commits into
BaaRai:mainfrom
renaesop:codex/s3-task-archives
Nov 19, 2026
Merged

BaaRai merged 6 commits into
BaaRai:mainfrom
renaesop:codex/s3-task-archives

Conversation

@renaesop

@renaesop renaesop commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem and benefit

Cold archives consume application-server disk, especially large APKs and app-analysis artifacts produced through REA MCP. This adds encrypted S3-compatible package storage with bounded transfer memory. It is part 1 of the review-requested split; it adds the backend, connectivity checker, SDK dependencies and backend CI coverage. Server behavior is connected in draft #318; UI/operator documentation is in draft #319.

Behavior

  • Require HTTPS except for loopback fixtures, and constrain locations to the configured bucket/prefix.
  • Explicitly request SSE-S3 (AES256, default) or SSE-KMS; HEAD must confirm the mode. A configured KMS key must be a full key ARN and match HEAD exactly; aliases and bare IDs are rejected. With no configured KMS key, require a nonempty provider-managed key identifier.
  • Verify complete size/SHA-256 before upload succeeds. On failed verification, retain the source and delete the uploaded object, targeting its returned version ID on versioned buckets. Return both errors if cleanup fails.
  • Abort failed/cancelled multipart uploads with a separate 30-second context. Cleanup failures remain visible.
  • Reject every HTTP redirect, including 307/308. Load shared AWS configuration, including AWS_CA_BUNDLE, with both default-chain and explicit credentials; retain refreshable role credentials and session-token support.
  • Read ETag-pinned 8 MiB ranges with at most five HTTP attempts per range, covering transient status errors and interrupted bodies without multiplying SDK retries. Each attempt is limited to two minutes.
  • Include archivestore in the CI race package list.

The optional connectivity command uploads, verifies, downloads and deletes a unique probe without touching task data. Configuration, permissions, recovery and version-retention limits are documented in part 3.

Validation

Validation ran on shein-artex in isolated worktrees against upstream 65045ef.

Tests use isolated HTTP/metadata endpoints and temporary PostgreSQL, not a live cloud bucket. Abrupt process death and failed version cleanup still require bucket lifecycle/version-retention policies. Server wiring and UI remain separate dependent drafts.

Size

The earlier full-integration measurement with Go 1.26.10, Linux amd64, and -trimpath -ldflags='-s -w' was 32,944,393 → 41,754,889 bytes (+8.40 MiB) against upstream ac5e714. That measurement predates this follow-up and newer main; SDK dependencies are unchanged by these fixes. This backend is not linked into the server until #318.

@BaaRai BaaRai left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the scope and the test work. Before any code review, please answer these design points and split the PR.

  1. Encryption at rest. The packages contain pentest data, and PutObject sets no ServerSideEncryption and nothing encrypts client-side. Please set SSE explicitly (configurable) or encrypt client-side with the instance key, and document what the bucket's default encryption must be.
  2. Credentials. Only static keys from the environment are supported, with no IAM role or IMDS. Please justify this or add the alternatives.
  3. Lifecycle. Please document that the local package is deleted only after the remote copy is verified and the database commit succeeds, and describe the recovery path for each interruption. Also cover the change in recoverTaskArchiveStages (server/task_archive_package.go:431), which affects local mode too and has no dedicated test.

Suggested split: backend and tests (archivestore, the check command, dependencies, encryption); server and database wiring (remote journals, recovery, with Postgres and a fake S3 tests for each interruption point); UI and docs. Please note the roughly 8 MB binary size increase in the PR.

@renaesop renaesop changed the title feat: archive cold tasks to S3-compatible object storage feat: add encrypted S3 archive storage backend Nov 18, 2026
@renaesop

Copy link
Copy Markdown
Contributor Author

@BaaRai Split and updated as requested:

Each follow-up has its own scoped diff. The combined tested implementation is retained at da7cc62; the server draft explicitly notes that its standalone head needs the backend dependency before building.

Encryption: uploads now explicitly request AES256 by default, with configurable aws:kms and a KMS key. HEAD must confirm the requested mode before full content verification succeeds. There is no unencrypted option. The document requires matching bucket-default encryption and bucket policy, lists KMS permissions, and explains retaining old keys for restores.

Credentials: added the SDK default chain, including IAM roles through ECS/EC2 IMDS, shared profiles and web identity. Dedicated static or temporary-key overrides remain available and incomplete overrides fail closed. Fixtures exercise environment/profile/ECS/IMDSv2 and session-token credentials.

Lifecycle: local packages/source staging are removed only after full remote verification and the database commit. The document now has a recovery matrix for archive/restore/delete interruptions. Part 2 adds 13 persisted-boundary tests with PostgreSQL and fake S3, repeated recovery, plus dedicated local and S3 tests for the location/checksum staging rule in ready, restore-queued, delete-queued and uncommitted states. It also handles a temporary package left before the upload journal exists, deleting it only after source rollback succeeds.

Size: the complete Linux amd64 server binary grows from 32,944,393 to 41,754,889 bytes: 8.40 MiB / 8.81 MB, measured with Go 1.26.10 and identical -trimpath -ldflags='-s -w' settings. This is recorded in the PR descriptions.

Backend race tests, standalone backend build/vet/tidy checks, the complete integrated server suite and UI type checking all passed on the remote machine. Tests use isolated fixtures, not a live bucket. Configuration and the interruption matrix are in the documentation PR.

@BaaRai BaaRai left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the backend is close. Three things block it; I checked each at a40781a.

  1. KMS key id is not compared (archivestore/s3.go:197-200). After upload, HEAD only checks that SSEKMSKeyId is non-empty. A bucket default or an S3-compatible provider that encrypts with a different key passes. Compare the returned key with the configured one, and add a test where HEAD returns another key. If you want aliases to keep working, resolve them at startup; otherwise document that the configured value must be the key ARN or id S3 returns.

  2. A verification failure leaves the object in the bucket (s3.go:149-185). Upload returns the error but the object, possibly not encrypted as requested, stays under the deterministic key. Delete it when readVerified fails (Delete already exists), and return both errors if the delete also fails. Add a test that the object is gone and the source file is untouched.

  3. The custom HTTP client follows redirects (s3.go:99-101). The SDK's default client does not, so a signed request and its session token are not replayed to another host. Set CheckRedirect to return http.ErrUseLastResponse and cover it with a fake endpoint that redirects.

Non-blocking, but worth addressing in the same pass:

  • check that a cancelled multipart upload is aborted, not left as billable parts;
  • the explicit-credentials path builds aws.Config directly and skips LoadDefaultConfig, so settings such as AWS_CA_BUNDLE do not apply; say so or load the shared config and override credentials;
  • the chunk retry loop (5) sits on top of the SDK's own retries; bound the total;
  • add ./archivestore to the go test -race package list in ci.yml, it is not covered today.

Please add the three tests and run the archivestore suite with -race.

@renaesop

Copy link
Copy Markdown
Contributor Author

Addressed all three blockers and all four follow-ups in 3755e95; merged current upstream main in fa84c00.

  1. Configured KMS keys now require a full key ARN and must exactly match HEAD. Aliases/bare IDs are rejected rather than silently accepting a different key. A fake HEAD returning another key is rejected.
  2. Failed upload verification deletes the just-uploaded object/version with an independent 30-second cleanup context, preserving the source. If deletion also fails, both errors survive. Tests cover wrong KMS, corrupt readback, cancellation, version-specific deletion, denied cleanup and unchanged source bytes.
  3. Signed requests never follow redirects. Tests cover 301/302/303/307/308 and assert zero requests reach the redirect target. The pinned SDK actually still permits 307/308, so its default redirect behavior alone was insufficient.
  4. Failed/cancelled multipart uploads explicitly abort with a fresh bounded context; abort failures are returned. Cancellation and denied-abort regressions cover this.
  5. Both credential modes use LoadDefaultConfig, including AWS_CA_BUNDLE. The resulting CA-configured transport is used by a no-redirect client. TLS fixture tests verify both default and explicit credentials.
  6. Each range has at most five HTTP attempts total, with SDK retries disabled for that GetObject call. Tests assert five requests for 429/503 and one for non-retryable 403; interrupted-body retries remain covered.
  7. Added archivestore to ci.yml's race package list.

The dependent drafts also fix partial compression-file recovery and the false Local storage label during uploads. Their docs now specify full KMS ARNs, DeleteObjectVersion permissions, CA/redirect behavior and remaining crash/version-retention limits. #318 stays draft until this backend is available on main; #319 follows #318.

@BaaRai BaaRai left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked at fa84c00: the KMS key must now be a full key ARN and HEAD must return that exact ARN (B1), a failed post-upload verification deletes the object, including its version (B2), redirects are refused with ErrUseLastResponse while the CA-configured transport is kept (N1), and ./archivestore is in the -race list. Multipart abort and the retry loop are covered. CI green.

@BaaRai
BaaRai merged commit 05d3208 into BaaRai:main Nov 19, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants