Repository navigation
feat: add encrypted S3 archive storage backend - #285
Conversation
BaaRai
left a comment
There was a problem hiding this comment.
Thanks for the scope and the test work. Before any code review, please answer these design points and split the PR.
- Encryption at rest. The packages contain pentest data, and
PutObjectsets noServerSideEncryptionand nothing encrypts client-side. Please set SSE explicitly (configurable) or encrypt client-side with the instance key, and document what the bucket's default encryption must be. - Credentials. Only static keys from the environment are supported, with no IAM role or IMDS. Please justify this or add the alternatives.
- Lifecycle. Please document that the local package is deleted only after the remote copy is verified and the database commit succeeds, and describe the recovery path for each interruption. Also cover the change in
recoverTaskArchiveStages(server/task_archive_package.go:431), which affects local mode too and has no dedicated test.
Suggested split: backend and tests (archivestore, the check command, dependencies, encryption); server and database wiring (remote journals, recovery, with Postgres and a fake S3 tests for each interruption point); UI and docs. Please note the roughly 8 MB binary size increase in the PR.
|
@BaaRai Split and updated as requested:
Each follow-up has its own scoped diff. The combined tested implementation is retained at da7cc62; the server draft explicitly notes that its standalone head needs the backend dependency before building. Encryption: uploads now explicitly request AES256 by default, with configurable Credentials: added the SDK default chain, including IAM roles through ECS/EC2 IMDS, shared profiles and web identity. Dedicated static or temporary-key overrides remain available and incomplete overrides fail closed. Fixtures exercise environment/profile/ECS/IMDSv2 and session-token credentials. Lifecycle: local packages/source staging are removed only after full remote verification and the database commit. The document now has a recovery matrix for archive/restore/delete interruptions. Part 2 adds 13 persisted-boundary tests with PostgreSQL and fake S3, repeated recovery, plus dedicated local and S3 tests for the location/checksum staging rule in ready, restore-queued, delete-queued and uncommitted states. It also handles a temporary package left before the upload journal exists, deleting it only after source rollback succeeds. Size: the complete Linux amd64 server binary grows from 32,944,393 to 41,754,889 bytes: 8.40 MiB / 8.81 MB, measured with Go 1.26.10 and identical Backend race tests, standalone backend build/vet/tidy checks, the complete integrated server suite and UI type checking all passed on the remote machine. Tests use isolated fixtures, not a live bucket. Configuration and the interruption matrix are in the documentation PR. |
BaaRai
left a comment
There was a problem hiding this comment.
Thanks, the backend is close. Three things block it; I checked each at a40781a.
-
KMS key id is not compared (archivestore/s3.go:197-200). After upload, HEAD only checks that SSEKMSKeyId is non-empty. A bucket default or an S3-compatible provider that encrypts with a different key passes. Compare the returned key with the configured one, and add a test where HEAD returns another key. If you want aliases to keep working, resolve them at startup; otherwise document that the configured value must be the key ARN or id S3 returns.
-
A verification failure leaves the object in the bucket (s3.go:149-185). Upload returns the error but the object, possibly not encrypted as requested, stays under the deterministic key. Delete it when readVerified fails (Delete already exists), and return both errors if the delete also fails. Add a test that the object is gone and the source file is untouched.
-
The custom HTTP client follows redirects (s3.go:99-101). The SDK's default client does not, so a signed request and its session token are not replayed to another host. Set CheckRedirect to return http.ErrUseLastResponse and cover it with a fake endpoint that redirects.
Non-blocking, but worth addressing in the same pass:
- check that a cancelled multipart upload is aborted, not left as billable parts;
- the explicit-credentials path builds aws.Config directly and skips LoadDefaultConfig, so settings such as AWS_CA_BUNDLE do not apply; say so or load the shared config and override credentials;
- the chunk retry loop (5) sits on top of the SDK's own retries; bound the total;
- add ./archivestore to the go test -race package list in ci.yml, it is not covered today.
Please add the three tests and run the archivestore suite with -race.
|
Addressed all three blockers and all four follow-ups in 3755e95; merged current upstream main in fa84c00.
The dependent drafts also fix partial compression-file recovery and the false Local storage label during uploads. Their docs now specify full KMS ARNs, DeleteObjectVersion permissions, CA/redirect behavior and remaining crash/version-retention limits. #318 stays draft until this backend is available on main; #319 follows #318. |
BaaRai
left a comment
There was a problem hiding this comment.
Re-checked at fa84c00: the KMS key must now be a full key ARN and HEAD must return that exact ARN (B1), a failed post-upload verification deletes the object, including its version (B2), redirects are refused with ErrUseLastResponse while the CA-configured transport is kept (N1), and ./archivestore is in the -race list. Multipart abort and the retry loop are covered. CI green.
Problem and benefit
Cold archives consume application-server disk, especially large APKs and app-analysis artifacts produced through REA MCP. This adds encrypted S3-compatible package storage with bounded transfer memory. It is part 1 of the review-requested split; it adds the backend, connectivity checker, SDK dependencies and backend CI coverage. Server behavior is connected in draft #318; UI/operator documentation is in draft #319.
Behavior
The optional connectivity command uploads, verifies, downloads and deletes a unique probe without touching task data. Configuration, permissions, recovery and version-retention limits are documented in part 3.
Validation
Validation ran on shein-artex in isolated worktrees against upstream 65045ef.
-race, including mismatched KMS keys, failed verification/version cleanup, cancellation/abort, all redirect statuses, private CA bundles and the five-request retry ceiling.go build -p 1 ./...,go vet -p 1 ./..., andgo mod tidy -diff.tsc --noEmit --incremental falsepassed in the remote UI worktree.Tests use isolated HTTP/metadata endpoints and temporary PostgreSQL, not a live cloud bucket. Abrupt process death and failed version cleanup still require bucket lifecycle/version-retention policies. Server wiring and UI remain separate dependent drafts.
Size
The earlier full-integration measurement with Go 1.26.10, Linux amd64, and -trimpath -ldflags='-s -w' was 32,944,393 → 41,754,889 bytes (+8.40 MiB) against upstream ac5e714. That measurement predates this follow-up and newer main; SDK dependencies are unchanged by these fixes. This backend is not linked into the server until #318.