From 1ecb676e343be634ed5b96a6825ecb0d36a051a5 Mon Sep 17 00:00:00 2001 From: Chubboy Ren Date: Wed, 18 Nov 2026 02:05:19 +0000 Subject: [PATCH 1/3] feat: expose archive storage and document S3 lifecycle --- README.md | 6 + docs/task-archive-storage.md | 160 +++++++++++++++++++++ web/src/app/(main)/function/tasks/page.tsx | 5 +- web/src/lib/types.ts | 1 + 4 files changed, 170 insertions(+), 2 deletions(-) create mode 100644 docs/task-archive-storage.md diff --git a/README.md b/README.md index ad8efec17..2d9139708 100644 --- a/README.md +++ b/README.md @@ -421,6 +421,12 @@ runtime supports it. `/chat/completions` (or `/responses` for `openai-responses`), so their base **does** end in `/v1` (for example `https://api.openai.com/v1`). +**Task archive storage**: local disk by default. Set `ARTEX_ARCHIVE_STORAGE=s3` to +store cold task packages in an S3-compatible bucket and release their local +package space after verification. Existing local archives remain usable. See +[Task archive storage](docs/task-archive-storage.md) for configuration, permissions, +integrity checks, temporary disk requirements and recovery. + **Concurrency**: work agents per task, in "System Settings" (default 3). **Untrusted-output boundary (on by default)**: tool results for the worker, planner, chat and main diff --git a/docs/task-archive-storage.md b/docs/task-archive-storage.md new file mode 100644 index 000000000..113921a00 --- /dev/null +++ b/docs/task-archive-storage.md @@ -0,0 +1,160 @@ +# Task archive storage + +Task archives use local disk by default. The optional S3-compatible backend moves +cold task packages to object storage while retaining Archive, Restore and Delete. +Active workspaces and PostgreSQL remain local. This is task cold storage, not a +full-installation backup; a recovery plan must preserve the database, data +directory, bucket objects, and the credentials or roles needed to read them. + +## Configuration + +Use a private bucket and a separate prefix for each ARTEX installation. Configure +the bucket's default encryption to match the selected mode below, and enforce +that mode/key with a bucket policy. ARTEX does not provision or inspect bucket +encryption settings. + +Set the following server environment and restart ARTEX: + +```dotenv +ARTEX_ARCHIVE_STORAGE=s3 +ARTEX_ARCHIVE_S3_ENDPOINT=https://s3.us-east-1.amazonaws.com +ARTEX_ARCHIVE_S3_REGION=us-east-1 +ARTEX_ARCHIVE_S3_BUCKET=your-artex-archives +ARTEX_ARCHIVE_S3_PREFIX=artex/tasks +ARTEX_ARCHIVE_S3_SSE=AES256 +``` + +With an EC2 instance role, ECS task role or other SDK-supported credential source, +no archive-specific access keys are needed. + +| Variable | Meaning | +| --- | --- | +| `ARTEX_ARCHIVE_STORAGE` | `local` (default) or `s3`. | +| `ARTEX_ARCHIVE_S3_ENDPOINT` | Required HTTPS origin, without bucket/path/query. Loopback HTTP is allowed for local tests. | +| `ARTEX_ARCHIVE_S3_REGION` | Signing region; defaults to `us-east-1`. | +| `ARTEX_ARCHIVE_S3_BUCKET` | Required bucket name. | +| `ARTEX_ARCHIVE_S3_PREFIX` | Object prefix; defaults to `artex/tasks`. | +| `ARTEX_ARCHIVE_S3_SSE` | `AES256` (default, SSE-S3) or `aws:kms` (SSE-KMS). There is no unencrypted mode. | +| `ARTEX_ARCHIVE_S3_KMS_KEY_ID` | Optional KMS key ARN/ID/alias; valid only with `aws:kms`. Omit to use the provider's managed KMS key. | +| `ARTEX_ARCHIVE_S3_ACCESS_KEY_ID` | Optional dedicated credential override; must be supplied with the secret key. | +| `ARTEX_ARCHIVE_S3_SECRET_ACCESS_KEY` | Secret for the dedicated override. | +| `ARTEX_ARCHIVE_S3_SESSION_TOKEN` | Optional session token for dedicated temporary credentials. | +| `ARTEX_ARCHIVE_S3_PATH_STYLE` | `true` for path-style servers; otherwise virtual-hosted addressing is used. | + +An invalid mode or partial credential override fails configuration instead of +silently falling back to local storage or ambient credentials. Preserve the +bucket, endpoint, prefix and read/delete access while remote archives remain. +Older local archives remain usable; they are not uploaded automatically. + +## Encryption and permissions + +Every single-object upload and multipart initiation explicitly requests the +configured server-side encryption. Upload verification also requires HEAD to +report the requested mode, and a KMS key identifier for SSE-KMS, before accepting +the object. Size and full SHA-256 readback remain mandatory; encryption metadata +does not replace content verification. + +For SSE-KMS, set both the bucket's default encryption and ARTEX to `aws:kms`; +set `ARTEX_ARCHIVE_S3_KMS_KEY_ID` to the intended key when using a customer-managed +key. An explicit `AES256` request takes precedence over a bucket's KMS default, +so do not leave ARTEX on SSE-S3 when the bucket is intended to use KMS. + +The principal needs object upload, read (HEAD and ranged GET), delete and +multipart upload/abort permissions. KMS uploads/readback also need the applicable +key-policy and IAM permissions for `kms:GenerateDataKey` and `kms:Decrypt`. +Retain old KMS keys and decrypt permissions when changing the key or mode; restore +does not require an older object to use the current upload policy. + +Compatible providers must support the requested SSE mode and its HEAD metadata, +plus conditional ranged reads. A provider that ignores the encryption request +or cannot confirm it is rejected; the original task is retained. Local source and +temporary files keep their existing local-disk protection and restricted modes. + +See the official [PutObject encryption and permission contract](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html) +and [HEAD encryption response rules](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html). + +## Credential sources + +Without the archive-specific override, ARTEX uses the AWS SDK default credential +chain, including standard AWS environment variables, shared profiles, web +identity, ECS task credentials and EC2 instance credentials through IMDS. Role +credentials retain the SDK's automatic refresh behavior. A named shared profile +can be selected with `AWS_PROFILE`. + +For an external S3-compatible provider, dedicated keys can be supplied through +the archive-specific variables. Both key fields are required; a session token +is supported for temporary credentials. This explicit override takes precedence +over ambient AWS credentials. Operator-supplied temporary credentials must be +rotated by the operator; they are not an IAM role provider. + +Keep secrets in a protected service environment file outside the repository and +frontend. A systemd `EnvironmentFile=` can point to a root-owned mode-0600 file. +See the [AWS SDK credential-chain documentation](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-gosdk.html). + +## Lifecycle + +1. Pause or finish a task and use its **Archive** action. +2. ARTEX stages the source files and builds a local `.tar.zst` package. Large + packages use multipart upload, initially 8 MiB parts with two upload workers; + the SDK increases the part size for exceptionally large seekable files. +3. ARTEX confirms encryption metadata and reads the complete object back, + checking size and SHA-256. ETags and uploaded checksum metadata alone are + insufficient. +4. **Only after verification and a successful PostgreSQL commit** does ARTEX + remove the local package and staged source files. The committed archive + location is an `s3://` URI; there is no schema or archive-format migration. +5. **Restore** downloads and verifies before installing files and database rows. + The remote package is consumed only after restoration commits. +6. **Delete** commits removal of the archive stub, then cleans up the object. + A durable journal retains failed cleanup work. + +The archive list identifies **Object storage** and **Local storage** and shows +upload/download verification phases. Reads use bounded 8 MiB ranges pinned to the +object's ETag, with retries for interrupted ranges rather than the entire object. + +## Interruption recovery + +Source staging and restore journals live with task archives. Remote cleanup +journals live under `data/archives/tasks/.remote`; they contain locations, not +credentials. Keep this directory with PostgreSQL and the installation data. + +| Interruption | Recovery | +| --- | --- | +| During staging/compression, before an upload journal exists | Roll staged source files back. Remove any temporary package only after rollback succeeds. The task remains in PostgreSQL. | +| After the upload journal, before successful verification | Roll source files back and clean up any uncommitted object/package. Failed cleanup retains its journal for retry. | +| After verification, before the database commit | Treat the object as uncommitted: retain the original task and clean up the orphan object. Verification alone never authorizes local deletion. | +| After the database commit, before package/staging cleanup | Keep the committed object and finish local cleanup. The persisted location and checksum establish commit, even if Restore or Delete has since been queued. | +| Download/extraction before restored files are installed | Remove temporary extraction data; keep the remote object and archive stub for another attempt. | +| Restored files installed, before database restoration commits | Roll back installed files using the restore journal. Retain the remote package. | +| Database restoration committed, before remote cleanup | Recognize the already-restored task and finish cleanup without replaying database rows. | +| Object consumed, before the restored archive stub is removed | Skip download/replay; repeat idempotent object deletion and remove the stub. | +| Delete journal written, before database deletion commits | Keep the object while its archive stub exists. The queued Delete operation can retry. | +| Database deletion committed, before object deletion succeeds | The durable outbox retains the URI and retries object deletion. | +| An old failed upload survives a retry using local storage | Remove the abandoned remote object, but retain a local package referenced by the committed archive row. | + +The staging recovery rule also applies to local archives. Dedicated tests cover +both storage modes with ready, restore-queued, delete-queued and uncommitted +states. Additional PostgreSQL/HTTP-S3 tests reconstruct interrupted operations at +13 persisted boundaries and repeat recovery to check idempotence. + +Handled multipart-upload failures are aborted by the SDK. Abrupt process death +may leave unfinished multipart uploads; configure bucket lifecycle expiration +for those parts. Existing object versions, if versioning is enabled, are governed +by the bucket's version-retention policy. + +## Disk and transfer requirements + +Archive creation still requires temporary local space for the compressed +package. Restore requires space for the download and extracted files. Full +upload readback and restores incur storage reads and applicable transfer charges. +Failed remote cleanup does not prevent unrelated local archive jobs from being +processed once the failed cleanup request returns. + +## Optional connectivity check + +Run `go run ./cmd/artex-archive-check` with the archive environment to upload, +verify, download and delete a small unique object. It performs real bucket +operations but does not touch task data, and reports cleanup failures. + +Automated tests use HTTP fixtures for S3 and metadata credentials, plus isolated +PostgreSQL. They do not establish compatibility with a live cloud bucket. diff --git a/web/src/app/(main)/function/tasks/page.tsx b/web/src/app/(main)/function/tasks/page.tsx index 6b8ba8e82..c2f9cc349 100644 --- a/web/src/app/(main)/function/tasks/page.tsx +++ b/web/src/app/(main)/function/tasks/page.tsx @@ -1494,7 +1494,7 @@ function ArchiveConfirmDialog({ {count === 1 ? "Archive task" : `Archive ${count} tasks`} - Archiving stops task scheduling and compresses the graph, LLM history, files, and exclusive assets and traffic into local cold storage. Once archived, a task can be restored from "Archived". + Archiving stops task scheduling and compresses the graph, LLM history, files, and exclusive assets and traffic into the configured archive storage. Once archived, a task can be restored from "Archived". Restoring requires free space on this server. @@ -1934,6 +1934,7 @@ function TaskArchivesPanel({ onChanged }: { onChanged: () => void }) { title={`Before compression: ${formatArchiveBytes(archive.original_size)}`} > {archiveCompressionLabel(archive)} +
{archive.storage === "s3" ? "Object storage" : "Local storage"}
{archiveDataTotal(archive).toLocaleString()} @@ -1952,7 +1953,7 @@ function TaskArchivesPanel({ onChanged }: { onChanged: () => void }) {

))} - {archive.phase} · {archive.progress}% + {archive.phase === "upload_and_verify" ? "Uploading and verifying" : archive.phase === "download_and_verify" ? "Downloading and verifying" : archive.phase} · {archive.progress}%
diff --git a/web/src/lib/types.ts b/web/src/lib/types.ts index cd53b6464..f31b18ca1 100644 --- a/web/src/lib/types.ts +++ b/web/src/lib/types.ts @@ -101,6 +101,7 @@ export interface TaskArchiveTokenStats { } export interface TaskArchive { + storage?: "local" | "s3"; id: number; task_id: number; state: TaskArchiveState; From 091209798fd01ab7ea6155739796ce99537f8508 Mon Sep 17 00:00:00 2001 From: Chubboy Ren Date: Thu, 19 Nov 2026 03:33:51 +0000 Subject: [PATCH 2/3] fix: hide pending archive storage and document S3 safeguards --- docs/task-archive-storage.md | 42 +++++++++++++++------- web/src/app/(main)/function/tasks/page.tsx | 4 ++- 2 files changed, 32 insertions(+), 14 deletions(-) diff --git a/docs/task-archive-storage.md b/docs/task-archive-storage.md index 113921a00..4e07222f4 100644 --- a/docs/task-archive-storage.md +++ b/docs/task-archive-storage.md @@ -35,7 +35,7 @@ no archive-specific access keys are needed. | `ARTEX_ARCHIVE_S3_BUCKET` | Required bucket name. | | `ARTEX_ARCHIVE_S3_PREFIX` | Object prefix; defaults to `artex/tasks`. | | `ARTEX_ARCHIVE_S3_SSE` | `AES256` (default, SSE-S3) or `aws:kms` (SSE-KMS). There is no unencrypted mode. | -| `ARTEX_ARCHIVE_S3_KMS_KEY_ID` | Optional KMS key ARN/ID/alias; valid only with `aws:kms`. Omit to use the provider's managed KMS key. | +| `ARTEX_ARCHIVE_S3_KMS_KEY_ID` | Optional full KMS key ARN (not an alias or bare key ID); valid only with `aws:kms`. Omit to use the provider's managed KMS key. | | `ARTEX_ARCHIVE_S3_ACCESS_KEY_ID` | Optional dedicated credential override; must be supplied with the secret key. | | `ARTEX_ARCHIVE_S3_SECRET_ACCESS_KEY` | Secret for the dedicated override. | | `ARTEX_ARCHIVE_S3_SESSION_TOKEN` | Optional session token for dedicated temporary credentials. | @@ -50,7 +50,8 @@ Older local archives remain usable; they are not uploaded automatically. Every single-object upload and multipart initiation explicitly requests the configured server-side encryption. Upload verification also requires HEAD to -report the requested mode, and a KMS key identifier for SSE-KMS, before accepting +report the requested mode and a nonempty KMS key identifier for SSE-KMS. When a +key ARN is configured, HEAD must return that exact ARN before ARTEX accepts the object. Size and full SHA-256 readback remain mandatory; encryption metadata does not replace content verification. @@ -60,7 +61,9 @@ key. An explicit `AES256` request takes precedence over a bucket's KMS default, so do not leave ARTEX on SSE-S3 when the bucket is intended to use KMS. The principal needs object upload, read (HEAD and ranged GET), delete and -multipart upload/abort permissions. KMS uploads/readback also need the applicable +multipart upload/abort permissions. For versioned buckets, grant +`s3:DeleteObjectVersion` as well: failed verification deletes the version just +uploaded, rather than creating a delete marker. KMS uploads/readback also need the applicable key-policy and IAM permissions for `kms:GenerateDataKey` and `kms:Decrypt`. Retain old KMS keys and decrypt permissions when changing the key or mode; restore does not require an older object to use the current upload policy. @@ -87,6 +90,10 @@ is supported for temporary credentials. This explicit override takes precedence over ambient AWS credentials. Operator-supplied temporary credentials must be rotated by the operator; they are not an IAM role provider. +Both credential modes load shared AWS configuration, including `AWS_CA_BUNDLE` +for private certificate authorities. HTTP redirects are rejected for all archive +requests, including 307/308; configure the final endpoint directly. + Keep secrets in a protected service environment file outside the repository and frontend. A systemd `EnvironmentFile=` can point to a root-owned mode-0600 file. See the [AWS SDK credential-chain documentation](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-gosdk.html). @@ -108,9 +115,13 @@ See the [AWS SDK credential-chain documentation](https://docs.aws.amazon.com/sdk 6. **Delete** commits removal of the archive stub, then cleans up the object. A durable journal retains failed cleanup work. -The archive list identifies **Object storage** and **Local storage** and shows -upload/download verification phases. Reads use bounded 8 MiB ranges pinned to the -object's ETag, with retries for interrupted ranges rather than the entire object. +The archive list identifies **Object storage** or **Local storage** only once a +stored location exists; queued/uploading archives do not default to a local +label. Upload/download verification phases remain visible. Reads use bounded +8 MiB ranges pinned to the object's ETag. Each range has at most five HTTP +attempts total, including transient service errors and interrupted bodies; +SDK retries are disabled inside that loop. Each attempt has a two-minute limit +and cancellation interrupts both requests and retry delays. ## Interruption recovery @@ -120,7 +131,7 @@ credentials. Keep this directory with PostgreSQL and the installation data. | Interruption | Recovery | | --- | --- | -| During staging/compression, before an upload journal exists | Roll staged source files back. Remove any temporary package only after rollback succeeds. The task remains in PostgreSQL. | +| During staging/compression, before an upload journal exists | Roll staged source files back. Remove any temporary package, including a partially written `.partial` file, only after rollback succeeds. Keep the staging journal if cleanup fails so recovery can retry. The task remains in PostgreSQL. | | After the upload journal, before successful verification | Roll source files back and clean up any uncommitted object/package. Failed cleanup retains its journal for retry. | | After verification, before the database commit | Treat the object as uncommitted: retain the original task and clean up the orphan object. Verification alone never authorizes local deletion. | | After the database commit, before package/staging cleanup | Keep the committed object and finish local cleanup. The persisted location and checksum establish commit, even if Restore or Delete has since been queued. | @@ -135,12 +146,17 @@ credentials. Keep this directory with PostgreSQL and the installation data. The staging recovery rule also applies to local archives. Dedicated tests cover both storage modes with ready, restore-queued, delete-queued and uncommitted states. Additional PostgreSQL/HTTP-S3 tests reconstruct interrupted operations at -13 persisted boundaries and repeat recovery to check idempotence. - -Handled multipart-upload failures are aborted by the SDK. Abrupt process death -may leave unfinished multipart uploads; configure bucket lifecycle expiration -for those parts. Existing object versions, if versioning is enabled, are governed -by the bucket's version-retention policy. +14 persisted boundaries and repeat recovery to check idempotence. + +Handled multipart-upload failures, including cancellation, trigger an abort with +a separate 30-second cleanup context. A failed encryption/content verification +similarly deletes the newly uploaded object (its exact version when available) +while retaining the source. Cleanup failures are returned together with the +original error. Abrupt process death may still leave unfinished multipart uploads +or object versions; configure bucket lifecycle expiration for incomplete uploads +and a version-retention policy. The URI-based recovery journal does not retain +version IDs, so lifecycle policy remains necessary after a crash or failed +version cleanup. ## Disk and transfer requirements diff --git a/web/src/app/(main)/function/tasks/page.tsx b/web/src/app/(main)/function/tasks/page.tsx index c2f9cc349..8caee9c5d 100644 --- a/web/src/app/(main)/function/tasks/page.tsx +++ b/web/src/app/(main)/function/tasks/page.tsx @@ -1934,7 +1934,9 @@ function TaskArchivesPanel({ onChanged }: { onChanged: () => void }) { title={`Before compression: ${formatArchiveBytes(archive.original_size)}`} > {archiveCompressionLabel(archive)} -
{archive.storage === "s3" ? "Object storage" : "Local storage"}
+ {(archive.storage === "s3" || archive.storage === "local") && ( +
{archive.storage === "s3" ? "Object storage" : "Local storage"}
+ )} {archiveDataTotal(archive).toLocaleString()} From 801b1e155096d8d7d6187c2ef2162ebbfdff2d0c Mon Sep 17 00:00:00 2001 From: renaesop Date: Sat, 21 Nov 2026 02:57:34 +0000 Subject: [PATCH 3/3] fix: complete archive translations and document startup credential handling --- docs/task-archive-storage.md | 7 +++++++ web/src/i18n/locales/zh-CN.json | 6 +++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/task-archive-storage.md b/docs/task-archive-storage.md index 4e07222f4..cd2f49dd2 100644 --- a/docs/task-archive-storage.md +++ b/docs/task-archive-storage.md @@ -90,6 +90,13 @@ is supported for temporary credentials. This explicit override takes precedence over ambient AWS credentials. Operator-supplied temporary credentials must be rotated by the operator; they are not an IAM role provider. +At server startup, the archive client captures its credential configuration before +ARTEX removes archive-specific keys and supported ambient AWS credential variables +from the process environment. Agent shells, tools and MCP stdio servers therefore +do not inherit those values. Role providers retain the configuration needed for +refresh. This environment cleanup does not restrict access to shared credential +files, token files or instance metadata; those require their own access controls. + Both credential modes load shared AWS configuration, including `AWS_CA_BUNDLE` for private certificate authorities. HTTP redirects are rejected for all archive requests, including 307/308; configure the final endpoint directly. diff --git a/web/src/i18n/locales/zh-CN.json b/web/src/i18n/locales/zh-CN.json index cddee1419..0b64bd7c6 100644 --- a/web/src/i18n/locales/zh-CN.json +++ b/web/src/i18n/locales/zh-CN.json @@ -5,6 +5,10 @@ "-1 = no rerun; the intent is marked blocked outright": "-1 = 不重跑,该意图直接判为 blocked", "-1 = no retries at all; the failure bubbles up immediately": "-1 = 一次都不重试,失败立刻上抛", "-1 = transient failures never trip (deterministic failures still do)": "-1 = 瞬时失败永不熔断(确定性失败仍然熔断)", + "Downloading and verifying": "正在下载并校验", + "Local storage": "本地存储", + "Object storage": "对象存储", + "Uploading and verifying": "正在上传并校验", "— (task deleted)": "—(任务已删除)", "— agent generates —": "— 由智能体生成 —", "… {count} errors in total": "…共 {count} 个错误", @@ -307,7 +311,7 @@ "Archived": "已归档", "Archived at": "归档时间", "Archiving": "归档中", - "Archiving stops task scheduling and compresses the graph, LLM history, files, and exclusive assets and traffic into local cold storage. Once archived, a task can be restored from \"Archived\".": "归档会停止任务调度,将图谱、LLM 历史、文件以及独占资产和流量压缩到本地冷存储。归档完成后可从“已归档”中还原。", + "Archiving stops task scheduling and compresses the graph, LLM history, files, and exclusive assets and traffic into the configured archive storage. Once archived, a task can be restored from \"Archived\". Restoring requires free space on this server.": "归档会停止任务调度,将图谱、LLM 历史、文件以及独占资产和流量压缩到已配置的归档存储中。归档完成后可从“已归档”中还原。还原需要此服务器有足够的可用空间。", "Are you sure you want to delete asset interception rule \"{pattern}\"?": "确定删除资产拦截规则“{pattern}”?", "Arguments": "参数", "Arguments (space-separated)": "参数(空格分隔)",