diff --git a/README.md b/README.md index 057d744aa..8d25da20f 100644 --- a/README.md +++ b/README.md @@ -436,6 +436,12 @@ catalog (`GET /v1/models`) and pick a model; models the provider adds later appe next refresh. For example, [NotHoly](https://notholy.com) (offensive-security models) uses base URL `https://api.notholy.com/v1` with an API key from its console. +**Task archive storage**: local disk by default. Set `ARTEX_ARCHIVE_STORAGE=s3` to +store cold task packages in an S3-compatible bucket and release their local +package space after verification. Existing local archives remain usable. See +[Task archive storage](docs/task-archive-storage.md) for configuration, permissions, +integrity checks, temporary disk requirements and recovery. + **Concurrency**: work agents per task, in "System Settings" (default 3). **Untrusted-output boundary (on by default)**: tool results for the worker, planner, chat and main diff --git a/docs/task-archive-storage.md b/docs/task-archive-storage.md new file mode 100644 index 000000000..cd2f49dd2 --- /dev/null +++ b/docs/task-archive-storage.md @@ -0,0 +1,183 @@ +# Task archive storage + +Task archives use local disk by default. The optional S3-compatible backend moves +cold task packages to object storage while retaining Archive, Restore and Delete. +Active workspaces and PostgreSQL remain local. This is task cold storage, not a +full-installation backup; a recovery plan must preserve the database, data +directory, bucket objects, and the credentials or roles needed to read them. + +## Configuration + +Use a private bucket and a separate prefix for each ARTEX installation. Configure +the bucket's default encryption to match the selected mode below, and enforce +that mode/key with a bucket policy. ARTEX does not provision or inspect bucket +encryption settings. + +Set the following server environment and restart ARTEX: + +```dotenv +ARTEX_ARCHIVE_STORAGE=s3 +ARTEX_ARCHIVE_S3_ENDPOINT=https://s3.us-east-1.amazonaws.com +ARTEX_ARCHIVE_S3_REGION=us-east-1 +ARTEX_ARCHIVE_S3_BUCKET=your-artex-archives +ARTEX_ARCHIVE_S3_PREFIX=artex/tasks +ARTEX_ARCHIVE_S3_SSE=AES256 +``` + +With an EC2 instance role, ECS task role or other SDK-supported credential source, +no archive-specific access keys are needed. + +| Variable | Meaning | +| --- | --- | +| `ARTEX_ARCHIVE_STORAGE` | `local` (default) or `s3`. | +| `ARTEX_ARCHIVE_S3_ENDPOINT` | Required HTTPS origin, without bucket/path/query. Loopback HTTP is allowed for local tests. | +| `ARTEX_ARCHIVE_S3_REGION` | Signing region; defaults to `us-east-1`. | +| `ARTEX_ARCHIVE_S3_BUCKET` | Required bucket name. | +| `ARTEX_ARCHIVE_S3_PREFIX` | Object prefix; defaults to `artex/tasks`. | +| `ARTEX_ARCHIVE_S3_SSE` | `AES256` (default, SSE-S3) or `aws:kms` (SSE-KMS). There is no unencrypted mode. | +| `ARTEX_ARCHIVE_S3_KMS_KEY_ID` | Optional full KMS key ARN (not an alias or bare key ID); valid only with `aws:kms`. Omit to use the provider's managed KMS key. | +| `ARTEX_ARCHIVE_S3_ACCESS_KEY_ID` | Optional dedicated credential override; must be supplied with the secret key. | +| `ARTEX_ARCHIVE_S3_SECRET_ACCESS_KEY` | Secret for the dedicated override. | +| `ARTEX_ARCHIVE_S3_SESSION_TOKEN` | Optional session token for dedicated temporary credentials. | +| `ARTEX_ARCHIVE_S3_PATH_STYLE` | `true` for path-style servers; otherwise virtual-hosted addressing is used. | + +An invalid mode or partial credential override fails configuration instead of +silently falling back to local storage or ambient credentials. Preserve the +bucket, endpoint, prefix and read/delete access while remote archives remain. +Older local archives remain usable; they are not uploaded automatically. + +## Encryption and permissions + +Every single-object upload and multipart initiation explicitly requests the +configured server-side encryption. Upload verification also requires HEAD to +report the requested mode and a nonempty KMS key identifier for SSE-KMS. When a +key ARN is configured, HEAD must return that exact ARN before ARTEX accepts +the object. Size and full SHA-256 readback remain mandatory; encryption metadata +does not replace content verification. + +For SSE-KMS, set both the bucket's default encryption and ARTEX to `aws:kms`; +set `ARTEX_ARCHIVE_S3_KMS_KEY_ID` to the intended key when using a customer-managed +key. An explicit `AES256` request takes precedence over a bucket's KMS default, +so do not leave ARTEX on SSE-S3 when the bucket is intended to use KMS. + +The principal needs object upload, read (HEAD and ranged GET), delete and +multipart upload/abort permissions. For versioned buckets, grant +`s3:DeleteObjectVersion` as well: failed verification deletes the version just +uploaded, rather than creating a delete marker. KMS uploads/readback also need the applicable +key-policy and IAM permissions for `kms:GenerateDataKey` and `kms:Decrypt`. +Retain old KMS keys and decrypt permissions when changing the key or mode; restore +does not require an older object to use the current upload policy. + +Compatible providers must support the requested SSE mode and its HEAD metadata, +plus conditional ranged reads. A provider that ignores the encryption request +or cannot confirm it is rejected; the original task is retained. Local source and +temporary files keep their existing local-disk protection and restricted modes. + +See the official [PutObject encryption and permission contract](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html) +and [HEAD encryption response rules](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html). + +## Credential sources + +Without the archive-specific override, ARTEX uses the AWS SDK default credential +chain, including standard AWS environment variables, shared profiles, web +identity, ECS task credentials and EC2 instance credentials through IMDS. Role +credentials retain the SDK's automatic refresh behavior. A named shared profile +can be selected with `AWS_PROFILE`. + +For an external S3-compatible provider, dedicated keys can be supplied through +the archive-specific variables. Both key fields are required; a session token +is supported for temporary credentials. This explicit override takes precedence +over ambient AWS credentials. Operator-supplied temporary credentials must be +rotated by the operator; they are not an IAM role provider. + +At server startup, the archive client captures its credential configuration before +ARTEX removes archive-specific keys and supported ambient AWS credential variables +from the process environment. Agent shells, tools and MCP stdio servers therefore +do not inherit those values. Role providers retain the configuration needed for +refresh. This environment cleanup does not restrict access to shared credential +files, token files or instance metadata; those require their own access controls. + +Both credential modes load shared AWS configuration, including `AWS_CA_BUNDLE` +for private certificate authorities. HTTP redirects are rejected for all archive +requests, including 307/308; configure the final endpoint directly. + +Keep secrets in a protected service environment file outside the repository and +frontend. A systemd `EnvironmentFile=` can point to a root-owned mode-0600 file. +See the [AWS SDK credential-chain documentation](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-gosdk.html). + +## Lifecycle + +1. Pause or finish a task and use its **Archive** action. +2. ARTEX stages the source files and builds a local `.tar.zst` package. Large + packages use multipart upload, initially 8 MiB parts with two upload workers; + the SDK increases the part size for exceptionally large seekable files. +3. ARTEX confirms encryption metadata and reads the complete object back, + checking size and SHA-256. ETags and uploaded checksum metadata alone are + insufficient. +4. **Only after verification and a successful PostgreSQL commit** does ARTEX + remove the local package and staged source files. The committed archive + location is an `s3://` URI; there is no schema or archive-format migration. +5. **Restore** downloads and verifies before installing files and database rows. + The remote package is consumed only after restoration commits. +6. **Delete** commits removal of the archive stub, then cleans up the object. + A durable journal retains failed cleanup work. + +The archive list identifies **Object storage** or **Local storage** only once a +stored location exists; queued/uploading archives do not default to a local +label. Upload/download verification phases remain visible. Reads use bounded +8 MiB ranges pinned to the object's ETag. Each range has at most five HTTP +attempts total, including transient service errors and interrupted bodies; +SDK retries are disabled inside that loop. Each attempt has a two-minute limit +and cancellation interrupts both requests and retry delays. + +## Interruption recovery + +Source staging and restore journals live with task archives. Remote cleanup +journals live under `data/archives/tasks/.remote`; they contain locations, not +credentials. Keep this directory with PostgreSQL and the installation data. + +| Interruption | Recovery | +| --- | --- | +| During staging/compression, before an upload journal exists | Roll staged source files back. Remove any temporary package, including a partially written `.partial` file, only after rollback succeeds. Keep the staging journal if cleanup fails so recovery can retry. The task remains in PostgreSQL. | +| After the upload journal, before successful verification | Roll source files back and clean up any uncommitted object/package. Failed cleanup retains its journal for retry. | +| After verification, before the database commit | Treat the object as uncommitted: retain the original task and clean up the orphan object. Verification alone never authorizes local deletion. | +| After the database commit, before package/staging cleanup | Keep the committed object and finish local cleanup. The persisted location and checksum establish commit, even if Restore or Delete has since been queued. | +| Download/extraction before restored files are installed | Remove temporary extraction data; keep the remote object and archive stub for another attempt. | +| Restored files installed, before database restoration commits | Roll back installed files using the restore journal. Retain the remote package. | +| Database restoration committed, before remote cleanup | Recognize the already-restored task and finish cleanup without replaying database rows. | +| Object consumed, before the restored archive stub is removed | Skip download/replay; repeat idempotent object deletion and remove the stub. | +| Delete journal written, before database deletion commits | Keep the object while its archive stub exists. The queued Delete operation can retry. | +| Database deletion committed, before object deletion succeeds | The durable outbox retains the URI and retries object deletion. | +| An old failed upload survives a retry using local storage | Remove the abandoned remote object, but retain a local package referenced by the committed archive row. | + +The staging recovery rule also applies to local archives. Dedicated tests cover +both storage modes with ready, restore-queued, delete-queued and uncommitted +states. Additional PostgreSQL/HTTP-S3 tests reconstruct interrupted operations at +14 persisted boundaries and repeat recovery to check idempotence. + +Handled multipart-upload failures, including cancellation, trigger an abort with +a separate 30-second cleanup context. A failed encryption/content verification +similarly deletes the newly uploaded object (its exact version when available) +while retaining the source. Cleanup failures are returned together with the +original error. Abrupt process death may still leave unfinished multipart uploads +or object versions; configure bucket lifecycle expiration for incomplete uploads +and a version-retention policy. The URI-based recovery journal does not retain +version IDs, so lifecycle policy remains necessary after a crash or failed +version cleanup. + +## Disk and transfer requirements + +Archive creation still requires temporary local space for the compressed +package. Restore requires space for the download and extracted files. Full +upload readback and restores incur storage reads and applicable transfer charges. +Failed remote cleanup does not prevent unrelated local archive jobs from being +processed once the failed cleanup request returns. + +## Optional connectivity check + +Run `go run ./cmd/artex-archive-check` with the archive environment to upload, +verify, download and delete a small unique object. It performs real bucket +operations but does not touch task data, and reports cleanup failures. + +Automated tests use HTTP fixtures for S3 and metadata credentials, plus isolated +PostgreSQL. They do not establish compatibility with a live cloud bucket. diff --git a/web/src/app/(main)/function/tasks/page.tsx b/web/src/app/(main)/function/tasks/page.tsx index 7aa259bd0..d6ac89a10 100644 --- a/web/src/app/(main)/function/tasks/page.tsx +++ b/web/src/app/(main)/function/tasks/page.tsx @@ -1555,7 +1555,7 @@ function ArchiveConfirmDialog({ {count === 1 ? t("Archive task") : t("Archive {count} tasks", { count })} {t( - "Archiving stops task scheduling and compresses the graph, LLM history, files, and exclusive assets and traffic into local cold storage. Once archived, a task can be restored from \"Archived\".", + "Archiving stops task scheduling and compresses the graph, LLM history, files, and exclusive assets and traffic into the configured archive storage. Once archived, a task can be restored from \"Archived\". Restoring requires free space on this server.", )} @@ -1999,6 +1999,11 @@ function TaskArchivesPanel({ onChanged }: { onChanged: () => void }) { title={t("Before compression: {size}", { size: formatArchiveBytes(archive.original_size) })} > {archiveCompressionLabel(archive)} + {(archive.storage === "s3" || archive.storage === "local") && ( +
+ {archive.storage === "s3" ? t("Object storage") : t("Local storage")} +
+ )} {archiveDataTotal(archive).toLocaleString(intlLocale)} @@ -2017,7 +2022,11 @@ function TaskArchivesPanel({ onChanged }: { onChanged: () => void }) {

))} - {t(archive.phase)} · {archive.progress}% + {archive.phase === "upload_and_verify" + ? t("Uploading and verifying") + : archive.phase === "download_and_verify" + ? t("Downloading and verifying") + : t(archive.phase)} · {archive.progress}%
diff --git a/web/src/i18n/locales/tr.json b/web/src/i18n/locales/tr.json index 7158f7956..5df2a691f 100644 --- a/web/src/i18n/locales/tr.json +++ b/web/src/i18n/locales/tr.json @@ -5,6 +5,10 @@ "-1 = no rerun; the intent is marked blocked outright": "-1 = yeniden çalıştırma yok; niyet doğrudan engellendi olarak işaretlenir", "-1 = no retries at all; the failure bubbles up immediately": "-1 = hiç yeniden deneme yok; hata hemen yukarı iletilir", "-1 = transient failures never trip (deterministic failures still do)": "-1 = geçici hatalar asla devre kesmez (kesin hatalar yine keser)", + "Downloading and verifying": "İndiriliyor ve doğrulanıyor", + "Local storage": "Yerel depolama", + "Object storage": "Nesne depolama", + "Uploading and verifying": "Karşıya yükleniyor ve doğrulanıyor", "— (task deleted)": "— (görev silinmiş)", "— agent generates —": "— ajan oluşturur —", "… {count} errors in total": "… toplam {count} hata", @@ -307,7 +311,7 @@ "Archived": "Arşivlenenler", "Archived at": "Arşivlenme zamanı", "Archiving": "Arşivleniyor", - "Archiving stops task scheduling and compresses the graph, LLM history, files, and exclusive assets and traffic into local cold storage. Once archived, a task can be restored from \"Archived\".": "Arşivleme, görev zamanlamasını durdurur ve grafiği, LLM geçmişini, dosyaları, göreve özel varlıkları ve trafiği yerel soğuk depolamaya sıkıştırır. Arşivlenen görev \"Arşivlenenler\" sekmesinden geri yüklenebilir.", + "Archiving stops task scheduling and compresses the graph, LLM history, files, and exclusive assets and traffic into the configured archive storage. Once archived, a task can be restored from \"Archived\". Restoring requires free space on this server.": "Arşivleme, görev zamanlamasını durdurur ve grafiği, LLM geçmişini, dosyaları, özel varlıkları ve trafiği yapılandırılmış arşiv depolama alanına sıkıştırır. Arşivlenen bir görev, \"Arşivlenenler\" bölümünden geri yüklenebilir. Geri yükleme, bu sunucuda boş alan gerektirir.", "Are you sure you want to delete asset interception rule \"{pattern}\"?": "\"{pattern}\" varlık yakalama kuralını silmek istediğinizden emin misiniz?", "Arguments": "Argümanlar", "Arguments (space-separated)": "Argümanlar (boşlukla ayrılmış)", diff --git a/web/src/i18n/locales/zh-CN.json b/web/src/i18n/locales/zh-CN.json index cddee1419..0b64bd7c6 100644 --- a/web/src/i18n/locales/zh-CN.json +++ b/web/src/i18n/locales/zh-CN.json @@ -5,6 +5,10 @@ "-1 = no rerun; the intent is marked blocked outright": "-1 = 不重跑,该意图直接判为 blocked", "-1 = no retries at all; the failure bubbles up immediately": "-1 = 一次都不重试,失败立刻上抛", "-1 = transient failures never trip (deterministic failures still do)": "-1 = 瞬时失败永不熔断(确定性失败仍然熔断)", + "Downloading and verifying": "正在下载并校验", + "Local storage": "本地存储", + "Object storage": "对象存储", + "Uploading and verifying": "正在上传并校验", "— (task deleted)": "—(任务已删除)", "— agent generates —": "— 由智能体生成 —", "… {count} errors in total": "…共 {count} 个错误", @@ -307,7 +311,7 @@ "Archived": "已归档", "Archived at": "归档时间", "Archiving": "归档中", - "Archiving stops task scheduling and compresses the graph, LLM history, files, and exclusive assets and traffic into local cold storage. Once archived, a task can be restored from \"Archived\".": "归档会停止任务调度,将图谱、LLM 历史、文件以及独占资产和流量压缩到本地冷存储。归档完成后可从“已归档”中还原。", + "Archiving stops task scheduling and compresses the graph, LLM history, files, and exclusive assets and traffic into the configured archive storage. Once archived, a task can be restored from \"Archived\". Restoring requires free space on this server.": "归档会停止任务调度,将图谱、LLM 历史、文件以及独占资产和流量压缩到已配置的归档存储中。归档完成后可从“已归档”中还原。还原需要此服务器有足够的可用空间。", "Are you sure you want to delete asset interception rule \"{pattern}\"?": "确定删除资产拦截规则“{pattern}”?", "Arguments": "参数", "Arguments (space-separated)": "参数(空格分隔)", diff --git a/web/src/lib/types.ts b/web/src/lib/types.ts index 798b5a387..e8b23a205 100644 --- a/web/src/lib/types.ts +++ b/web/src/lib/types.ts @@ -101,6 +101,7 @@ export interface TaskArchiveTokenStats { } export interface TaskArchive { + storage?: "local" | "s3"; id: number; task_id: number; state: TaskArchiveState;