Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -436,6 +436,12 @@ catalog (`GET /v1/models`) and pick a model; models the provider adds later appe
next refresh. For example, [NotHoly](https://notholy.com) (offensive-security models) uses
base URL `https://api.notholy.com/v1` with an API key from its console.

**Task archive storage**: local disk by default. Set `ARTEX_ARCHIVE_STORAGE=s3` to
store cold task packages in an S3-compatible bucket and release their local
package space after verification. Existing local archives remain usable. See
[Task archive storage](docs/task-archive-storage.md) for configuration, permissions,
integrity checks, temporary disk requirements and recovery.

**Concurrency**: work agents per task, in "System Settings" (default 3).

**Untrusted-output boundary (on by default)**: tool results for the worker, planner, chat and main
Expand Down
183 changes: 183 additions & 0 deletions docs/task-archive-storage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,183 @@
# Task archive storage

Task archives use local disk by default. The optional S3-compatible backend moves
cold task packages to object storage while retaining Archive, Restore and Delete.
Active workspaces and PostgreSQL remain local. This is task cold storage, not a
full-installation backup; a recovery plan must preserve the database, data
directory, bucket objects, and the credentials or roles needed to read them.

## Configuration

Use a private bucket and a separate prefix for each ARTEX installation. Configure
the bucket's default encryption to match the selected mode below, and enforce
that mode/key with a bucket policy. ARTEX does not provision or inspect bucket
encryption settings.

Set the following server environment and restart ARTEX:

```dotenv
ARTEX_ARCHIVE_STORAGE=s3
ARTEX_ARCHIVE_S3_ENDPOINT=https://s3.us-east-1.amazonaws.com
ARTEX_ARCHIVE_S3_REGION=us-east-1
ARTEX_ARCHIVE_S3_BUCKET=your-artex-archives
ARTEX_ARCHIVE_S3_PREFIX=artex/tasks
ARTEX_ARCHIVE_S3_SSE=AES256
```

With an EC2 instance role, ECS task role or other SDK-supported credential source,
no archive-specific access keys are needed.

| Variable | Meaning |
| --- | --- |
| `ARTEX_ARCHIVE_STORAGE` | `local` (default) or `s3`. |
| `ARTEX_ARCHIVE_S3_ENDPOINT` | Required HTTPS origin, without bucket/path/query. Loopback HTTP is allowed for local tests. |
| `ARTEX_ARCHIVE_S3_REGION` | Signing region; defaults to `us-east-1`. |
| `ARTEX_ARCHIVE_S3_BUCKET` | Required bucket name. |
| `ARTEX_ARCHIVE_S3_PREFIX` | Object prefix; defaults to `artex/tasks`. |
| `ARTEX_ARCHIVE_S3_SSE` | `AES256` (default, SSE-S3) or `aws:kms` (SSE-KMS). There is no unencrypted mode. |
| `ARTEX_ARCHIVE_S3_KMS_KEY_ID` | Optional full KMS key ARN (not an alias or bare key ID); valid only with `aws:kms`. Omit to use the provider's managed KMS key. |
| `ARTEX_ARCHIVE_S3_ACCESS_KEY_ID` | Optional dedicated credential override; must be supplied with the secret key. |
| `ARTEX_ARCHIVE_S3_SECRET_ACCESS_KEY` | Secret for the dedicated override. |
| `ARTEX_ARCHIVE_S3_SESSION_TOKEN` | Optional session token for dedicated temporary credentials. |
| `ARTEX_ARCHIVE_S3_PATH_STYLE` | `true` for path-style servers; otherwise virtual-hosted addressing is used. |

An invalid mode or partial credential override fails configuration instead of
silently falling back to local storage or ambient credentials. Preserve the
bucket, endpoint, prefix and read/delete access while remote archives remain.
Older local archives remain usable; they are not uploaded automatically.

## Encryption and permissions

Every single-object upload and multipart initiation explicitly requests the
configured server-side encryption. Upload verification also requires HEAD to
report the requested mode and a nonempty KMS key identifier for SSE-KMS. When a
key ARN is configured, HEAD must return that exact ARN before ARTEX accepts
the object. Size and full SHA-256 readback remain mandatory; encryption metadata
does not replace content verification.

For SSE-KMS, set both the bucket's default encryption and ARTEX to `aws:kms`;
set `ARTEX_ARCHIVE_S3_KMS_KEY_ID` to the intended key when using a customer-managed
key. An explicit `AES256` request takes precedence over a bucket's KMS default,
so do not leave ARTEX on SSE-S3 when the bucket is intended to use KMS.

The principal needs object upload, read (HEAD and ranged GET), delete and
multipart upload/abort permissions. For versioned buckets, grant
`s3:DeleteObjectVersion` as well: failed verification deletes the version just
uploaded, rather than creating a delete marker. KMS uploads/readback also need the applicable
key-policy and IAM permissions for `kms:GenerateDataKey` and `kms:Decrypt`.
Retain old KMS keys and decrypt permissions when changing the key or mode; restore
does not require an older object to use the current upload policy.

Compatible providers must support the requested SSE mode and its HEAD metadata,
plus conditional ranged reads. A provider that ignores the encryption request
or cannot confirm it is rejected; the original task is retained. Local source and
temporary files keep their existing local-disk protection and restricted modes.

See the official [PutObject encryption and permission contract](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html)
and [HEAD encryption response rules](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html).

## Credential sources

Without the archive-specific override, ARTEX uses the AWS SDK default credential
chain, including standard AWS environment variables, shared profiles, web
identity, ECS task credentials and EC2 instance credentials through IMDS. Role
credentials retain the SDK's automatic refresh behavior. A named shared profile
can be selected with `AWS_PROFILE`.

For an external S3-compatible provider, dedicated keys can be supplied through
the archive-specific variables. Both key fields are required; a session token
is supported for temporary credentials. This explicit override takes precedence
over ambient AWS credentials. Operator-supplied temporary credentials must be
rotated by the operator; they are not an IAM role provider.

At server startup, the archive client captures its credential configuration before
ARTEX removes archive-specific keys and supported ambient AWS credential variables
from the process environment. Agent shells, tools and MCP stdio servers therefore
do not inherit those values. Role providers retain the configuration needed for
refresh. This environment cleanup does not restrict access to shared credential
files, token files or instance metadata; those require their own access controls.

Both credential modes load shared AWS configuration, including `AWS_CA_BUNDLE`
for private certificate authorities. HTTP redirects are rejected for all archive
requests, including 307/308; configure the final endpoint directly.

Keep secrets in a protected service environment file outside the repository and
frontend. A systemd `EnvironmentFile=` can point to a root-owned mode-0600 file.
See the [AWS SDK credential-chain documentation](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-gosdk.html).

## Lifecycle

1. Pause or finish a task and use its **Archive** action.
2. ARTEX stages the source files and builds a local `.tar.zst` package. Large
packages use multipart upload, initially 8 MiB parts with two upload workers;
the SDK increases the part size for exceptionally large seekable files.
3. ARTEX confirms encryption metadata and reads the complete object back,
checking size and SHA-256. ETags and uploaded checksum metadata alone are
insufficient.
4. **Only after verification and a successful PostgreSQL commit** does ARTEX
remove the local package and staged source files. The committed archive
location is an `s3://` URI; there is no schema or archive-format migration.
5. **Restore** downloads and verifies before installing files and database rows.
The remote package is consumed only after restoration commits.
6. **Delete** commits removal of the archive stub, then cleans up the object.
A durable journal retains failed cleanup work.

The archive list identifies **Object storage** or **Local storage** only once a
stored location exists; queued/uploading archives do not default to a local
label. Upload/download verification phases remain visible. Reads use bounded
8 MiB ranges pinned to the object's ETag. Each range has at most five HTTP
attempts total, including transient service errors and interrupted bodies;
SDK retries are disabled inside that loop. Each attempt has a two-minute limit
and cancellation interrupts both requests and retry delays.

## Interruption recovery

Source staging and restore journals live with task archives. Remote cleanup
journals live under `data/archives/tasks/.remote`; they contain locations, not
credentials. Keep this directory with PostgreSQL and the installation data.

| Interruption | Recovery |
| --- | --- |
| During staging/compression, before an upload journal exists | Roll staged source files back. Remove any temporary package, including a partially written `.partial` file, only after rollback succeeds. Keep the staging journal if cleanup fails so recovery can retry. The task remains in PostgreSQL. |
| After the upload journal, before successful verification | Roll source files back and clean up any uncommitted object/package. Failed cleanup retains its journal for retry. |
| After verification, before the database commit | Treat the object as uncommitted: retain the original task and clean up the orphan object. Verification alone never authorizes local deletion. |
| After the database commit, before package/staging cleanup | Keep the committed object and finish local cleanup. The persisted location and checksum establish commit, even if Restore or Delete has since been queued. |
| Download/extraction before restored files are installed | Remove temporary extraction data; keep the remote object and archive stub for another attempt. |
| Restored files installed, before database restoration commits | Roll back installed files using the restore journal. Retain the remote package. |
| Database restoration committed, before remote cleanup | Recognize the already-restored task and finish cleanup without replaying database rows. |
| Object consumed, before the restored archive stub is removed | Skip download/replay; repeat idempotent object deletion and remove the stub. |
| Delete journal written, before database deletion commits | Keep the object while its archive stub exists. The queued Delete operation can retry. |
| Database deletion committed, before object deletion succeeds | The durable outbox retains the URI and retries object deletion. |
| An old failed upload survives a retry using local storage | Remove the abandoned remote object, but retain a local package referenced by the committed archive row. |

The staging recovery rule also applies to local archives. Dedicated tests cover
both storage modes with ready, restore-queued, delete-queued and uncommitted
states. Additional PostgreSQL/HTTP-S3 tests reconstruct interrupted operations at
14 persisted boundaries and repeat recovery to check idempotence.

Handled multipart-upload failures, including cancellation, trigger an abort with
a separate 30-second cleanup context. A failed encryption/content verification
similarly deletes the newly uploaded object (its exact version when available)
while retaining the source. Cleanup failures are returned together with the
original error. Abrupt process death may still leave unfinished multipart uploads
or object versions; configure bucket lifecycle expiration for incomplete uploads
and a version-retention policy. The URI-based recovery journal does not retain
version IDs, so lifecycle policy remains necessary after a crash or failed
version cleanup.

## Disk and transfer requirements

Archive creation still requires temporary local space for the compressed
package. Restore requires space for the download and extracted files. Full
upload readback and restores incur storage reads and applicable transfer charges.
Failed remote cleanup does not prevent unrelated local archive jobs from being
processed once the failed cleanup request returns.

## Optional connectivity check

Run `go run ./cmd/artex-archive-check` with the archive environment to upload,
verify, download and delete a small unique object. It performs real bucket
operations but does not touch task data, and reports cleanup failures.

Automated tests use HTTP fixtures for S3 and metadata credentials, plus isolated
PostgreSQL. They do not establish compatibility with a live cloud bucket.
13 changes: 11 additions & 2 deletions web/src/app/(main)/function/tasks/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -1555,7 +1555,7 @@ function ArchiveConfirmDialog({
<AlertDialogTitle>{count === 1 ? t("Archive task") : t("Archive {count} tasks", { count })}</AlertDialogTitle>
<AlertDialogDescription className="[overflow-wrap:anywhere]">
{t(
"Archiving stops task scheduling and compresses the graph, LLM history, files, and exclusive assets and traffic into local cold storage. Once archived, a task can be restored from \"Archived\".",
"Archiving stops task scheduling and compresses the graph, LLM history, files, and exclusive assets and traffic into the configured archive storage. Once archived, a task can be restored from \"Archived\". Restoring requires free space on this server.",
)}
</AlertDialogDescription>
</AlertDialogHeader>
Expand Down Expand Up @@ -1999,6 +1999,11 @@ function TaskArchivesPanel({ onChanged }: { onChanged: () => void }) {
title={t("Before compression: {size}", { size: formatArchiveBytes(archive.original_size) })}
>
{archiveCompressionLabel(archive)}
{(archive.storage === "s3" || archive.storage === "local") && (
<div className="text-muted-foreground">
{archive.storage === "s3" ? t("Object storage") : t("Local storage")}
</div>
)}
</TableCell>
<TableCell className="text-xs tabular-nums">{archiveDataTotal(archive).toLocaleString(intlLocale)}</TableCell>
<TableCell>
Expand All @@ -2017,7 +2022,11 @@ function TaskArchivesPanel({ onChanged }: { onChanged: () => void }) {
</p>
))}
<span className="text-muted-foreground text-xs">
{t(archive.phase)} · {archive.progress}%
{archive.phase === "upload_and_verify"
? t("Uploading and verifying")
: archive.phase === "download_and_verify"
? t("Downloading and verifying")
: t(archive.phase)} · {archive.progress}%
</span>
</div>
</TableCell>
Expand Down
6 changes: 5 additions & 1 deletion web/src/i18n/locales/tr.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@
"-1 = no rerun; the intent is marked blocked outright": "-1 = yeniden çalıştırma yok; niyet doğrudan engellendi olarak işaretlenir",
"-1 = no retries at all; the failure bubbles up immediately": "-1 = hiç yeniden deneme yok; hata hemen yukarı iletilir",
"-1 = transient failures never trip (deterministic failures still do)": "-1 = geçici hatalar asla devre kesmez (kesin hatalar yine keser)",
"Downloading and verifying": "İndiriliyor ve doğrulanıyor",
"Local storage": "Yerel depolama",
"Object storage": "Nesne depolama",
"Uploading and verifying": "Karşıya yükleniyor ve doğrulanıyor",
"— (task deleted)": "— (görev silinmiş)",
"— agent generates —": "— ajan oluşturur —",
"… {count} errors in total": "… toplam {count} hata",
Expand Down Expand Up @@ -307,7 +311,7 @@
"Archived": "Arşivlenenler",
"Archived at": "Arşivlenme zamanı",
"Archiving": "Arşivleniyor",
"Archiving stops task scheduling and compresses the graph, LLM history, files, and exclusive assets and traffic into local cold storage. Once archived, a task can be restored from \"Archived\".": "Arşivleme, görev zamanlamasını durdurur ve grafiği, LLM geçmişini, dosyaları, göreve özel varlıkları ve trafiği yerel soğuk depolamaya sıkıştırır. Arşivlenen görev \"Arşivlenenler\" sekmesinden geri yüklenebilir.",
"Archiving stops task scheduling and compresses the graph, LLM history, files, and exclusive assets and traffic into the configured archive storage. Once archived, a task can be restored from \"Archived\". Restoring requires free space on this server.": "Arşivleme, görev zamanlamasını durdurur ve grafiği, LLM geçmişini, dosyaları, özel varlıkları ve trafiği yapılandırılmış arşiv depolama alanına sıkıştırır. Arşivlenen bir görev, \"Arşivlenenler\" bölümünden geri yüklenebilir. Geri yükleme, bu sunucuda boş alan gerektirir.",
"Are you sure you want to delete asset interception rule \"{pattern}\"?": "\"{pattern}\" varlık yakalama kuralını silmek istediğinizden emin misiniz?",
"Arguments": "Argümanlar",
"Arguments (space-separated)": "Argümanlar (boşlukla ayrılmış)",
Expand Down
6 changes: 5 additions & 1 deletion web/src/i18n/locales/zh-CN.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@
"-1 = no rerun; the intent is marked blocked outright": "-1 = 不重跑,该意图直接判为 blocked",
"-1 = no retries at all; the failure bubbles up immediately": "-1 = 一次都不重试,失败立刻上抛",
"-1 = transient failures never trip (deterministic failures still do)": "-1 = 瞬时失败永不熔断(确定性失败仍然熔断)",
"Downloading and verifying": "正在下载并校验",
"Local storage": "本地存储",
"Object storage": "对象存储",
"Uploading and verifying": "正在上传并校验",
"— (task deleted)": "—(任务已删除)",
"— agent generates —": "— 由智能体生成 —",
"… {count} errors in total": "…共 {count} 个错误",
Expand Down Expand Up @@ -307,7 +311,7 @@
"Archived": "已归档",
"Archived at": "归档时间",
"Archiving": "归档中",
"Archiving stops task scheduling and compresses the graph, LLM history, files, and exclusive assets and traffic into local cold storage. Once archived, a task can be restored from \"Archived\".": "归档会停止任务调度,将图谱、LLM 历史、文件以及独占资产和流量压缩到本地冷存储。归档完成后可从“已归档”中还原。",
"Archiving stops task scheduling and compresses the graph, LLM history, files, and exclusive assets and traffic into the configured archive storage. Once archived, a task can be restored from \"Archived\". Restoring requires free space on this server.": "归档会停止任务调度,将图谱、LLM 历史、文件以及独占资产和流量压缩到已配置的归档存储中。归档完成后可从“已归档”中还原。还原需要此服务器有足够的可用空间。",
"Are you sure you want to delete asset interception rule \"{pattern}\"?": "确定删除资产拦截规则“{pattern}”?",
"Arguments": "参数",
"Arguments (space-separated)": "参数(空格分隔)",
Expand Down
1 change: 1 addition & 0 deletions web/src/lib/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ export interface TaskArchiveTokenStats {
}

export interface TaskArchive {
storage?: "local" | "s3";
id: number;
task_id: number;
state: TaskArchiveState;
Expand Down
Loading