diff --git a/README.md b/README.md index bc111bc6..94e554d8 100644 --- a/README.md +++ b/README.md @@ -11,159 +11,179 @@ [![Arius.Core Version](https://img.shields.io/nuget/v/WouterVanRanst.Arius.Core?logo=nuget)](https://www.nuget.org/packages/WouterVanRanst.Arius.Core) [![ClickOnce](https://img.shields.io/badge/Windows-ClickOnce-dsfs?logo=windows&logoColor=lightblue)](https://woutervanranst.github.io/Arius/Arius.Explorer.application) -Arius is a lightweight archival solution, specifically built to leverage the Azure Blob Archive tier. - -The name derives from the Greek for 'immortal'. - -## Why? - -1. **Supporting 3-2-1 Backup Strategy**: Arius offers a secure and cost-effective offsite backup solution, complementing offline disk backups. The use of Azure Blob Archive tier allows for low-cost archiving, approximately 1 EUR per TB per month. - -1. **Single Pane of Glass**: Arius ensures seamless visibility to offline backups without requiring a separate application. By creating small "pointers" locally, they remain present in the local filesystem and easily discoverable in Windows Explorer. - -1. **Encryption**: For those with privacy concerns, Arius uses AES256/openssl-compatible encryption, providing robust client-side encryption. - -1. **Deduplication**: Arius does not store duplicate files or file parts. It offers file-level deduplication by default and allows optional variable block size deduplication within files. +Arius is a cross-platform archival solution that keeps cold backups affordable without giving up visibility into your files. It encrypts and deduplicates data locally, stores the content in Azure Blob Storage (Archive/Cool/Cold/Hot tiers), and leaves *pointer* files behind so your backup still looks like a regular folder structure. + +--- + +- [Key features](#key-features) +- [Quick start](#quick-start) + - [Prerequisites](#prerequisites) + - [Get the CLI](#get-the-cli) + - [Configure Azure Storage](#configure-azure-storage) + - [Archive data](#archive-data) + - [Restore data](#restore-data) +- [Command reference](#command-reference) + - [Archive](#archive) + - [Restore](#restore) + - [Running in Docker](#running-in-docker) +- [Arius Explorer (Windows UI)](#arius-explorer-windows-ui) +- [Documentation](#documentation) +- [Manual disaster recovery](#manual-disaster-recovery) +- [Architecture](#architecture) + +## Key features + +* **3-2-1 friendly off-site backups** – Archive cold data into low-cost Azure Blob tiers while keeping your production storage clean. +* **Client-side encryption** – AES-256 encryption happens before anything leaves your machine; only encrypted blobs reach Azure. +* **Chunk-level deduplication** – Identical content is stored only once, even when filenames or locations differ. +* **Pointer-aware file system integration** – Tiny `.arius.pointer` files keep your directory tree browsable and searchable. +* **Automatable CLI & Docker support** – Run Arius interactively, on a schedule, or in containers as part of your backup pipeline. ![](docs/overview.png) -## How +## Quick start -Arius is a command-line tool (CLI) that can be run manually or scheduled. It can also be run as a Docker container. +### Prerequisites -Arius Explorer is a Windows application that offers a graphical user interface into an Arius repository. +1. [.NET SDK 9.0 or later](https://dotnet.microsoft.com/download) (for running or building the CLI locally). +2. An Azure Storage account with [hierarchical namespace disabled](https://learn.microsoft.com/azure/storage/blobs/data-lake-storage-namespace) and the Archive tier available. +3. A secure passphrase that will be used to encrypt and decrypt your blobs. -### Arius Explorer +### Get the CLI -![](docs/arius.explorer.png) +* **Download a build:** Pre-built binaries for Linux, macOS, and Windows are published on the [GitHub Releases](https://github.com/woutervanranst/Arius/releases) page. +* **Build from source:** + ```bash + git clone https://github.com/woutervanranst/Arius.git + cd Arius + dotnet publish src/Arius.Cli -c Release -r linux-x64 --self-contained false -o ./publish + ./publish/arius --version + ``` + Adjust `-r` for your OS (`win-x64`, `osx-arm64`, ...). Replace `--self-contained false` with `true` if you want a single-file build that includes the runtime. -Install it via ClickOnce by clicking on the shield at the top of the page. +### Configure Azure Storage -The status icon is a combination of the state in the local file system (left half moon) and the state in the repository (right half moon). For example: +1. Create (or reuse) a blob container that will store Arius data. +2. Generate a storage account access key (Key 1 or Key 2 in the Azure Portal). +3. Optionally set environment variables so you don't have to pass credentials on every run: + ```bash + export ARIUS_ACCOUNT_NAME= + export ARIUS_ACCOUNT_KEY= + ``` -| Icon | Local file systen | Azure Repository | -| --- |---|---| -| ![](docs/status/NNYC.png) | Not present | Pointer and Binary | -| ![](docs/status/NYYC.png) | Only the Pointer | Pointer and Binary | -| ![](docs/status/YYYC.png) | Pointer and Binary | Pointer and Binary | +### Archive data +```bash +arius archive /path/to/data \ + --accountname \ + --accountkey \ + --container \ + --passphrase +``` -### Arius CLI +During the run Arius will: -#### Archive to Azure +* hash and deduplicate data +* compress and encrypt before upload +* write progress updates to the console +* leave `.arius.pointer` files in place of archived files (unless `--remove-local` is specified) -CLI: +Logs are written to `%LOCALAPPDATA%/Arius/logs` on Windows and `~/.local/share/Arius/logs` on Linux/macOS. When running in Docker the logs are written to `/logs`. - +### Restore data -``` -arius archive - --accountname - --accountkey - --passphrase - --container - [--remove-local] - [--tier=] - [--dedup] - [--fasthash] +```bash +arius restore --root /restore/location "Photos/2020" "Documents/report.docx" \ + --accountname \ + --accountkey \ + --container \ + --passphrase \ + --download --include-pointers ``` -CLI in Docker: - -``` -docker run - -v :/archive - [-v :/logs] - woutervanranst/arius - - archive - --accountname - --accountkey - --passphrase - --container - [--remove-local] - [--tier=] - [--dedup] - [--fasthash] +* `--download` hydrates blobs from Azure and writes the binary content locally. +* `--include-pointers` recreates pointer files next to restored binaries so the folder stays "Arius-aware". +* Omitting `--download` performs a *sync-only* run that updates/creates pointer files without pulling full data. + +## Command reference + +### Archive + +| Parameter | Description | Notes | +| --- | --- | --- | +| `` | Directory to archive. | A pointer file replaces each archived file unless `--remove-local` is used. | +| `--accountname`, `-n` | Azure Storage account name. | Can also be supplied via `ARIUS_ACCOUNT_NAME`. | +| `--accountkey`, `-k` | Azure Storage account key. | Can also be supplied via `ARIUS_ACCOUNT_KEY`. | +| `--container`, `-c` | Blob container that will hold chunks, pointers, and state. | Required. | +| `--passphrase`, `-p` | Encryption passphrase. | Required – keep it safe. | +| `--tier` | Storage tier for uploaded chunks. | Defaults to `archive`. Supported values: `hot`, `cool`, `cold`, `archive`. | +| `--remove-local` | Delete local binaries after successful upload. | Pointer files remain so the folder stays browsable. | + +### Restore + +| Parameter | Description | Notes | +| --- | --- | --- | +| `--root`, `-r` | Local root folder to use for restores. | Defaults to the current working directory. Required when running from outside the target folder. | +| `Targets` | Optional paths (files or directories) under the root to restore. | Defaults to `./` (entire repository). | +| `--accountname`, `-n` | Azure Storage account name. | Supports `ARIUS_ACCOUNT_NAME`. | +| `--accountkey`, `-k` | Azure Storage account key. | Supports `ARIUS_ACCOUNT_KEY`. | +| `--container`, `-c` | Azure Blob container to read from. | Required. | +| `--passphrase`, `-p` | Passphrase used during archive. | Required. | +| `--download` | Retrieve the binary content. | Without this flag the command only syncs pointer files. | +| `--include-pointers` | Keep or recreate pointer files alongside restored binaries. | Recommended for ongoing Arius use. | + +### Running in Docker + +The official container image (`woutervanranst/arius`) contains the CLI and expects bind mounts for both your data and optional logs. + +```bash +docker run --rm \ + -v /absolute/path/to/data:/archive \ + -v /absolute/path/to/logs:/logs \ + woutervanranst/arius archive \ + --accountname \ + --accountkey \ + --container \ + --passphrase ``` -### Restore from Azure +For restore runs, replace `archive` with `restore` and add `--download` / `--include-pointers` as needed. The container always works inside `/archive`, so you do not pass a positional path parameter. -CLI: +## Arius Explorer (Windows UI) -``` -arius restore - --accountname - --accountkey - --passphrase - --container - [--synchronize] - [--download] - [--keep-pointers] -``` - -CLI in Docker: +![](docs/arius.explorer.png) -``` -docker run - -v :/archive - [-v :/logs] - woutervanranst/arius - - restore - --accountname - --accountkey - --passphrase - --container - [--synchronize] - [--download] - [--keep-pointers] -``` +Arius Explorer provides a Windows desktop experience on top of the same repository: -### Arguments +1. Install it via the [ClickOnce installer](https://woutervanranst.github.io/Arius/Arius.Explorer.application). +2. Sign in with the same storage account, container, and passphrase that you use for the CLI. +3. Browse pointer status – icons combine local state (left half) and cloud state (right half). -| Argument | Description | Notes | -| - | - | - | -| path | The path on the local file system | For `archive`:
The root directory to archive

For `restore`:
  • If path is a directory: restore all pointer files in the (sub)directories.
  • If path is a file: restore this file.
-| logpath | Path to the folder to store the logs | OPTIONAL. NOTE: Only for Docker. -| ‑‑accountname, ‑n | Storage Account Name -| ‑‑accountkey, ‑k | [Storage Account Key](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal) | Can be set through:
  • Argument
  • Environment variable `ARIUS_ACCOUNT_KEY`
  • Docker environment variable `ARIUS_ACCOUNT_KEY`
-| ‑‑passphrase, ‑p | Passphrase with which the blobs are encrypted -| ‑‑container, ‑c | Blob container to use | OPTIONAL. Default: 'arius'. -| ‑‑remove-local | Remove local file after a successful upload | `archive`-only
OPTIONAL. Default: Local files are not deleted after archiving. -| ‑‑tier | [Blob storage tier (hot/cool/archive)](https://docs.microsoft.com/en-us/azure/storage/blobs/access-tiers-overview) | `archive`-only
OPTIONAL. Default: 'archive'. -| ‑‑dedup | Deduplicate on block level | `archive`-only
OPTIONAL. Default: deduplicate on file level. -| ‑‑fasthash | When a pointer file is present, use that hash instead of re-hashing the full file again | `archive`-only
OPTIONAL. Default: false.
NOTE: Do **NOT** use this if the contents of the files are modified. Arius will not pick up the changes. -| ‑‑synchronize | Bring the structure of the local file system (pointer files) in line with the latest state of the remote repository | `restore`-only
OPTIONAL. Default: do not synchronize.
This command only touches the pointers (ie. `.pointer.arius` files). Other files are left untouched:
  • Pointers that exist in the archive but not locally are created.
  • Pointers that exist locally but not in the archive are deleted
-| ‑‑download | Download and restore the actual file (contents) | `restore`-only
OPTIONAL. Default: do not download.
NOTE: If the file is in the archive blob tier, hydration to an online tier is started. Run the restore command again after ~15 hours to download the file. -| ‑‑keep-pointers | Keep pointer files after downloading content files | `restore`-only
OPTIONAL. Default: keep the pointers. +| Icon | Local file system | Azure repository | +| --- | --- | --- | +| ![](docs/status/NNYC.png) | Not present | Pointer and binary | +| ![](docs/status/NYYC.png) | Pointer only | Pointer and binary | +| ![](docs/status/YYYC.png) | Pointer and binary | Pointer and binary | -### Restoring manually +## Documentation -For detailed instructions on restoring files manually, _without_ using any of the Arius tools, please consult the [Restore Manually](docs/manualrestore.md) section. +Additional technical documentation lives in the [`docs/`](docs) folder: -## Advanced +* [Archive command deep dive](docs/ArchiveCommand.md) +* [Storage abstractions](docs/Storage.md) +* [Versioning model](docs/versioning.md) +* [State repository internals](docs/StateRepository.md) -### Deduplication +## Manual disaster recovery -Arius employs deduplication to optimize storage efficiency. For detailed explanations, diagrams, and benchmarks on deduplication, refer to the [Deduplication](docs/deduplication.md) section in the documentation. +Need to recover data without Arius? Follow the step-by-step [manual restore guide](docs/manualrestore.md) to decrypt blobs and recreate files directly from Azure Storage Explorer. -## Technical Documentation +## Architecture ### Dependencies ![](http://www.plantuml.com/plantuml/proxy?cache=no&src=https://raw.githubusercontent.com/woutervanranst/Arius/main/docs/dependencies.puml) -### Arius.Core Architecture +### Arius.Core architecture ![](docs/AriusFlows-Arius.Core%20Structure.drawio.svg) - -### Arius.Core Domain Model - -![](http://www.plantuml.com/plantuml/proxy?cache=no&src=https://raw.githubusercontent.com/woutervanranst/Arius/main/docs/classdiagram.puml) - -# Attributions - -Arius Icon by [Freepik](https://www.flaticon.com/free-icon/iceberg_2055379?related_id=2055379). - -Many thanks to [ReSharper](https://www.jetbrains.com/community/opensource/#support) and [PostSharp](https://www.postsharp.net/metalama/pricing) for their open source generosity.