Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 116 additions & 0 deletions README.md
Comment thread
freedom3219 marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,124 @@ storify put -R local/dir remote/dir

# Copy within storage
storify cp source/path dest/path
# Move/rename within storage
storify mv source/path dest/path

# Create directories
storify mkdir path/to/dir
storify mkdir -p path/to/nested/dir # create parents

# Display file contents
storify cat path/to/file

# Display beginning of file
storify head path/to/file # first 10 lines (default)
storify head -n 20 path/to/file # first 20 lines
storify head -c 1024 path/to/file # first 1024 bytes
storify head -q file1 file2 # suppress headers

# Display end of file
storify tail path/to/file # last 10 lines (default)
storify tail -n 20 path/to/file # last 20 lines
storify tail -c 1024 path/to/file # last 1024 bytes
storify tail -v path/to/file # always show header

# Search for patterns
storify grep "pattern" path/to/file # basic search
storify grep -i "pattern" path/to/file # case-insensitive
storify grep -n "pattern" path/to/file # show line numbers
storify grep -R "pattern" path/ # recursive

# Diff two files (unified diff)
storify diff left/file right/file # unified diff with 3 lines context
storify diff -U 1 left/file right/file # set context lines
storify diff -w left/file right/file # ignore trailing whitespace
storify diff --size-limit 1 -f left right # size guard and force

# Find objects by name/regex/type
storify find path/ --name '**/*.log' # glob on full path
storify find path/ --regex '.*\\.(csv|parquet)$' # regex on full path
storify find path/ --type f # filter by type: f|d|o

# Show directory structure as a tree
storify tree path/to/dir # show full tree
storify tree path/to/dir -d 1 # limit depth to 1
storify tree path/to/dir --dirs-only # show directories only

# Show disk usage
storify du path/to/dir
storify du path/to/dir -s # summary only

# Delete files/directories
storify rm path/to/file
storify rm path/to/dir -R # recursive
storify rm path/to/dir -Rf # recursive + force (no confirmation)

# Show object metadata
storify stat path/to/file # human-readable
storify stat path/to/file --json # JSON output
storify stat path/to/file --raw # raw key=value format

# Create files (touch)
storify touch path/to/file # create if missing; no change if exists
storify touch -t path/to/file # truncate to 0 bytes if exists
storify touch -c path/to/missing # do not create; succeed silently
storify touch -p path/to/nested/file # create parents when applicable

# Append data to file
storify append local.txt path/to/file # alias: local first, dest second
storify append path/to/file --src local.txt # append local file to remote file (canonical)
echo "line" | storify append path/to/file --stdin # append from stdin
storify append path/to/file --src local.txt -c # fail if missing (no-create)
```

## Command Reference

### Storage Commands

| Command | Description | Options |
|---------|-------------|---------|
| `ls` | List directory contents | `-L` (detailed), `-R` (recursive) |
| `get` | Download files from remote |
| `put` | Upload files to remote | `-R` (recursive) |
| `cp` | Copy files within storage |
| `mv` | Move/rename files within storage |
| `mkdir` | Create directories | `-p` (create parents) |
| `touch` | Create files |
| `append` | Append data to a remote file | `--src <PATH>` or `--stdin`, `-c` (no-create), `-p` (parents) |
| `cat` | Display file contents |
| `head` | Display beginning of file | `-n` (lines), `-c` (bytes), `-q` (quiet), `-v` (verbose) |
| `tail` | Display end of file | `-n` (lines), `-c` (bytes), `-q` (quiet), `-v` (verbose) |
| `grep` | Search for patterns in files | `-i` (case-insensitive), `-n` (line numbers) ,`-R` (recursive) |
| `find` | Find objects by name/regex/type | `--name <GLOB>`, `--regex <RE>`, `--type <f|d|o>` |
| `rm` | Delete files/directories | `-R` (recursive), `-f` (force) |
| `tree` | View directory structure as a tree | `-d <DEPTH>`, `--dirs-only` |
| `du` | Show disk usage | `-s` (summary) |
| `stat` | Show object metadata | `--json`, `--raw` |
| `diff` | Compare two files (unified diff) | `-U <N>` (context), `-w` (ignore-space), `--size-limit <MB>`, `-f` (force) |

### Config Commands

| Command | Description | Options |
|---------|-------------|---------|
| `config create` | Create/update profile | Provider-specific flags, `--anonymous`, `--make-default`, `--force` |
| `config list` | List all profiles | `--show-secrets` |
| `config show` | Show configuration | `--profile <NAME>`, `--default`, `--show-secrets` |
| `config set` | Set/clear default profile | `<NAME>` or `--clear` |
| `config delete` | Delete a profile | `<NAME>`, `--force` |

## Supported Providers

| Provider | Type | Anonymous Support |
|----------|------|-------------------|
| **OSS** | Alibaba Cloud Object Storage | ✅ Yes |
| **S3** | Amazon S3 | ✅ Yes |
| **MinIO** | Self-hosted S3-compatible | ✅ Yes |
| **COS** | Tencent Cloud Object Storage | ❌ No |
| **FS** | Local Filesystem | ✅ Yes (always) |
| **HDFS** | Hadoop Distributed File System | ❌ No |
| **Azblob** | Azure Cloud Object Storage | ❌ No |

More commands: see [`docs/usage.md`](docs/usage.md).

## Documentation
Expand Down
7 changes: 7 additions & 0 deletions docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,13 @@ This page lists the common Storify CLI commands with short, copy-pastable exampl
- Ignore trailing whitespace: `storify diff -w left/file right/file`
- Guard against large files and force: `storify diff --size-limit 1 -f left right`

## Append
- Append a local file: `storify append remote/path --src ./local.txt`
- Append via stdin: `echo "line" | storify append remote/path --stdin`
- Alias form (local first, remote second): `storify append ./local.txt remote/path`
- Require existing file: add `-c/--no-create`
- Auto-create parent directories (filesystem providers): add `-p/--parents`

## Options cheat sheet
- `-R`: recursive (works with `ls`, `put`, `rm`, `find`)
- `-L`: long/detailed listing
Expand Down
6 changes: 4 additions & 2 deletions src/cli/entry.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ use super::{
context::CliContext,
prompts::Prompt,
storage::{
self, CatArgs, CpArgs, DiffArgs, DuArgs, GetArgs, GrepArgs, HeadArgs, LsArgs, MkdirArgs,
MvArgs, PutArgs, RmArgs, StatArgs, TailArgs, TouchArgs, TreeArgs,
self, AppendArgs, CatArgs, CpArgs, DiffArgs, DuArgs, GetArgs, GrepArgs, HeadArgs, LsArgs,
MkdirArgs, MvArgs, PutArgs, RmArgs, StatArgs, TailArgs, TouchArgs, TreeArgs,
},
};

Expand Down Expand Up @@ -91,6 +91,8 @@ pub enum Command {
Diff(DiffArgs),
/// Create empty files or update metadata (best-effort)
Touch(TouchArgs),
/// Append data to a remote file
Append(AppendArgs),
}

#[derive(Subcommand, Debug, Clone)]
Expand Down
2 changes: 2 additions & 0 deletions src/cli/prompts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,8 @@ impl Prompt {
}
}

// Default is auto-derived above.

fn join_error(err: task::JoinError) -> Error {
Error::Io {
source: io::Error::other(err.to_string()),
Expand Down
78 changes: 78 additions & 0 deletions src/cli/storage.rs
Original file line number Diff line number Diff line change
Expand Up @@ -296,6 +296,36 @@ pub struct TouchArgs {
pub parents: bool,
}

#[derive(ClapArgs, Debug, Clone)]
#[command(group = clap::ArgGroup::new("stdin_or_src").args(["stdin", "src"]).multiple(false))]
pub struct AppendArgs {
/// Remote destination file path
#[arg(value_name = "DEST", value_parser = parse_validated_path)]
pub dest: String,

/// Optional positional when using the alias form: `append <SRC> <DEST>`
/// When --stdin/--src are both absent and this is provided, CLI treats:
/// dest (first positional) as local SRC, and alt_dest as remote DEST.
#[arg(value_name = "ALT_DEST", required = false, value_parser = parse_validated_path)]
pub alt_dest: Option<String>,

/// Local source file to append (mutually exclusive with --stdin)
#[arg(long = "src", value_name = "SRC")]
pub src: Option<String>,

/// Read content from standard input
#[arg(long)]
pub stdin: bool,

/// Do not create the file if it doesn't exist
#[arg(short = 'c', long = "no-create")]
pub no_create: bool,

/// Create parent directories when needed (filesystem providers)
#[arg(short = 'p', long = "parents")]
pub parents: bool,
}

pub async fn execute(command: &Command, ctx: &CliContext) -> Result<()> {
let config = ctx.storage_config()?;
let client = StorageClient::new(config.clone()).await?;
Expand Down Expand Up @@ -452,6 +482,54 @@ pub async fn execute(command: &Command, ctx: &CliContext) -> Result<()> {
)
.await?;
}
Command::Append(append_args) => {
if append_args.stdin {
if append_args.alt_dest.is_some() {
return Err(Error::InvalidArgument {
message: "Too many positionals: use `append <DEST> --stdin`".to_string(),
});
}
let opts = crate::storage::AppendOptions {
no_create: append_args.no_create,
parents: append_args.parents,
};
client.append_from_stdin(&append_args.dest, opts).await?;
} else {
// Two supported forms (mutually exclusive with --stdin):
// 1) append <DEST> --src <SRC> (canonical)
// 2) append <SRC> <DEST> (alias)
if let Some(src_flag) = append_args.src.as_ref() {
if append_args.alt_dest.is_some() {
return Err(Error::InvalidArgument {
message: "Too many positionals: use `append <DEST> --src <SRC>`"
.to_string(),
});
}
let opts = crate::storage::AppendOptions {
no_create: append_args.no_create,
parents: append_args.parents,
};
client
.append_from_local(src_flag, &append_args.dest, opts)
.await?;
} else if let Some(dest2) = append_args.alt_dest.as_ref() {
// Alias: first positional is LOCAL, second is DEST
let local_src = &append_args.dest;
let remote_dest = dest2;
let opts = crate::storage::AppendOptions {
no_create: append_args.no_create,
parents: append_args.parents,
};
client
.append_from_local(local_src, remote_dest, opts)
.await?;
} else {
return Err(Error::InvalidArgument {
message: "missing SRC: use `append <DEST> --src <SRC>` or alias `append <SRC> <DEST>`".to_string(),
});
}
}
}
Command::Config(_) => {
unreachable!("Config commands are handled separately")
}
Expand Down
3 changes: 3 additions & 0 deletions src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,9 @@ pub enum Error {
#[snafu(display("Failed to touch '{path}': {source}"))]
TouchFailed { path: String, source: Box<Error> },

#[snafu(display("Failed to append to '{path}': {source}"))]
AppendFailed { path: String, source: Box<Error> },

#[snafu(display("Invalid argument: {message}"))]
InvalidArgument { message: String },

Expand Down
48 changes: 46 additions & 2 deletions src/storage.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,10 @@ use opendal::Operator;
pub mod constants;
mod operations;
mod utils;
pub use self::operations::AppendOptions;
pub use self::utils::OutputFormat;

use self::operations::append::OpenDalAppender;
use self::operations::cat::OpenDalFileReader;
use self::operations::copy::OpenDalCopier;
use self::operations::delete::OpenDalDeleter;
Expand All @@ -25,8 +27,8 @@ use self::operations::tree::OpenDalTreer;
use self::operations::upload::OpenDalUploader;
use self::operations::usage::OpenDalUsageCalculator;
use self::operations::{
Cater, Copier, Deleter, Differ, Downloader, Greper, Header, Lister, Mkdirer, Mover, Stater,
Tailer, Toucher, Treer, Uploader, UsageCalculator,
Appender, Cater, Copier, Deleter, Differ, Downloader, Greper, Header, Lister, Mkdirer, Mover,
Stater, Tailer, Toucher, Treer, Uploader, UsageCalculator,
};
use crate::storage::utils::error::IntoStorifyError;
use crate::wrap_err;
Expand Down Expand Up @@ -791,4 +793,46 @@ impl StorageClient {
.try_for_each(|_| async { Ok(()) })
.await
}

pub async fn append_from_local(
&self,
local_path: &str,
remote_path: &str,
opts: AppendOptions,
) -> Result<()> {
log::debug!(
"append_from_local provider={:?} local_path={} remote_path={} no_create={} parents={}",
self.provider,
local_path,
remote_path,
opts.no_create,
opts.parents
);
let appender = OpenDalAppender::new(self.operator.clone());
wrap_err!(
appender
.append_from_local(local_path, remote_path, &opts)
.await,
AppendFailed {
path: remote_path.to_string()
}
)
}

pub async fn append_from_stdin(&self, remote_path: &str, opts: AppendOptions) -> Result<()> {
log::debug!(
"append_from_stdin provider={:?} remote_path={} no_create={} parents={}",
self.provider,
remote_path,
opts.no_create,
opts.parents
);
let appender = OpenDalAppender::new(self.operator.clone());
wrap_err!(
appender.append_from_stdin(remote_path, &opts).await,
AppendFailed {
path: remote_path.to_string()
}
)
}
}
Loading