Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 22 additions & 5 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ The CLI owns:

- command-line argument parsing;
- reading an input file;
- applying CLI-specific encoded-input size policy before parsing;
- selecting text or JSON output;
- rendering diagnostics;
- mapping failures to process exit codes.
Expand Down Expand Up @@ -291,7 +292,9 @@ Benefits:
- result types do not need lifetimes tied to the source slice;
- future WebAssembly integration is easier.

A malicious input can still contain a very large number of real frame/block headers, so metadata counts are a resource consideration. The v0.1 design must leave room for future configurable limits without changing the fundamental result model.
A malicious input can still contain a very large number of real frame/block headers, so metadata counts are a resource consideration. `inspect_with_limits()` addresses that concern with configurable frame/block count budgets without changing the fundamental result model.

The public library APIs still require the caller to own the complete input slice. Metadata-count limits therefore do not bound the caller's input buffer. File-size policy belongs to the file-reading application layer: the CLI applies a separate default 256 MiB encoded-input guard, while an accepted input remains fully resident in memory during inspection.

## 9. Error architecture

Expand All @@ -310,6 +313,8 @@ Human-readable `Display` messages are important, but callers must be able to mat

Empty input and trailing partial top-level magic may use the same typed EOF/truncation error category as other incomplete structures.

CLI-only resource policy failures, such as exceeding the configured encoded-input byte limit, remain structured CLI errors rather than `ZstdError` variants because they are not Zstandard format errors.

## 10. Validation policy

ZstdScope v0.1 is strict about structural rules it understands.
Expand Down Expand Up @@ -345,15 +350,17 @@ The CLI enables serialization support for `--json`. JSON field names and seriali

## 12. CLI architecture

Initial command:
Current command:

```text
zstdscope inspect <FILE> [--json]
zstdscope inspect <FILE> [--json] [--max-input-bytes <BYTES>]
```

v0.1 may read the file into memory and pass a slice to the library. This is simple and consistent with the accepted initial API, but it is a known limitation for very large files.
The CLI keeps parsing in the library. It opens the input file, rejects files larger than the configured byte limit before the full read when file metadata already proves the limit would be exceeded, and also caps the actual read at one byte beyond the limit so file growth cannot silently bypass the configured boundary. The default limit is 268,435,456 bytes (256 MiB); callers can deliberately raise or lower it with `--max-input-bytes`.

A streaming/file-backed inspection API is a later milestone and must not be simulated inside the CLI with duplicate parsing code.
After the bounded read succeeds, the CLI passes the complete byte slice to the existing public `inspect(&[u8])` API. The CLI does not contain an independent Zstandard parser, and the library API contract is unchanged.

This input-size guard bounds the encoded input buffer for the default CLI path but does not make inspection streaming: accepted input remains resident in memory, and metadata allocations remain governed separately by parser behavior or `inspect_with_limits()`. A streaming/file-backed inspection API is a later milestone and must not be simulated inside the CLI with duplicate parsing code.

## 13. Dependency policy

Expand Down Expand Up @@ -402,11 +409,15 @@ Primary controls:
- centralized bounds-checked cursor;
- checked integer arithmetic;
- no copies of opaque payload data;
- configurable metadata-count limits for callers that opt into them;
- a default bounded encoded-input read in the CLI;
- no project-authored `unsafe` in the initial parser;
- typed errors;
- targeted boundary tests;
- fuzz testing.

The CLI's default 256 MiB guard reduces the risk of an unexpectedly large whole-file allocation in normal CLI use. Raising `--max-input-bytes` is an explicit choice to accept a larger input buffer; it does not change the parser's in-memory architecture.

The "arbitrary input does not panic" invariant refers to malformed parser input; it does not claim recoverability from process-level failures such as global allocator exhaustion.

## 16. Testing architecture
Expand All @@ -421,6 +432,10 @@ For bit-field calculations, cursor reads, field-width rules, size derivation, ch

For complete standard frames, skippable frames, concatenation, empty/truncated input, and malformed streams.

### CLI boundary tests

For input-size limits, exact-boundary acceptance, rejection before parsing, diagnostic behavior, and CLI help/default-policy visibility.

### Reference-generated fixtures

Use the official `zstd` implementation to generate representative valid samples. Fixture provenance should be documented.
Expand Down Expand Up @@ -468,4 +483,6 @@ The v0.1 architecture is governed by:
- ADR 0003 — start with an in-memory `&[u8]` inspection API;
- ADR 0004 — v0.1 public API and licensing policy.

Issue #37 adds a CLI-specific bounded-read policy without changing the public parser API, so it does not supersede ADR 0003 or require a new public parsing architecture decision.

A future implementation that intentionally diverges from these decisions should introduce a new ADR rather than silently changing the architecture.
20 changes: 17 additions & 3 deletions README.ja.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ let file = inspect_with_limits(&bytes, limits)?;

上記の値は例であり、すべての用途に対する安全なdefault値ではありません。用途と想定入力に応じて設定してください。設定値ちょうどまでは許可され、さらに1件のFrame/Blockを解析しようとした時点で、対象構造の開始offsetを持つ`ZstdError::ResourceLimitExceeded`を返します。

この制限はFrame/Block metadataの**件数**を制御するもので、入力byte数そのものを制限したりstreaming化したりするものではありません。Block/Skippable payloadを解析結果へコピーしない既存方針も維持します。
この制限はFrame/Block metadataの**件数**を制御するもので、callerが保持する入力sliceのbyte数を制限したりstreaming化したりするものではありません。Block/Skippable payloadを解析結果へコピーしない既存方針も維持します。

Public API方針は[Public API設計](docs/API-DESIGN.md)で管理しています。

Expand All @@ -115,7 +115,21 @@ cargo run -p zstdscope-cli -- inspect sample.zst
cargo run -p zstdscope-cli -- inspect sample.zst --json
```

I/O errorやparse errorは非0で終了し、diagnosticをstderrへ出力します。pipe先が通常どおり先に閉じた場合のbroken pipeはpanicさせず正常終了として扱います。
### 大きな入力ファイル

CLIは引き続きin-memoryのlibrary APIを利用するため、受け入れた入力ファイルは解析中にメモリ上へ保持されます。defaultのCLI動作をboundedにするため、`zstdscope inspect`は **268,435,456 bytes(256 MiB)** を超えるencoded inputをparse前に拒否します。

上限を明示的に変更する場合は`--max-input-bytes`を指定します。

```text
zstdscope inspect large.zst --max-input-bytes 1073741824
```

上限を引き上げると、encoded input bufferに対して許容する最大メモリ量も増えます。CLIは可能な場合は全体read前にfile sizeを確認し、実際のread自体にも上限を適用するため、read中にファイルが増加しても設定した上限をそのまま通過しません。

このCLIのbyte上限と、libraryの`inspect_with_limits()`によるFrame/Block metadata件数上限は別のresource policyです。どちらもstreaming化するものではありません。入力全体をメモリへ保持したくない非常に大きなファイルについては、将来のstreaming/file-backed library APIで対応する方針です。

I/O error、parse error、input size limit超過は非0で終了し、diagnosticをstderrへ出力します。pipe先が通常どおり先に閉じた場合のbroken pipeはpanicさせず正常終了として扱います。

再利用可能な`zstdscope` libraryは[crates.io](https://crates.io/crates/zstdscope)で公開済みです。API documentationは[docs.rs](https://docs.rs/zstdscope)で確認できます。

Expand Down Expand Up @@ -157,7 +171,7 @@ Parserでは特に以下を重視します。
- v0.1ではproject codeに`unsafe`を使用しない
- Zstdの公開仕様を根拠に実装する

CLIとparserは引き続き入力全体をメモリ上で扱います。`inspect_with_limits()`によりFrame/Block metadata件数は制限できますが、入力ファイル自体のbyte数は制限しません。streaming/file-backed APIは後続milestoneで整備します。
Public parser APIは引き続き入力全体をメモリ上で扱います。`inspect_with_limits()`はFrame/Block metadata件数を制限し、CLIはそれとは別にdefault 256 MiBのencoded-input guardを持ちます。`--max-input-bytes`で明示的に上限を変更できますが、受け入れたCLI入力は依然として全体がメモリ上に存在します。streaming/file-backed APIは後続milestoneで整備します。

## ライセンス

Expand Down
18 changes: 16 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,20 @@ cargo run -p zstdscope-cli -- inspect sample.zst

The output reports frame type and boundaries, header metadata, block types and sizes, Skippable payload metadata, and stored checksum metadata when present.

### Large input files

The CLI still uses the in-memory library API, so an accepted input file is resident in memory while it is inspected. To keep the default CLI behavior bounded, `zstdscope inspect` rejects encoded input larger than **268,435,456 bytes (256 MiB)** before parsing.

Raise or lower that boundary explicitly with `--max-input-bytes`:

```text
zstdscope inspect large.zst --max-input-bytes 1073741824
```

Raising the limit also raises the maximum memory commitment for the encoded input buffer. The CLI checks the file size before the full read when possible and also bounds the actual read, so a file that grows while being read cannot silently bypass the configured limit.

This CLI byte limit is separate from `inspect_with_limits()`, which only limits frame/block metadata counts in the library. Neither mechanism makes inspection streaming. A future streaming/file-backed library API remains the path for files that should not be held fully in memory.

### JSON output

Use `--json` for machine-readable output:
Expand All @@ -163,7 +177,7 @@ cargo run -p zstdscope-cli -- inspect sample.zst --json

JSON field names and serialized enum values use explicit `snake_case` representations. `FrameKind` uses a tagged `type` / `data` shape, and Inspector-specific distinctions such as absent versus explicitly encoded zero Dictionary IDs and RLE declared versus encoded sizes are preserved.

I/O and parse failures return a non-zero exit status, write diagnostics to stderr, and do not emit partial-success JSON. Output write failures are handled without panicking; a downstream process closing a pipe normally is treated as a normal CLI termination.
I/O and parse failures return a non-zero exit status, write diagnostics to stderr, and do not emit partial-success JSON. An input that exceeds the CLI byte limit also returns a non-zero exit status with a structured CLI error. Output write failures are handled without panicking; a downstream process closing a pipe normally is treated as a normal CLI termination.

### Release distribution

Expand Down Expand Up @@ -230,7 +244,7 @@ Where the current reference specification and the RFC differ, the difference mus

ZstdScope treats every input byte as untrusted. Parser reads and skips are bounds-checked, offset/size arithmetic is checked, opaque payloads are not copied into the inspection model, and the project forbids authored `unsafe` Rust in the core crate.

The parser and CLI remain intentionally in-memory. `inspect_with_limits()` can bound frame/block metadata counts for untrusted inputs, while `inspect()` retains the original unlimited count behavior. The complete input is still resident in memory; streaming/file-backed inspection remains later hardening work.
The public parser APIs remain intentionally in-memory. `inspect_with_limits()` can bound frame/block metadata counts for untrusted inputs, while `inspect()` retains the original unlimited count behavior. The CLI adds a separate default 256 MiB encoded-input guard with `--max-input-bytes` as an explicit override, but any accepted CLI input is still held fully in memory. Streaming/file-backed inspection remains later hardening work.

Parser fuzzing is available through `cargo-fuzz`; successful fuzz parses are also checked against structural model invariants. See [FUZZING.md](FUZZING.md) for setup, execution, and regression-handling instructions. Fuzzing is manual initially and is not part of normal pull-request CI.

Expand Down
5 changes: 3 additions & 2 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,13 +49,14 @@ Status: **in progress on `main`; not yet released.**
Already implemented on `main` after v0.1.0:

- configurable parser resource limits;
- a manual `cargo-fuzz` target with structural model-invariant checks on successful parses.
- a manual `cargo-fuzz` target with structural model-invariant checks on successful parses;
- bounded CLI file reads with a default 256 MiB encoded-input guard and explicit `--max-input-bytes` override.

Remaining candidate scope:

- richer field-level spans for hex-view mapping;
- more detailed diagnostics;
- streaming or file-backed inspection API for very large files;
- streaming or file-backed library inspection API for very large files;
- scheduled or continuous fuzzing automation;
- JSON schema documentation;
- CLI output refinements.
Expand Down
92 changes: 82 additions & 10 deletions crates/zstdscope-cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,16 @@ mod render;

use std::{
fmt, fs,
io::{self, Write},
path::PathBuf,
io::{self, Read, Write},
path::{Path, PathBuf},
process::ExitCode,
};

use clap::{Parser, Subcommand};
use zstdscope::{ZstdError, ZstdFile};

const DEFAULT_MAX_INPUT_BYTES: u64 = 256 * 1024 * 1024;

#[derive(Debug, Parser)]
#[command(
name = "zstdscope",
Expand All @@ -31,6 +33,13 @@ enum Command {
/// Emit machine-readable JSON instead of the human-readable summary.
#[arg(long)]
json: bool,
/// Maximum encoded input size to read into memory.
#[arg(
long,
default_value_t = DEFAULT_MAX_INPUT_BYTES,
value_name = "BYTES"
)]
max_input_bytes: u64,
},
}

Expand All @@ -52,19 +61,64 @@ fn main() -> ExitCode {

fn run<W: Write>(cli: Cli, writer: &mut W) -> Result<(), CliError> {
match cli.command {
Command::Inspect { file, json } => inspect_file(file, json, writer),
Command::Inspect {
file,
json,
max_input_bytes,
} => inspect_file(file, json, max_input_bytes, writer),
}
}

fn inspect_file<W: Write>(path: PathBuf, json: bool, writer: &mut W) -> Result<(), CliError> {
let input = fs::read(&path).map_err(|source| CliError::Io {
path: path.clone(),
source,
})?;
fn inspect_file<W: Write>(
path: PathBuf,
json: bool,
max_input_bytes: u64,
writer: &mut W,
) -> Result<(), CliError> {
let input = read_input_with_limit(&path, max_input_bytes)?;
let file = zstdscope::inspect(&input).map_err(CliError::Parse)?;
write_inspection(writer, &file, json)
}

fn read_input_with_limit(path: &Path, max_input_bytes: u64) -> Result<Vec<u8>, CliError> {
let file = fs::File::open(path).map_err(|source| CliError::Io {
path: path.to_path_buf(),
source,
})?;

let metadata = file.metadata().map_err(|source| CliError::Io {
path: path.to_path_buf(),
source,
})?;
if metadata.len() > max_input_bytes {
return Err(CliError::InputTooLarge {
path: path.to_path_buf(),
limit: max_input_bytes,
observed: metadata.len(),
});
}

let probe_limit = max_input_bytes.saturating_add(1);
let mut input = Vec::new();
file.take(probe_limit)
.read_to_end(&mut input)
.map_err(|source| CliError::Io {
path: path.to_path_buf(),
source,
})?;

let observed = input.len() as u64;
if observed > max_input_bytes {
return Err(CliError::InputTooLarge {
path: path.to_path_buf(),
limit: max_input_bytes,
observed,
});
}

Ok(input)
}

fn write_inspection<W: Write>(writer: &mut W, file: &ZstdFile, json: bool) -> Result<(), CliError> {
if json {
serde_json::to_writer_pretty(&mut *writer, file).map_err(CliError::Json)?;
Expand All @@ -78,7 +132,15 @@ fn write_inspection<W: Write>(writer: &mut W, file: &ZstdFile, json: bool) -> Re

#[derive(Debug)]
enum CliError {
Io { path: PathBuf, source: io::Error },
Io {
path: PathBuf,
source: io::Error,
},
InputTooLarge {
path: PathBuf,
limit: u64,
observed: u64,
},
Parse(ZstdError),
Json(serde_json::Error),
Output(io::Error),
Expand All @@ -89,7 +151,7 @@ impl CliError {
match self {
Self::Json(source) => source.io_error_kind() == Some(io::ErrorKind::BrokenPipe),
Self::Output(source) => source.kind() == io::ErrorKind::BrokenPipe,
Self::Io { .. } | Self::Parse(_) => false,
Self::Io { .. } | Self::InputTooLarge { .. } | Self::Parse(_) => false,
}
}
}
Expand All @@ -104,6 +166,15 @@ impl fmt::Display for CliError {
path.display()
)
}
Self::InputTooLarge {
path,
limit,
observed,
} => write!(
formatter,
"input too large: {} exceeded the {limit} bytes limit (observed at least {observed} bytes); use --max-input-bytes to raise the limit",
path.display()
),
Self::Parse(source) => write!(formatter, "parse error: {source}"),
Self::Json(source) => write!(formatter, "JSON serialization error: {source}"),
Self::Output(source) => write!(formatter, "output error: {source}"),
Expand All @@ -118,6 +189,7 @@ impl std::error::Error for CliError {
Self::Parse(source) => Some(source),
Self::Json(source) => Some(source),
Self::Output(source) => Some(source),
Self::InputTooLarge { .. } => None,
}
}
}
Expand Down
Loading