Repository navigation
feat(cli): redesign the commands and flags, add entry filters, --rm, --dereference and a 'test' command - #25
Merged
Merged
Conversation
- New 'tests/python/cli/' now has a module per command: test_compress.py, test_extract.py, test_inspect.py, and test_list.py - New test_app.py holds the shared formatters, app structure, progress bar, exit codes, interruption, and threads - New CLI conftest.py has the shared fixtures and helpers used in CLI tests - 'TestInspectAndList' becomes 'TestInspect' and 'TestList', one per module
- cli/_dispatch.py becomes frontend/dispatch.py, since it only holds dispatch rules and no CLI logic - New test_dispatch.py tests format detection, '.comp' handling, zip recognition, directory handling, and '-o' into an existing directory - looks_like_archive()'s bare 'except' gains the reason it is bare, that an unreadable file is not an archive
- plan_compress() and plan_extract() now take a frozen 'CompressRequest' or 'ExtractRequest' and return the planned job, or one per input - Command handlers take a job and are left with rendering, and read the plan rather than their own arguments - Single-file format with several inputs is now a 'ValueError' from the planner, which 'compress' turns into its usage exit - test_dispatch.py now asserts on the job each request plans - 'TestSmartCompress' and 'TestSmartExtract' keep one round trip each, asserting the commands run what the planner returns
…lting to zip - plan_archive() now reads an unset format off the output's suffix, falling back to zip - 'ArchiveOptions.format' defaults to None, and a caller passing a named output gets the format they named - 'ArchiveJob.run' asserts that the planner resolved it - Format tables, canonical_standalone_format(), infer_format(), and 'DEFAULT_ARCHIVE_FORMAT' move to a new frontend/_formats.py
- '_render' becomes a package holding all CLi printing and output logic - New jobs.py holds the before-and-after report for each job kind - New info.py holds the entry listing and the archive and single-file metadata reports - The package root holds formatting, progress and exit codes - Commands are simplified to the plan check, timing, progress bar, and the exit, with each report now one call taking a plan
- New cli/_run.py holds run_job(), which times one job under its bar and raises 'InputFailed' - run_all() runs each in turn, reports a failure with the input's name and continues, returning the exit code - report_error() splits out of fail(), so a failure can be printed without exiting - Drops exit_for_result(), as its cancel-and-fail logic is now included in run_job() - extract() now stats its inputs up front, so a path that isn't a file is still a usage error with nothing run - An unrecognised format or a truncated stream now fails that input alone - Drops the timing, bar and exit handling from the four command bodies - compress() now runs through the same path with its single job
- 'formats' now prints backends, standalone containers, and archive formats - Includes algorithms with level ranges, standalone formats with aliases/extensions, and archive formats with compressed variants - The variants are dervied from the core rather than assumed, so a codec that isn't compiled doesn't get listed - Moves rendering to _render/formats.py alongside the other reports, with the heading and item layout shared by the three sections
- 'tabulate' now imports when needed, instead of for every command - 'asyncio' is now only imported when run_async() is called, rather than for sync commands as well
- Merges 'extract --list' and 'inspect --entries' into a single 'list' command - 'list' takes several archives, prints each under a '==> path <==' header as head does, and goes through run_all() - A single-file container or an unreadable input fails alone and the rest still list - New '--long' adds each entry's mode, mtime and, where the format tracks them, its compressed size and CRC - '--json' is one array item per archive, carrying the entries in full - Drops the '--list' and '--entries' flags from 'extract' and 'inspect' - Updates 'formats' interrupted message
- Renames the 'inspect' command to 'info' - 'info' now dispatches on the detected format - An archive gets its summary, a '.comp' its header detail, and a standalone codec file its format and size - Now takes several inputs, each headed with '==> path <==', and runs through run_all() - An unrecognised or unreadable input fails alone, while with the rest still described - '--json' is one array item per input, each carrying its format - The render calls now return their JSON item and print only the text, so one print_json() serialises what both commands collected - require_files() moves into _run.py for the shared up-front stat - Adds an 'OSError' branch to run_all(), so a file made unreadable since that stat is reported against itself - Drops 'ExtractJob.list_contents' along with 'extract --list'
- Each command now has a single initial alias, rather than multiple - Drops the 'comp' console script, so the name isn't claimed on PATH twice - 'formats' moves ahead of 'benchmark' in the help - Drops the now unused 'TestMergedVerbCompatibility'
- Drops the multiple overwrite flags on the 'compress' and 'extract' commands - New '--on-conflict' option takes overwrite mode by name, with '-f/--force' as shorthand for overwrite - New conflict_mode() in _run.py resolves the two for both commands, refusing '-f' alongside a mode that contradicts it - '--format' now uses a '-F' alias, rather than '-f' - 'OverwriteMode' and 'SymlinkMode' now list their members default-first, so the help shows the choices in that order
- Each finished input now outputs one line: 'name before → after (n%) in time' - Renames are noted where the output's name was taken, and a 'skipped, already exists' line under '--on-conflict skip' - The plan and stats blocks are now shown with a '-v/--verbose' flag - The '-q' flag still prints nothing, and both used together is a usage error - report_error() now flushes stdout first, so a failure's error doesn't appear above the summary lines of earlier inputs - plan_decompression() now reports an OSError's own 'strerror' rather than wrapping it in 'Cannot read file'
- Both flags on formats other than '.comp' now makes an unrunnable plan, so the command exits 2 with nothing written - '-a' with '-s' is a now a usage error, as they are mutually exclusive - 'strategy' now defaults to None, which lets the CLI tell a given strategy from an unspecified one - The help panel still shows 'balanced' through 'show_default', and the planner applies it - 'auto' now folds into None where the other normalising happens, rather than inside the request construction
- New '-C/--directory' flag is now where to write, for any input - The '-o' flag is now the one file to write, for a single input that isn't an archive - Drops the now unused resolve_single_output() - Mutually exclusive combinations are refused: '-o' with '-C', '-o' with several inputs, and '-o' on an archive - 'compress -o' now takes the path as given, rather than treating an existing directory as the output directory - '-a' stops showing 'auto' as its default
- New '--member/-m' flag is repeatable and takes an entry path, selecting that entry or a directory and everything below it - plan_extraction() filters the entries, so the plan, progress total, and listing count all describe what is actually written - A name matching nothing fails that archive after the rest is extracted - '-m' on a non-archive input fails, since a single-file container has no entries to name
- Adds '--no-same-permissions', '--touch/-m', '--overwrite-dir', and '--max-depth' flags to the 'extract' command - '-p' with '--no-same-permissions' is a now a usage error rather than silent precedence - Changes '--member/-m' becomes to '--entry/-e' - Changes '--max-total-size' to '--max-size' and now takes a suffix, so a cap reads as '500M' or '10G' rather than a byte count - Sizes that doesn't parse fail before any work - Directories that don't exist are now created for single-file inputs as well - Sorts the vcpkg dependencies into alphabetical order
…ting to _render/ - print_results() becomes cli/_render/benchmark.py alongside the rest of the CLI output - New benchmark_header() takes over the settings block the command built inline - benchmark_file() drops the redundant warnings about a compressed size that changed between runs and a decompressed size that didn't match - The '--algos/--strategies/--levels' become repeatable '--algo/-a', '--strategy/-s' and '--level/-l' - Values no longer have to be split out of a comma-separated string - '--repeats/-r' is now bounded with a minimum of 1 - Adds new test_benchmark.py that covers each flag, header and table printing, and non-int/'auto' level exiting 2
- summary() now takes a 'ratio' flag, off for extraction and decompression - Moves the leading blank line out of summary() into the command, printed once before the jobs run - Several inputs now get one blank line before, between and after their lines - skipped() now ends with a blank line too, so it spaces like the other outputs
- New path_matches() tests a pattern against any run of whole components in an entry path - The matcher uses fnmatch(3) syntax with no flags, and backtracks only to the last '*', keeping it linear time based on path length - An unterminated '[' is a literal bracket, and a zip directory's trailing slash is trimmed before matching - Adds 16 test case rows, with the usual '_Static_assert' guarding the 'TEST_RANGE'
- create_archive()/extract_archive() now take 'include' and 'exclude' patterns, held in a 'PathFilter' - An exclude wins, and with any includes an entry must match one - Creating matches before the stat, so an excluded tree is never descended into - A directory that doesn't match an include is kept only provisionally, pruned afterwards if nothing below it survived - An 'include' adds to 'files', an exclude overrides both, and a directory entry is kept only if it matches itself - Includes that match no entry join the unmatched names in 'IncompleteExtractionError', once the rest is extracted, so a typo isn't silent - Excludes are applied after that check, so a name or include the filter drops still counts as found - entry_name() becomes walk_entry_name(), so the walk and the filter agree on the name a pattern is matched against - The converter refuses a bare str or bytes, which would otherwise be taken as a sequence of one-character patterns - Adds new tests covering includes, excludes, and combinations
- Adds both flags to 'compress' and 'extract' commands, wiring them to 'ArchiveOptions'/'ExtractOptions' and the C filter - A single-file input or output refuses them, since a glob has nothing to select in a one-entry container - The plan now applies the same globs, prunes excluded directories, and then lstats each file - Now counts hardlinked files once where the format stores links, so the progress bar's total matches the archive - path_matches() and format_stores_hardlinks() are exported from '_core' so the plan matches entry names exactly as the walk and the extractor do - plan_extraction() mirrors the extractor,with name or include selecting, exclude overrides both, so planned entries are the ones written - A run whose excludes leave nothing still exits 0, but informs the user through new report_warning() (unless '-q') - Moves the parenthesised defaults in '-o' and '--symlinks' help into 'show_default'
- New '--rm' flag on 'compress'/'extract' deletes each input once its output is complete - New ''--keep/-k' flag maintains the default behaviour of retaining each input - New 'remove_sources' on create_archive() and 'ArchiveOptions' does the same for an archive - Removes deepest first, so only what was stored goes and a directory holding anything unstored stays - '--rm' is refused with '--on-conflict skip', and on extraction with '--entry', '--include' or '--exclude' - In each case the output holds less than the input, so deleting it would lose data - A source that isn't removed is a warning after the normal summary and a non-zero exit - New 'SourceRemovalError' carries the path written plus every path left behind with its reason - A special file or symlink the platform or user can't create now joins 'IncompleteExtractionError' instead of logging a warning - A run that couldn't write everything now fails and '--rm' can't take the archive with it - Windows now maps a missing privilege, an unsupported operation, and a volume that can't do it to EPERM - Anything else from a failed symlink is now EIO, so a full disk still fails the extraction rather than reading as a skipped link - '-q' no longer returns before the work after the summary, so a quiet run still removes its inputs
- New '-h/--dereference' flag on 'compress', and 'dereference' on create_archive() and 'ArchiveOptions', stores a link's target in its place - A link to a directory between it and its source stays a link, so the walk can't loop, as does a dangling one, which has nothing to store - 'WalkItem' tracks each item's parent to make that check - Dangling links warn only where the data was wanted (named as a source, or followed), since otherwise it is a faithful copy and nothing is lost - '--rm' deletes the links it followed and leaves what they lead to, unlike GNU tar, so dereferencing can't delete the originals - New fs_stat_target() stats through a link - On Windows the stat splits so both paths share it, and a directory now gets an identity, which the ancestor check compares - Plans now follow the same rules, and a missing source is now 'lexists', so a dangling link is counted - Adds tests covering dereferencing, planning, combined with '--rm', links to ancestors, and dangling links
- New 'test' command decodes each input in full and writes nothing, checking integrity based on format - A plain '.tar' has no checksum over its data, so only its headers and the data's presence are checked - Trailing data fails, by raising 'TrailingDataWarning' for the run - New '_core.verify_file' checks it, with 'VerificationPlan', plan_verification() and 'VerificationJob' added to the frontend - An unrecognised file is refused at the plan rather than part-way through - list_archive_contents() becomes a generic run_reader_pass() that the check now reuses - A plain archive now gets a headers-only pass first, so progress counts what its entries hold rather than the file's size - A NULL 'OutputTarget.path' now means discard, writing to the null device and committing nothing - Zip now reads one byte past a link target, since libzip only checks the CRC once a read reaches the end - A cancel while the reader opens is now reported as cancelled rather than an unexplained failure - Adds new test_verify.py and test_verification.py covering all test command behaviour
- A standalone gzip codec file now shows the name and mtime its header stores - Also shows the uncompressed size an xz index or a zstd, lz4 or snappy header states, and the ratio that size gives - stored_content_size() reads those without decoding, so a damaged file can still state a size - It answers None unless the structures account for the whole file - zstd frames are walked by RFC 8878 rather than through 'ZSTD_getFrameHeader', which needs static linking - xz goes through lzma_file_info_decoder(), so concatenated streams and their padding add up - 'lz4' now records the content size when compressing a file of known length - Every lz4 codec fixture hash is updated to match - Moves gzip_stored_mtime() into frontend/_formats.py - Splits fs_fseek() out of fs_stream_size() for the seeking the walks do - The gzip-header fuzz target now also drives all four size readers
- 'Strategy' becomes a 'StrEnum' in the frontend, exported and used for 'CompressionOptions.strategy' - '-s' and '-a' are now enum options, so a typo is refused while parsing with the valid names listed, and '--help' shows them - 'Algo' is now built from the core's capabilities, so only compiled backends are included - '-F comp' now names the Compresso container, as '.comp' on the output already did, and writes it whatever the file is called - An unknown '-F' now lists available formats, instead of refusing without saying what is valid - archive_formats() moves into frontend/_formats.py, where the 'formats' command and the plan both share it - plan_compression() now runs the core's check even with no level given, so an unknown algorithm is caught by the plan rather than at run time
- main() now wraps a terminal stdout and stderr so a run gets exactly one blank line above its first visible output and one below its last - Shared between the streams so they pad once between them, and nothing extra where the output already has one - Piped output isn't padded, since tar and gzip don't pad what a script reads - Drops every command's own leading and trailing print(), so a summary, a stats block and an error no longer each decide their own spacing - A verbose run instead gets a blank line after the progress bar, to set it apart from the stats - Drops speed, saved bytes, and the separate ratio and original-size from compress stats, which the plan block and the output line already show - Adds ratio to the archive output summary - Adds unit tests for the new terminal output wrapping
- New handles_errors() holds the Ctrl-C exit, the core-error exit and the catch-all that each command repeated - Drops the outer try block from the command bodies - Drops 'HeaderError' and 'BackendError' from the compress and extract clauses, as they are 'Error' subclasses already - Commands that only had a catch-all now name the operation for a core error, rather than calling it unexpected
- fs_stat_wide() now takes the type, size, mtime, identity and link count from the handle, which follows the link - A handle it can't open is a dangling link, which now fails with an errno rather than returning a row with a zeroed identity - main() now reconfigures stdout with 'backslashreplace', since a pipe takes the locale code page, which can't encode the '→' in a summary line - The entry-over-the-archive case is now skipped on Windows, which won't let libarchive's open file be replaced - The snappy parametrisation test now gets explicit ids, since raw bytes overflows Windows' 32767-character 'PYTEST_CURRENT_TEST'
- New 'FS_SEP' names the OS-specific separator beside 'FS_IS_SEP', which already accepted both - fs_join() now uses 'FS_SEP', rather than always inserting '/', which a Windows path mixed with the backslashes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(cli): redesign the commands and flags, add entry filters, --rm, --dereference and a 'test' command
The CLI is refactored so each flag has one meaning, modelled on tar/bsdtar for archives and gzip/zstd/xz for codecs. The format and output rules move out of the CLI into
frontend/dispatch.py, so the commands only parse, run and render. Several inputs now carry on past a failure, and each finished input prints one summary line. Adds new include and exclude globs for archive entries, deleting inputs with--rm, archiving symlink targets with--dereference, atestcommand that checks files for damage, andinfoshowing what a codec file's own header states.Breaking changes
compexecutable is removed; usecompressoinspectis renamedinfo, and the oldlist(backends and levels) is renamedformats;listis now a new command for an archive's entriesc,x,l,i,tandbextract --listandinspect --entriesare removed; uselistandlist --long--overwrite,--skip-existingand--error-on-conflictare replaced by--on-conflict <mode>, with-f/--forceas shorthand for overwritecompress --formatmoves from-fto-F, since-fis now--forceextract -o/--output-dirsplits into-C/--directory(where to write) and-o/--output(the one file to write, for a single input that isn't an archive)compress -otakes the path as given; an existing directory is no longer treated as the output directoryextract --max-total-sizebecomes--max-sizeand takes a suffix, e.g.500Mor10Gbenchmark --algos/--strategies/--levels, which took comma-separated values, become the repeatable-a/--algo,-s/--strategyand-l/--levelcompress -aor-son a format other than.compnow exits 2 with nothing written, and-awith-sis a usage error-aand-sonly accept the compiled backends and the known strategies, so a typo is refused while parsing, with the valid names listed.lz4bytes change and the codec manifest is re-recordedIncompleteExtractionErrorafter the rest is extracted, instead of logging a warning; on Windows a failed symlink maps toEPERMfor a missing privilege or unsupported volume, and toEIOotherwise, so a full disk still fails the extractionCompressionOptions.strategyis now aStrategyStrEnuminstead of astrprint_results()is removed fromcompressoandcompresso.introspect; the table now lives incli/_render/benchmark.pyExtractJob.list_contents()is removedArchiveOptions.formatdefaults toNoneinstead of'tar.zst';plan_archive()now infers it from the output's suffix, falling back to zipplan_decompression()reports anOSError's ownstrerrorinstead of wrapping it inCannot read file: …compressno longer prints speed, bytes saved, or the separate ratio and original size in its stats, since the plan block and the summary line already show themCommands
extract,list,infoandtestrun each input in turn; a failed input is reported against its name and the rest still run, with the exit code reflecting the failure; a path that isn't a file is still a usage error, checked before anything runsname before → after (n%) in time, noting a rename or a skip under--on-conflict skip; extraction and decompression leave out the ratio; the plan and stats blocks moved behind-v/--verbose, and-vwith-qis a usage error; on a terminal a run gets exactly one blank line above and below its output; piped output isn't paddedtest: new command that decodes each input in full and writes nothing, checking whatever the format checksums; a plain.tarhas no checksum over its data, so only its headers and the data's presence are checked; trailing data fails, and an unrecognised file is refused before decoding startslist: takes several archives, each under a==> path <==header asheadprints them;--longadds each entry's mode, mtime and, where the format tracks them, compressed size and CRC;--jsongives one array item per archiveinfo: dispatches on the detected format: an archive gets its summary, a.compits header and a codec file its format and size; a gzip file also shows its stored name and mtime; an xz, zstd, lz4 or snappy file shows the uncompressed size its index or header states, with the ratio that size gives, read without decoding so a damaged file still shows it; takes several inputs, and--jsongives one array item per inputformats: lists the.compalgorithms with their level ranges, the standalone containers with their aliases and extensions, and the archive formats with their compressed variants; the list comes from the core, so a codec that isn't compiled in isn't listedcompressandextract:--includeand--excludetake fnmatch globs matched against any run of whole path components; an exclude overrides everything else, and with any includes an entry must match one; on extract an include adds to--entry, and an include that matches nothing fails the archive after the rest is extracted; a single-file input or output refuses both flags--rmdeletes each input once its output is complete, and--keep/-kkeeps the default; an archive removes deepest first, so only what was stored is deleted; it's refused with--on-conflict skip, and on extraction with--entry,--includeor--exclude, since the output would hold less than the input; an input that can't be removed gives a warning and a non-zero exitcompress:-h/--dereferencestores what a symlink points to instead of the link; a link to one of its own ancestor directories stays a link so the walk can't loop, and so does a dangling one;--rmwith it deletes the links and leaves their targets-F compwrites the.compcontainer whatever the output is called, and an unknown-Flists the valid formatsextract:-e/--entryextracts only that entry, or a directory and everything below it; the plan, progress total and count describe only what is writtenl; a name matching nothing fails that archive after the rest is extracted, and-eon a non-archive input fails--no-same-permissions,-m/--touch,--overwrite-dirand--max-depthexpose the remaining extract options;-pwith--no-same-permissionsis a usage error-owith-C,-owith several inputs, and-oon an archivebenchmark:-r/--repeatsmust be at least 1;benchmark_file()drops the redundant warnings about sizes changing between runsasyncioandtabulateare imported only when used, which speeds up startup for the synchronous commandsCore
create_archive()andextract_archive()takeincludeandexcludepatterns; when creating, an excluded tree is never descended into, and a directory that matches no include is kept only if something below it survivespath_matches()andformat_stores_hardlinks(), so the planner matches entry names and counts hardlinks the same way the walk and the extractor docreate_archive()takesremove_sourcesanddereference; a source that couldn't be removed raises the newSourceRemovalError, which carries the path written and every path left behind with its reasonverify_file()fortest.list_archive_contents()becomes a generic reader pass thatverify_file()reuses; a plain archive gets a headers-only pass first, so progress counts what its entries hold rather than the file's sizestored_content_size()reads the size a codec file states, and returnsNoneunless the structures account for the whole file; zstd frames are walked per RFC 8878, becauseZSTD_getFrameHeaderneeds static linking; xz goes throughlzma_file_info_decoder(), so concatenated streams and their padding add upFrontend
frontend/dispatch.pytakes the dispatch rules fromcli/_dispatch.py;plan_compress()andplan_extract()take a frozenCompressRequestorExtractRequestand return the planned job, or one per input, so the GUI can reuse them unchangedValueErrorfrom the planner, whichcompressturns into a usage errorArchiveOptionsgainsinclude,exclude,remove_sourcesanddereference, andExtractOptionsgainsincludeandexclude; the plans apply the same rules as the core, so the planned entries are the ones writtenVerificationPlan,plan_verification()andVerificationJob, plusStrategy, exported fromcompressoandcompresso.frontendplan_compression()runs the core's check even with no level given, so the plan catches an unknown algorithm instead of the runcanonical_standalone_format(),infer_format(),archive_formats()andgzip_stored_mtime()move tofrontend/_formats.pyCLI internals
_renderbecomes a package that holds all CLI output:jobs.pyfor each job's plan, result and summary,info.pyforlistandinfotext and JSON, plusformats.pyandbenchmark.pycli/_run.pyholdsrun_job(),run_all(),conflict_mode(),require_files()andremove_source(), which replace each command's own timing, progress bar and exit handling;InputFailedandNotRemovedmark one input's failure forrun_all()to reporthandles_errors()decorator holds the Ctrl-C exit, the core-error exit and the catch-all that every command repeated, so a command now names its operation for a core error instead of calling it unexpectedreport_error()splits out offail()and flushes stdout first, so an error doesn't print above earlier inputs' summary linesmain()wraps a terminal's stdout and stderr to pad a run's output in one place, replacing each command's own leading and trailingprint()Python tests
test_cli.pysplits intotests/python/cli/, with one module per command,test_app.pyfor shared formatting, exit codes, interruption and output padding, and aconftest.pyfor the shared fixturesfrontend/test_dispatch.pyasserts on the job each request plans; the CLI round trips keep one case each and check that the commands run what the planner returnscli/test_benchmark.py,cli/test_verify.pyandfrontend/test_verification.pytest_archive_create.py,test_archive_extract.pyandtest_core.pycover include and exclude,--rm, dereferencing (including links to ancestors and dangling links), and the stored-size readersC tests
test_select.ccoverspath_matches()