diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..2f13ce7 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,40 @@ +name: CI + +on: + pull_request: + push: + branches: [main] + +permissions: + contents: read + +jobs: + dashboard: + name: Dashboard / Python ${{ matrix.python-version }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + python-version: ["3.10", "3.12"] + defaults: + run: + working-directory: dashboard + steps: + - uses: actions/checkout@v7 + - uses: actions/setup-python@v7 + with: + python-version: ${{ matrix.python-version }} + - name: Install test dependencies + run: python -m pip install -e '.[test]' + - name: Run tests + run: python -m pytest -q + + watchdog-downloader: + name: watchdogDownloader / Bash + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - name: Check syntax + run: bash -n watchdogDownloader/wdd + - name: Check command entry point + run: watchdogDownloader/wdd --version diff --git a/README.md b/README.md index cdcf848..333535e 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,95 @@ # MyPracticalTools -收录了一些trivial，好用，可复用的个人化小工具 -## 工具 +[![CI](https://github.com/Whning0513/MyPracticalTools/actions/workflows/ci.yml/badge.svg)](https://github.com/Whning0513/MyPracticalTools/actions/workflows/ci.yml) -- [Practical Run Dashboard](dashboard/README.md)：长时间训练/评测任务的 - 进度、检查点、ETA、阶段和 GPU 状态 TUI。 +Small, reusable tools and reference assets for long-running ML, data, and +automation workflows. -## Tools +这是一个个人实用工具集合。每个工具都有独立文档和明确的使用边界，可以单独安装或复制， +不需要采用整个仓库的其他部分。 -- [watchdogDownloader](watchdogDownloader/) - Automatic downloading with retrying and integrity checking. +## Included projects + +| Project | Runtime | Purpose | +| --- | --- | --- | +| [Practical Run Dashboard](dashboard/README.md) | Python 3.10+ | Read-only terminal dashboard for checkpointed training and evaluation jobs, including progress, ETA, GPU use, and recoverable state. | +| [watchdogDownloader](watchdogDownloader/README.md) | Bash on Linux | Resumable manifest-based downloads with low-speed detection, process supervision, and size or SHA-256 verification. | +| [ACA small v0.2](datasets/ACA_small_v0.2/) | Zstandard JSONL | Versioned train/test dataset package with a manifest, blind-gate metadata, and audit reports. | +| [Dataset design notes](docs/dataset-and-datapackage-design.md) | Markdown | Reproducibility, split isolation, replay, validator, reference, and release requirements for the ACA data package. | +| [User workstyle skill](whn_skill/whn_skill.md) | Markdown | A Codex skill for evidence-driven coding, research, and technical communication. | + +## Quick start + +### Practical Run Dashboard + +Install directly from this repository: + +```bash +python -m pip install \ + "git+https://github.com/Whning0513/MyPracticalTools.git#subdirectory=dashboard" +``` + +Copy and edit the example configuration, then start the live terminal view: + +```bash +practical-dashboard --config dashboard/examples/aca-rl-matrix.json +``` + +See the [dashboard documentation](dashboard/README.md) for JSON output, +checkpoint semantics, GPU ownership, and expected status files. + +### watchdogDownloader + +Install the script into a directory on `PATH`: + +```bash +git clone https://github.com/Whning0513/MyPracticalTools.git +install -Dm755 MyPracticalTools/watchdogDownloader/wdd "$HOME/.local/bin/wdd" +wdd --help +``` + +Create a tab-separated manifest, initialize a download project, and start it: + +```bash +wdd init /srv/download-state /srv/files ./manifest.tsv +wdd start /srv/download-state +wdd watchdog-start /srv/download-state +wdd status /srv/download-state +``` + +See the [watchdogDownloader documentation](watchdogDownloader/README.md) for +the manifest format, retry behavior, verification, and tuning controls. + +## Dataset package + +`datasets/ACA_small_v0.2/` contains compressed train/test problems and +submissions. Use `manifest.json` as the versioned source of truth and inspect +`audit/report.json` before consuming the package. The accompanying +[design document](docs/dataset-and-datapackage-design.md) distinguishes frozen +benchmark evidence from replay output, constructed references, validators, and +blind probes. + +## Development + +Run the Python tests: + +```bash +python -m pip install -e './dashboard[test]' +python -m pytest dashboard/tests -q +``` + +Check the Bash script syntax: + +```bash +bash -n watchdogDownloader/wdd +watchdogDownloader/wdd --version +``` + +CI runs both checks for pull requests and changes to `main`. + +## Contributing + +Focused bug reports and pull requests are welcome. Include a minimal +reproduction, explain the behavioral change, and state the validation command. +Do not include credentials, private datasets, generated caches, or machine- +specific absolute paths. diff --git a/watchdogDownloader/README.md b/watchdogDownloader/README.md index b61d89d..d4f2f73 100644 --- a/watchdogDownloader/README.md +++ b/watchdogDownloader/README.md @@ -11,20 +11,23 @@ It is intentionally conservative: - A watchdog can restart the whole downloader process group if aggregate downloaded bytes stop increasing. - Files are checked against expected byte sizes from a manifest. -## Install +## Requirements -The main command is: +`watchdogdownloader` targets Linux and requires Bash, `curl`, `setsid`, +`flock`, and standard GNU utilities such as `readlink`, `stat`, and `sha256sum`. -```bash -/data/whn/tools/watchdogdownloader/wdd -``` +## Install -Optional convenience symlink: +Clone the repository and install `wdd` into a directory on `PATH`: ```bash -ln -sf /data/whn/tools/watchdogdownloader/wdd /data/whn/tools/wdd +git clone https://github.com/Whning0513/MyPracticalTools.git +install -Dm755 MyPracticalTools/watchdogDownloader/wdd "$HOME/.local/bin/wdd" +wdd --version ``` +Alternatively, run `watchdogDownloader/wdd` directly from the checkout. + ## Manifest Create a tab-separated manifest: @@ -48,40 +51,40 @@ Rules: Create a project: ```bash -/data/whn/tools/watchdogdownloader/wdd init \ - /data/whn/raw/mydownload \ - /data/whn/raw/mydownload/files \ - /data/whn/raw/mydownload/manifest.tsv +wdd init \ + /srv/mydownload-state \ + /srv/mydownload-files \ + /srv/mydownload-state/manifest.tsv ``` Start the downloader: ```bash -/data/whn/tools/watchdogdownloader/wdd start /data/whn/raw/mydownload +wdd start /srv/mydownload-state ``` Start the watchdog: ```bash -/data/whn/tools/watchdogdownloader/wdd watchdog-start /data/whn/raw/mydownload +wdd watchdog-start /srv/mydownload-state ``` Check status: ```bash -/data/whn/tools/watchdogdownloader/wdd status /data/whn/raw/mydownload +wdd status /srv/mydownload-state ``` Verify after completion: ```bash -/data/whn/tools/watchdogdownloader/wdd verify /data/whn/raw/mydownload +wdd verify /srv/mydownload-state ``` Stop both watchdog and downloader: ```bash -/data/whn/tools/watchdogdownloader/wdd stop /data/whn/raw/mydownload +wdd stop /srv/mydownload-state ``` ## Config @@ -146,17 +149,15 @@ Use parallelism only for independent files. Do not use it to split a single file An example manifest is included: -```bash -/data/whn/tools/watchdogdownloader/examples/ccplus_1x.manifest.tsv +```text +watchdogDownloader/examples/ccplus_1x.manifest.tsv ``` To create a fresh generic project for that dataset: ```bash -/data/whn/tools/watchdogdownloader/wdd init \ - /data/whn/raw/codecontestplus_wdd \ - /data/whn/raw/codecontestplus_wdd/ccplus_1x \ - /data/whn/tools/watchdogdownloader/examples/ccplus_1x.manifest.tsv +wdd init \ + /srv/codecontestplus-state \ + /srv/codecontestplus-files \ + ./watchdogDownloader/examples/ccplus_1x.manifest.tsv ``` - -This does not affect the older hand-written downloader currently running under `/data/whn/raw/codecontestplus`.