Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
name: CI

on:
pull_request:
push:
branches: [main]

permissions:
contents: read

jobs:
dashboard:
name: Dashboard / Python ${{ matrix.python-version }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.10", "3.12"]
defaults:
run:
working-directory: dashboard
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: ${{ matrix.python-version }}
- name: Install test dependencies
run: python -m pip install -e '.[test]'
- name: Run tests
run: python -m pytest -q

watchdog-downloader:
name: watchdogDownloader / Bash
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- name: Check syntax
run: bash -n watchdogDownloader/wdd
- name: Check command entry point
run: watchdogDownloader/wdd --version
96 changes: 90 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,95 @@
# MyPracticalTools
收录了一些trivial,好用,可复用的个人化小工具

## 工具
[![CI](https://github.com/Whning0513/MyPracticalTools/actions/workflows/ci.yml/badge.svg)](https://github.com/Whning0513/MyPracticalTools/actions/workflows/ci.yml)

- [Practical Run Dashboard](dashboard/README.md):长时间训练/评测任务的
进度、检查点、ETA、阶段和 GPU 状态 TUI。
Small, reusable tools and reference assets for long-running ML, data, and
automation workflows.

## Tools
这是一个个人实用工具集合。每个工具都有独立文档和明确的使用边界,可以单独安装或复制,
不需要采用整个仓库的其他部分。

- [watchdogDownloader](watchdogDownloader/) - Automatic downloading with retrying and integrity checking.
## Included projects

| Project | Runtime | Purpose |
| --- | --- | --- |
| [Practical Run Dashboard](dashboard/README.md) | Python 3.10+ | Read-only terminal dashboard for checkpointed training and evaluation jobs, including progress, ETA, GPU use, and recoverable state. |
| [watchdogDownloader](watchdogDownloader/README.md) | Bash on Linux | Resumable manifest-based downloads with low-speed detection, process supervision, and size or SHA-256 verification. |
| [ACA small v0.2](datasets/ACA_small_v0.2/) | Zstandard JSONL | Versioned train/test dataset package with a manifest, blind-gate metadata, and audit reports. |
| [Dataset design notes](docs/dataset-and-datapackage-design.md) | Markdown | Reproducibility, split isolation, replay, validator, reference, and release requirements for the ACA data package. |
| [User workstyle skill](whn_skill/whn_skill.md) | Markdown | A Codex skill for evidence-driven coding, research, and technical communication. |

## Quick start

### Practical Run Dashboard

Install directly from this repository:

```bash
python -m pip install \
"git+https://github.com/Whning0513/MyPracticalTools.git#subdirectory=dashboard"
```

Copy and edit the example configuration, then start the live terminal view:

```bash
practical-dashboard --config dashboard/examples/aca-rl-matrix.json
```

See the [dashboard documentation](dashboard/README.md) for JSON output,
checkpoint semantics, GPU ownership, and expected status files.

### watchdogDownloader

Install the script into a directory on `PATH`:

```bash
git clone https://github.com/Whning0513/MyPracticalTools.git
install -Dm755 MyPracticalTools/watchdogDownloader/wdd "$HOME/.local/bin/wdd"
wdd --help
```

Create a tab-separated manifest, initialize a download project, and start it:

```bash
wdd init /srv/download-state /srv/files ./manifest.tsv
wdd start /srv/download-state
wdd watchdog-start /srv/download-state
wdd status /srv/download-state
```

See the [watchdogDownloader documentation](watchdogDownloader/README.md) for
the manifest format, retry behavior, verification, and tuning controls.

## Dataset package

`datasets/ACA_small_v0.2/` contains compressed train/test problems and
submissions. Use `manifest.json` as the versioned source of truth and inspect
`audit/report.json` before consuming the package. The accompanying
[design document](docs/dataset-and-datapackage-design.md) distinguishes frozen
benchmark evidence from replay output, constructed references, validators, and
blind probes.

## Development

Run the Python tests:

```bash
python -m pip install -e './dashboard[test]'
python -m pytest dashboard/tests -q
```

Check the Bash script syntax:

```bash
bash -n watchdogDownloader/wdd
watchdogDownloader/wdd --version
```

CI runs both checks for pull requests and changes to `main`.

## Contributing

Focused bug reports and pull requests are welcome. Include a minimal
reproduction, explain the behavioral change, and state the validation command.
Do not include credentials, private datasets, generated caches, or machine-
specific absolute paths.
49 changes: 25 additions & 24 deletions watchdogDownloader/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,20 +11,23 @@ It is intentionally conservative:
- A watchdog can restart the whole downloader process group if aggregate downloaded bytes stop increasing.
- Files are checked against expected byte sizes from a manifest.

## Install
## Requirements

The main command is:
`watchdogdownloader` targets Linux and requires Bash, `curl`, `setsid`,
`flock`, and standard GNU utilities such as `readlink`, `stat`, and `sha256sum`.

```bash
/data/whn/tools/watchdogdownloader/wdd
```
## Install

Optional convenience symlink:
Clone the repository and install `wdd` into a directory on `PATH`:

```bash
ln -sf /data/whn/tools/watchdogdownloader/wdd /data/whn/tools/wdd
git clone https://github.com/Whning0513/MyPracticalTools.git
install -Dm755 MyPracticalTools/watchdogDownloader/wdd "$HOME/.local/bin/wdd"
wdd --version
```

Alternatively, run `watchdogDownloader/wdd` directly from the checkout.

## Manifest

Create a tab-separated manifest:
Expand All @@ -48,40 +51,40 @@ Rules:
Create a project:

```bash
/data/whn/tools/watchdogdownloader/wdd init \
/data/whn/raw/mydownload \
/data/whn/raw/mydownload/files \
/data/whn/raw/mydownload/manifest.tsv
wdd init \
/srv/mydownload-state \
/srv/mydownload-files \
/srv/mydownload-state/manifest.tsv
```

Start the downloader:

```bash
/data/whn/tools/watchdogdownloader/wdd start /data/whn/raw/mydownload
wdd start /srv/mydownload-state
```

Start the watchdog:

```bash
/data/whn/tools/watchdogdownloader/wdd watchdog-start /data/whn/raw/mydownload
wdd watchdog-start /srv/mydownload-state
```

Check status:

```bash
/data/whn/tools/watchdogdownloader/wdd status /data/whn/raw/mydownload
wdd status /srv/mydownload-state
```

Verify after completion:

```bash
/data/whn/tools/watchdogdownloader/wdd verify /data/whn/raw/mydownload
wdd verify /srv/mydownload-state
```

Stop both watchdog and downloader:

```bash
/data/whn/tools/watchdogdownloader/wdd stop /data/whn/raw/mydownload
wdd stop /srv/mydownload-state
```

## Config
Expand Down Expand Up @@ -146,17 +149,15 @@ Use parallelism only for independent files. Do not use it to split a single file

An example manifest is included:

```bash
/data/whn/tools/watchdogdownloader/examples/ccplus_1x.manifest.tsv
```text
watchdogDownloader/examples/ccplus_1x.manifest.tsv
```

To create a fresh generic project for that dataset:

```bash
/data/whn/tools/watchdogdownloader/wdd init \
/data/whn/raw/codecontestplus_wdd \
/data/whn/raw/codecontestplus_wdd/ccplus_1x \
/data/whn/tools/watchdogdownloader/examples/ccplus_1x.manifest.tsv
wdd init \
/srv/codecontestplus-state \
/srv/codecontestplus-files \
./watchdogDownloader/examples/ccplus_1x.manifest.tsv
```

This does not affect the older hand-written downloader currently running under `/data/whn/raw/codecontestplus`.