Skip to content

feat: add run-end terminal summary (#262) - #296

Open
ny-shy wants to merge 7 commits into
inclusionAI:mainfrom
ny-shy:feat/run-end-summary-262
Open

feat: add run-end terminal summary (#262)#296
ny-shy wants to merge 7 commits into
inclusionAI:mainfrom
ny-shy:feat/run-end-summary-262

Conversation

@ny-shy

@ny-shy ny-shy commented Jul 28, 2026

Copy link
Copy Markdown

What this PR does

When a training run finishes, the only way to figure out what happened is to dig through logs — how long it ran, what the final loss was, how many samples were processed, why it crashed. This PR adds a structured summary printed at the end of every run, regardless of whether it succeeded, was interrupted with Ctrl-C, or failed with an exception.

The summary includes:

  • Outcome (success / interrupted / error)
  • Wall-clock duration
  • Final step and epoch
  • Sample counts (processed / trained / skipped)
  • Last recorded metrics
  • Up to 5 error messages

On failure, the original traceback is preserved — the summary prints after the traceback, so neither is lost.

Two CLI flags control it: --summary/--no-summary (default: on) and --summary-json (default: off, emits JSON instead of text). With no flags, behavior is unchanged.

Files changed:

  • New areno/cli/run_summary.py — data container + formatting + output, stdlib only
  • New tests/test_run_summary_cpu.py — 21 CPU tests
  • Modified trainer_config.py — added summary_enabled / summary_json fields
  • Modified 4 trainers (policy_only / sft / dpo / ppo) — try/except/finally in fit(), data collection in training loop
  • Modified train.py — CLI options, all 4 config construction sites updated
  • Modified docs/cli/observability.rst — user documentation

Related issue

Fixes #262 Print a structured terminal summary when a run ends

Type of change

  • New feature

How was it tested?

CPU unit tests on macOS (no GPU):

python3 -m pytest tests/test_run_summary_cpu.py -v
# 21 passed in 0.04s

Ran real SFT training on Kaggle T4 x2 to verify end-to-end behavior:

git clone -b feat/run-end-summary-262 https://github.com/ny-shy/AReno.git
cd AReno
pip install -e . --no-build-isolation

PYTORCH_ALLOC_CONF=expandable_segments:True areno train \
  --algo sft \
  --ckpt Qwen/Qwen3-0.6B \
  --dataset-path yahma/alpaca-cleaned \
  --dataset-loader-fn examples/sft/alpaca/dataset_loader.py \
  --model-hub hf \
  --tp-size 2 \
  --world-size 2 \
  --batch-size 1 \
  --mini-bs 1 \
  --max-prompt-tokens 128 \
  --max-new-tokens 64 \
  --max-steps 3 \
  --activation-checkpointing \
  --disable-thinking \
  --summary

Three scenarios were observed:

  1. Success — 3 steps completed, outcome=success, metrics showed final loss=1.17, grad_norm=129.18, lr=9.9998e-07. Values matched train_stats in the training logs for the last step.

  2. OOM — CUDA out of memory after 1 step (single T4, batch_size=2). outcome=error, error message captured and truncated in "Errors (bounded)" section, full traceback printed after the summary block. Summary correctly reported steps=1, samples=2 trained, and the metrics from step 0.

  3. All data filtered--max-prompt-tokens 16 --max-new-tokens 8 caused all 51,760 rows to be skipped. outcome=error, steps=0, metrics showed (no metrics recorded), samples all zero. Error message: ValueError: SFT dataset produced no valid training rows.

Summary output was correct in all three cases. No local GPU available, so full training was only validated on Kaggle T4 x2.

Checklist

  • PR title summarizes the contribution
  • Linked the related issue
  • Existing tests pass (pytest tests/test_run_summary_cpu.py -v)
  • New behavior is covered by tests
  • Described test commands and hardware limitations
  • CLI changes are additive and backward-compatible

shy and others added 7 commits July 28, 2026 13:58
Add structured summary printed to stderr when a training run ends,
covering success, interruption, and error outcomes. Includes duration,
final metrics, sample counts, and bounded error messages.

- New: areno/cli/run_summary.py (RunSummaryData, format/print functions)
- New: tests/test_run_summary_cpu.py (21 CPU tests)
- Modified: trainer_config.py (summary_enabled, summary_json fields)
- Modified: policy_only/sft/dpo/ppo trainers (outcome capture + summary)
- Modified: train.py (--summary/--no-summary, --summary-json CLI flags)
- Modified: docs/cli/observability.rst (user documentation)
- Default summary_enabled to False for backward compatibility
- Fix UnboundLocalError on empty train_batch in policy_only.py
- Add getattr fallbacks for args.summary/args.summary_json in
  _trainer_config_from_options to avoid breaking existing tests
- Move areno.init() and _ensure_roles() inside try block so init
  failures still produce a summary
- Wrap _print_run_summary and areno.close() in best-effort try/except
  in finally block to prevent hiding original exceptions
- Record total dataset size as samples_processed in SFT/DPO
- Fix documentation: use --ckpt/--dataset-path instead of
  --model/--dataset, update default to disabled
- Bound JSON errors to 5 entries, filter NaN/Inf from metrics,
  add allow_nan=False and fallback for non-serializable values
- Add trailing newline to run_summary.py (ruff W292)
- Fix SFT/DPO samples_processed double-counting: remove += len(train_batch)
  from training loop since len(dataset) is already set at start
- Add TestFormatFloat (5 tests) for direct _format_float coverage
- Add TestSummaryIntegration (5 tests) for end-to-end summary flow:
  success, error, JSON, disabled, NaN filtering
- Total: 31 passed
- Remove unused 'Any' import (ruff F401)
- Remove dead code: _outcome_label was identity mapping
- Fix print_run_summary enabled default to False (consistent with config)
- Fix PolicyOnly samples_processed inflation across epochs (only count first epoch)
- Fix SFT/DPO samples_skipped always 0: track skipped count from _iter_train_batches
- Add deterministic local example to documentation (tiny JSONL + loader script)
- Update tests to pass enabled=True explicitly

Total: 31 passed
P1: len(self.dataset) not safe for non-sized datasets
- Add safe_len() helper in run_summary.py (try/except TypeError)
- Replace len(self.dataset) with safe_len(self.dataset) in SFT and DPO

P2: finally block silently swallows exceptions
- Replace 'except Exception: pass' with logger.exception() in all 4 trainers

P3: doc/impl inconsistency on traceback vs summary order
- Update observability.rst to accurately describe the order:
  summary in finally, traceback printed after by interpreter

P4: JSON fallback path lacks logging
- Add logger.warning in format_run_summary fallback path

Tests:
- Add TestSafeLen (4 tests): list, generator, iterator, None
- Add TestSummaryOnFailure (4 tests): zero duration, no metrics,
  minimal data, JSON fallback returns valid JSON
- Total: 39 passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Print a structured terminal summary when a run ends

1 participant