feat: add run-end terminal summary (#262) - #296
Open
ny-shy wants to merge 7 commits into
Open
Conversation
Add structured summary printed to stderr when a training run ends, covering success, interruption, and error outcomes. Includes duration, final metrics, sample counts, and bounded error messages. - New: areno/cli/run_summary.py (RunSummaryData, format/print functions) - New: tests/test_run_summary_cpu.py (21 CPU tests) - Modified: trainer_config.py (summary_enabled, summary_json fields) - Modified: policy_only/sft/dpo/ppo trainers (outcome capture + summary) - Modified: train.py (--summary/--no-summary, --summary-json CLI flags) - Modified: docs/cli/observability.rst (user documentation)
- Default summary_enabled to False for backward compatibility - Fix UnboundLocalError on empty train_batch in policy_only.py - Add getattr fallbacks for args.summary/args.summary_json in _trainer_config_from_options to avoid breaking existing tests - Move areno.init() and _ensure_roles() inside try block so init failures still produce a summary - Wrap _print_run_summary and areno.close() in best-effort try/except in finally block to prevent hiding original exceptions - Record total dataset size as samples_processed in SFT/DPO - Fix documentation: use --ckpt/--dataset-path instead of --model/--dataset, update default to disabled - Bound JSON errors to 5 entries, filter NaN/Inf from metrics, add allow_nan=False and fallback for non-serializable values
- Add trailing newline to run_summary.py (ruff W292) - Fix SFT/DPO samples_processed double-counting: remove += len(train_batch) from training loop since len(dataset) is already set at start - Add TestFormatFloat (5 tests) for direct _format_float coverage - Add TestSummaryIntegration (5 tests) for end-to-end summary flow: success, error, JSON, disabled, NaN filtering - Total: 31 passed
- Remove unused 'Any' import (ruff F401) - Remove dead code: _outcome_label was identity mapping - Fix print_run_summary enabled default to False (consistent with config) - Fix PolicyOnly samples_processed inflation across epochs (only count first epoch) - Fix SFT/DPO samples_skipped always 0: track skipped count from _iter_train_batches - Add deterministic local example to documentation (tiny JSONL + loader script) - Update tests to pass enabled=True explicitly Total: 31 passed
P1: len(self.dataset) not safe for non-sized datasets - Add safe_len() helper in run_summary.py (try/except TypeError) - Replace len(self.dataset) with safe_len(self.dataset) in SFT and DPO P2: finally block silently swallows exceptions - Replace 'except Exception: pass' with logger.exception() in all 4 trainers P3: doc/impl inconsistency on traceback vs summary order - Update observability.rst to accurately describe the order: summary in finally, traceback printed after by interpreter P4: JSON fallback path lacks logging - Add logger.warning in format_run_summary fallback path Tests: - Add TestSafeLen (4 tests): list, generator, iterator, None - Add TestSummaryOnFailure (4 tests): zero duration, no metrics, minimal data, JSON fallback returns valid JSON - Total: 39 passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
When a training run finishes, the only way to figure out what happened is to dig through logs — how long it ran, what the final loss was, how many samples were processed, why it crashed. This PR adds a structured summary printed at the end of every run, regardless of whether it succeeded, was interrupted with Ctrl-C, or failed with an exception.
The summary includes:
On failure, the original traceback is preserved — the summary prints after the traceback, so neither is lost.
Two CLI flags control it:
--summary/--no-summary(default: on) and--summary-json(default: off, emits JSON instead of text). With no flags, behavior is unchanged.Files changed:
areno/cli/run_summary.py— data container + formatting + output, stdlib onlytests/test_run_summary_cpu.py— 21 CPU teststrainer_config.py— added summary_enabled / summary_json fieldstrain.py— CLI options, all 4 config construction sites updateddocs/cli/observability.rst— user documentationRelated issue
Fixes #262 Print a structured terminal summary when a run ends
Type of change
How was it tested?
CPU unit tests on macOS (no GPU):
python3 -m pytest tests/test_run_summary_cpu.py -v # 21 passed in 0.04sRan real SFT training on Kaggle T4 x2 to verify end-to-end behavior:
Three scenarios were observed:
Success — 3 steps completed,
outcome=success, metrics showed final loss=1.17, grad_norm=129.18, lr=9.9998e-07. Values matchedtrain_statsin the training logs for the last step.OOM — CUDA out of memory after 1 step (single T4, batch_size=2).
outcome=error, error message captured and truncated in "Errors (bounded)" section, full traceback printed after the summary block. Summary correctly reportedsteps=1,samples=2 trained, and the metrics from step 0.All data filtered —
--max-prompt-tokens 16 --max-new-tokens 8caused all 51,760 rows to be skipped.outcome=error,steps=0, metrics showed(no metrics recorded), samples all zero. Error message:ValueError: SFT dataset produced no valid training rows.Summary output was correct in all three cases. No local GPU available, so full training was only validated on Kaggle T4 x2.
Checklist
pytest tests/test_run_summary_cpu.py -v)