Skip to content

feat: persist pragmata-owned train metadata#271

Merged
saschagobel merged 5 commits into
mainfrom
feat/eval-train-pragmata-meta
Jul 8, 2026
Merged

feat: persist pragmata-owned train metadata#271
saschagobel merged 5 commits into
mainfrom
feat/eval-train-pragmata-meta

Conversation

@saschagobel

Copy link
Copy Markdown
Collaborator

Summary

Persists pragmata-owned metadata for completed eval training runs.

This is needed because tlmtc owns the training artifacts but has no concept of pragmata eval tasks. By writing a small pragmata_train.meta.json sidecar with the run_id, task, and optional annotation_export_id, later predict workflows can select the latest compatible model for a task and validate explicit run IDs against the requested task.

Key changes

  • Add EvalTrainMeta to src/pragmata/core/schemas/eval_output.py.
  • Add resolve_eval_train_meta_path to src/pragmata/core/paths/eval_paths.py.
  • Add src/pragmata/core/eval/export.py.
  • Persist pragmata_train.meta.json after successful run_tlmtc_train.
  • Keep the public train_evaluator return value unchanged.
  • Store the sidecar beside the tlmtc train run artifacts under eval/train_outputs/<run_id>/.

Tests

  • Add unit tests for the eval train metadata schema.
  • Add unit coverage for the train metadata path resolver.
  • Add unit coverage for exporting the metadata sidecar.
  • Update eval API tests to cover metadata persistence orchestration.
  • Update eval integration tests to assert the sidecar is written for direct input, explicit export selection, and latest export selection.

@github-actions github-actions Bot added feature Adds or expands user-facing functionality size: M 200-499 LOC labels Jul 4, 2026
@saschagobel saschagobel marked this pull request as ready for review July 4, 2026 15:34
@saschagobel saschagobel added contracts Changes schemas, type contracts, or validation logic for inputs, outputs, and persisted artifacts paths Changes affecting filesystem layout, generated artifacts, metadata locations, path resolution api Changes affecting the public Python API or user-facing function signatures eval Changes affecting the evaluation tool labels Jul 4, 2026
@saschagobel saschagobel requested a review from lramir14 July 4, 2026 15:35

@lramir14 lramir14 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the nice work. I am only flagging a potential issue with the integration between Pragmata and TLMTC. "atomic_write_json" requires its parent directory to already exist, and every other caller in the repo satisfies that itself via an explicit ensure_dirs() on a pragmata-owned path bundle. export_eval_train_meta is the exception — it writes into train_outputs/<run_id>/, a directory pragmata never creates itself; it only exists because tlmtc happens to create it as a side effect of training. Would it be worth adding a defensive path.parent.mkdir(parents=True, exist_ok=True) before the write, so pragmata satisfies its own utility's contract directly instead of relying on tlmtc's internal behavior?

@saschagobel saschagobel force-pushed the feat/eval-train-pragmata-meta branch from 936f0c5 to 09f6d04 Compare July 8, 2026 07:30
@saschagobel

Copy link
Copy Markdown
Collaborator Author

Thanks for the nice work. I am only flagging a potential issue with the integration between Pragmata and TLMTC. "atomic_write_json" requires its parent directory to already exist, and every other caller in the repo satisfies that itself via an explicit ensure_dirs() on a pragmata-owned path bundle. export_eval_train_meta is the exception — it writes into train_outputs/<run_id>/, a directory pragmata never creates itself; it only exists because tlmtc happens to create it as a side effect of training. Would it be worth adding a defensive path.parent.mkdir(parents=True, exist_ok=True) before the write, so pragmata satisfies its own utility's contract directly instead of relying on tlmtc's internal behavior?

Good point about making the atomic_write_json precondition explicit. I avoided creating the parent directory here because train_outputs/<run_id>/ is the tlmtc-owned run directory; if it is missing after training returns, pragmata should fail clearly rather than create an orphan run directory. I added an explicit parent-dir check before writing the pragmata sidecar so the contract is local without taking over tlmtc directory ownership.

@saschagobel saschagobel merged commit 8edc52e into main Jul 8, 2026
4 checks passed
@saschagobel saschagobel deleted the feat/eval-train-pragmata-meta branch July 8, 2026 07:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api Changes affecting the public Python API or user-facing function signatures contracts Changes schemas, type contracts, or validation logic for inputs, outputs, and persisted artifacts eval Changes affecting the evaluation tool feature Adds or expands user-facing functionality paths Changes affecting filesystem layout, generated artifacts, metadata locations, path resolution size: M 200-499 LOC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants