Skip to content

feat(eval-author): add MLflow to ATIF converter - #1740

Draft
asutermo wants to merge 1 commit into
mainfrom
mlflow-to-atif/asutermorris
Draft

feat(eval-author): add MLflow to ATIF converter#1740
asutermo wants to merge 1 commit into
mainfrom
mlflow-to-atif/asutermorris

Conversation

@asutermo

@asutermo asutermo commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a direct, privacy-conscious path from canonical MLflow traces to ATIF v1.7 for Harbor and Eval Author audit coverage. Previously, MLflow traces had to be routed through another trace store or converted ad hoc; this PR provides a documented, deterministic converter for exported or bounded live traces.

Changes

  • Add the user-invocable mlflow-to-atif skill with offline and live-query workflows.
  • Convert one MLflow Trace.to_dict() value, an array of those values, or a { "traces": [...] } export into one ATIF v1.7 file per trace.
  • Reject incomplete paginated exports, unresolved parent references, and cyclic span graphs instead of silently emitting partial or invalid trajectories.
  • Distinguish missing, explicit-null, and populated tool outputs in namespaced result metadata and loss codes.
  • Support bounded live queries through mlflow.search_traces using an explicit experiment and time range.
  • Project LLM spans into agent steps and tool-like spans into paired ATIF tool calls and observations.
  • Preserve trace information, assessments, and raw spans under namespaced MLflow metadata while recording known lossy projections.
  • Preserve available model, token, cache, and cost metrics.
  • Write owner-private output directories and files, print only content-free summaries, and refuse overwrites by default.
  • Add optional Harbor validation and document downstream ATIF v1.7 compatibility.
  • Add contract coverage for accepted export shapes, graph and pagination validation, output states, permissions, required human input, overwrite protection, and Harbor validation.
  • Clarify the routing boundary between Eval Author, NeMo Intake, and direct MLflow-to-ATIF conversion.
  • Keep output on ATIF v1.7; note that ATIF v1.8 should be adopted together with downstream consumers.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with documentation updates
  • Documentation only
  • Contributor tooling or automation
  • CI, build, or test infrastructure

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Documentation updated for user-visible behavior
  • Documentation not applicable — justification:

Verification

  • Pull request title follows the repository's Conventional Commit format
  • Every commit includes an appropriate Signed-off-by: trailer
  • uv run pre-commit run -a passes, or any blocked checks are identified below
  • Targeted tests pass, or tests are marked not applicable above
  • No secrets, API keys, or credentials are included

Targeted validation:

  • flox activate -- uv run --frozen pre-commit run -a — passed all hooks.
  • flox activate -- uv run --frozen pytest -q plugins/nemo-eval-author/tests — 133 passed.
  • Converted the repository's existing MLflow fixture with --validate-with-harbor — passed as ATIF v1.7 with no validation errors.
  • Sampled four privately held traces across S01, S21, S30–S37, and S39 after transient recovery into canonical MLflow shape. All four converted through the bare-object input path and passed Harbor validation. No adapter for the source package's custom export envelope is included in this PR.

@github-actions github-actions Bot added the feat label Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor
Suite Lines Covered Line Rate Branch Rate
Unit Tests 38449/49020 78.4% 62.5%
Integration Tests 23138/46258 50.0% 22.7%

Signed-off-by: Andrew Suter-Morris <asutermorris@nvidia.com>
@asutermo
asutermo force-pushed the mlflow-to-atif/asutermorris branch from 9be4381 to 18a01c4 Compare September 2, 2026 21:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant