Skip to content

fix(trainer): report a missing TrainJob as not found instead of unowned - #158

Open
Krishna-kg732 wants to merge 1 commit into
kubeflow:mainfrom
Krishna-kg732:fix/trainer-ownership-not-found
Open

fix(trainer): report a missing TrainJob as not found instead of unowned#158
Krishna-kg732 wants to merge 1 commit into
kubeflow:mainfrom
Krishna-kg732:fix/trainer-ownership-not-found

Conversation

@Krishna-kg732

@Krishna-kg732 Krishna-kg732 commented Aug 12, 2026

Copy link
Copy Markdown
Member

Description

  • is_mcp_managed returned False for a TrainJob that does not exist, the same value it returns for one that exists without the MCP ownership label.
  • delete_training_job and update_training_job gate on ownership before existence, so a mistyped name was reported as VALIDATION_ERROR: "was not created by MCP" with a permissions hint, instead of not found.
  • The same missing job answered differently depending on the caller: platform-admin skips the gate and correctly got RESOURCE_NOT_FOUND, while every other persona got the misleading error.
  • Replaced with get_trainer_ownership, returning managed, unmanaged or missing. Missing now maps to RESOURCE_NOT_FOUND, and the ownership message is reserved for jobs that actually exist.
  • Both call sites now share one _require_mcp_ownership helper, removing duplicated error construction that had already drifted between the two tools.

No behaviour change for managed, unmanaged or API-failure outcomes.

Related Issue

No existing issue. Found while exercising the Katib optimizer client against a live cluster.

Checklist

  • I have read the CONTRIBUTING guide
  • Tests pass locally (make test-python)
  • Linting passes (make verify)
  • Documentation updated (if applicable)
  • My commits are signed off (git commit -s)

Testing

  • Unit tests

  • Integration tests

  • E2E tests

  • Manually tested (describe below)

  • Added test_missing_job_end_to_end_from_a_real_404, which mocks only the Kubernetes API so the real ownership helper runs. The other ownership tests stub the helper, so they pin the gate's branching but cannot catch this class of regression.

  • Verified the coverage actually bites: reintroducing the bug fails 3 tests, and all pass with the fix.

  • Reproduced the original behaviour first by driving both tools with a mocked 404 across all three personas.

  • Fills two existing TODO(test) markers in trainer/api/lifecycle_test.py.

  • make test-python: 485 passed. make verify: all checks passed.

/kind bug
/area trainer

Signed-off-by: krishna-kg732 <krishnagupta.kg2k6@gmail.com>
Copilot AI balanced review requested due to automatic review settings August 12, 2026 11:13
@google-oss-prow google-oss-prow Bot added kind/bug Something isn't working area/trainer TrainerClient tools labels Aug 12, 2026
@google-oss-prow
google-oss-prow Bot requested a review from jaiakash August 12, 2026 11:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@google-oss-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign andreyvelich for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow
google-oss-prow Bot requested a review from szaher August 12, 2026 11:13
@abhijeet-dhumal

Copy link
Copy Markdown
Member

/ok-to-test

@google-oss-prow google-oss-prow Bot added the ok-to-test Approve CI for external contributors label Aug 14, 2026
@abhijeet-dhumal

Copy link
Copy Markdown
Member

Thanks @Krishna-kg732, Nice catch and a clean fix,.. I guess the persona-dependent answer for a mistyped name was actively misleading agents. This is exactly what we needed 🎉

("update", lambda: update_training_job("external", "suspend"), "delete_job"),
],
)
def test_non_admin_blocked_from_mutating_external_job(tool, call, attr):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the update row still passes attr="delete_job", but update_training_job never calls client.delete_job.. it just patches via the custom-objects API. The patch assert below already covers it..
either drop attr for the update case or only assert delete_job on the delete row.

):
result = call()

assert result["error_code"] == "RESOURCE_NOT_FOUND"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: prefer RESOURCE_NOT_FOUND / SDK_ERROR from tests.common (same as VALIDATION_ERROR already imported) instead of raw strings.

@abhijeet-dhumal

Copy link
Copy Markdown
Member

@Krishna-kg732 are you still working on this ? Did you get time to resolve nit picks raised ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/trainer TrainerClient tools kind/bug Something isn't working ok-to-test Approve CI for external contributors size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants