Skip to content

create a registry of models used in wPY tests. - #1160

Draft
mborodii-prog wants to merge 2 commits into
mainfrom
feature/create-a-registry-of-models-used-in-wPY-tests
Draft

create a registry of models used in wPY tests.#1160
mborodii-prog wants to merge 2 commits into
mainfrom
feature/create-a-registry-of-models-used-in-wPY-tests

Conversation

@mborodii-prog

Copy link
Copy Markdown
Contributor

Add a distinct error for a model that doesn't exist, separate from access-denied

Problem

wrangles/data.py::_raise_model_response_error() mapped API responses to exceptions for 401/403, but had no 404 handling at all - any response outside 401/403 fell through to a generic RuntimeError('Something went wrong trying to {action} model {id}'), which doesn't distinguish a model that plain doesn't exist (bad ID, typo, deleted model) from a model that exists but the caller has no access to (AuthorizationError, "Access denied to model {id}"`).

This was part of investigating tests/connectors/test_train.py::TestTrainLookup::test_missing_columns_error_message intermittently surfacing Access denied to model bc3ee6a0-e104-4700 in a credentialed environment - the error message alone couldn't tell you whether that meant "fix the test's mocking" or "this model was deleted and the fixture needs replacing."

Fix

Added ModelNotFoundError and a 404 branch to _raise_model_response_error:

class ModelNotFoundError(RuntimeError):
    pass


def _raise_model_response_error(response, id: str, action: str) -> None:
    if response.status_code == 401:
        raise AuthenticationError(...)
    if response.status_code == 403:
        raise AuthorizationError(
            f"Access denied to model {id}. Check the user's model permissions."
        )
    if response.status_code == 404:
        raise ModelNotFoundError(
            f"Model {id} was not found. Check the model id is correct."
        )
    raise RuntimeError(f'Something went wrong trying to {action} model {id}')

This is the client-side half of a companion API-Core change that makes model_roles_required actually return 404 for a nonexistent model instead of a 403 indistinguishable from "exists but forbidden." Without that API-Core change, the backend still only ever sends 403 for both cases - this WranglesPY change is forward-compatible with it (a 404 from the API is now handled correctly) and is harmless before it (no behavior change for existing 403/401 responses).

Tests

Added test_model_endpoints_raise_not_found_error_for_404 to tests/test_data.py, parametrized across data.model, data.model_update, and data.model_content, mirroring the existing 401/403 tests. All 12 tests in tests/test_data.py pass.

Follow-up: model registry + preflight access check

While investigating the original failing test, also added:

  • tests/model_registry.py - a registry of model IDs referenced across the test suite (seeded by scanning tests/**/test_*.py, 39 entries, with two known-mocked-only IDs excluded and a note that it needs further human review for others).
  • scripts/check_model_registry_access.py - checks access to every registered model up front, using the new ModelNotFoundError distinction to report "not found" vs "access denied" separately, exiting non-zero if anything is inaccessible.
  • Wired as a "Check model registry access" step in .github/workflows/ci.yml, before the main credentialed pytest run, so a broken model fixture fails fast with a clear report instead of a cascade of individual test failures.
  • tests/test_check_model_registry_access.py - fully mocked unit tests for the checker's categorization logic (4 tests, no network).

@mborodii-prog mborodii-prog linked an issue Sep 1, 2026 that may be closed by this pull request
2 tasks
@mborodii-prog
mborodii-prog marked this pull request as draft September 1, 2026 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ensure Test Models are Still Avaialble

1 participant