-
Notifications
You must be signed in to change notification settings - Fork 34
Description
Bug Description
The leaderboard validation workflow fails for any submission where the assessed repository's .repository.url field uses the SSH format (git@github.com:org/repo.git) instead of HTTPS (https://github.com/org/repo.git). The ORG_REPO extraction uses sed that only strips https://github.com/ prefixes, leaving SSH URLs unparsed and causing downstream API calls to fail with invalid repo paths.
## To Reproduce
- Clone a repository that has an SSH remote (e.g. git@github.com:feast-dev/feast.git)
- Run agentready assess against it
- Run agentready submit to create a leaderboard PR
- Observe the "Verify submitter has access" step fails with Not Found (HTTP 404)
Expected Behavior
The workflow should correctly parse both SSH (git@github.com:org/repo.git) and HTTPS (https://github.com/org/repo.git) URLs from the assessment JSON, extract org/repo, and successfully validate repository access.
Actual Behavior
The sed command passes the raw SSH URL through unchanged, resulting in API calls to invalid paths like /repos/git@github.com:feast-dev/feast/collaborators/username, which return 404. The workflow then incorrectly reports the submitter has no access to the repository.
Additionally, the "Post validation results" step fails with a 403 (Resource not accessible by integration) on fork PRs because the pull_request trigger provides a read-only GITHUB_TOKEN that cannot post comments.
Environment
- OS: Ubuntu 24.04 (GitHub Actions runner)
- Version: 2.27.4
- Python Version: 3.12
Additional Context
Affected PRs:
- Leaderboard: feast-dev/feast (61.6/100 - Silver) #293 (feast-dev/feast, submitter: ntkathole)
- Leaderboard: opendatahub-io/ai-helpers (64.7/100 - Silver) #294 (opendatahub-io/ai-helpers, submitter: mprpic)
Both are cross-repository fork PRs. The failures are not related to the submitters' actual access to the target repositories. Previous submissions likely used HTTPS remotes, so this code path was never exercised until now.
Possible Solution
Fix (blocker): Add SSH URL handling to the sed pipeline in both the "Verify repository exists" and "Verify submitter has access" steps:
ORG_REPO=$(echo "$REPO_URL" | sed 's|git@github.com:||' | sed 's|https://github.com/||' | sed 's|.git$||')