feat(#4579): add GitHub AI adoption metric provider - #4580
feat(#4579): add GitHub AI adoption metric provider#4580fullsend-ai-coder[bot] wants to merge 3 commits into
Conversation
Add GithubAiAdoptionMetricProvider to the scorecard GitHub module. The provider calculates AI adoption rates for 7d, 30d, and 90d time ranges by analyzing commit trailers (Co-Authored-By, Co-authored-by, Assisted-by) for known AI tool identifiers (Claude, Cursor, Copilot, Codeium, Cody, Tabnine, Gemini, Amazon Q, Windsurf, Devin, Aider). Changes: - New GithubAiAdoptionMetricProvider with 3 metrics: github.aiAdoptionRate[7d], [30d], [90d] - New getCommitHistory method on GithubClient using GraphQL to fetch commit messages from the default branch - Merge commits are excluded from the ratio calculation - Provider registered in module.ts alongside GithubOpenPRsProvider - Config schema updated in config.d.ts for aiAdoption settings - Comprehensive test suite covering AI detection, merge commit filtering, time range calculation, and edge cases Closes #4579
|
Important This PR includes changes that affect public-facing API. Please ensure you are adding/updating documentation for new features or behavior. Missing ChangesetsThe following package(s) are changed by this PR but do not have a changeset:
See CONTRIBUTING.md for more information about how to add changesets. Changed Packages
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4580 +/- ##
==========================================
+ Coverage 62.37% 62.41% +0.03%
==========================================
Files 2607 2608 +1
Lines 104756 104865 +109
Branches 29430 29452 +22
==========================================
+ Hits 65346 65454 +108
- Misses 38819 38820 +1
Partials 591 591
*This pull request uses carry forward flags. Click here to find out more. Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
|
🤖 Finished Review · ✅ Success · Started 10:44 AM UTC · Completed 10:50 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Cost: $0.95 |
ReviewFindingsMedium
Low
Previous runReviewFindingsHigh
Medium
Low
Labels: PR adds a new feature to the scorecard workspace; enhancement label matches the linked issue classification. Previous run (2)Review — commentWell-structured addition that closely follows existing scorecard provider patterns. The Findings1. Missing
|
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
fullsend-ai-review
Bot
added
the
requires-manual-review
|
/fs-fix fix all review comments |
|
🤖 Finished Fix · ✅ Success · Started 3:43 PM UTC · Completed 3:56 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Cost: $1.95 |
- Add history: true to AI adoption metric definitions to match all other scorecard metric providers - Make trailer key matching fully case-insensitive by normalizing to lowercase before prefix comparison - Rename DEFAULT_DEPLOYMENT_FETCH_ITEMS_LIMIT to DEFAULT_FETCH_ITEMS_LIMIT to reflect its broader usage across deployments and commit history Addresses review feedback on #4580
|
🤖 Review · ❌ Terminated · Started 3:57 PM UTC · Ended 4:37 PM UTC Commit: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
fullsend-ai-review
Bot
added
enhancement
|
🤖 Finished Review · ✅ Success · Started 3:57 PM UTC · Completed 4:37 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Cost: $6.18 |
|
🤖 Finished Fix · ✅ Success · Started 4:38 PM UTC · Completed 4:57 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Cost: $4.48 |
- Rename GithubAiAdoptionMetricProvider to GithubAiAdoptionProvider to match the naming convention of GithubOpenPRsProvider - Fix false-positive AI detection by using exact name-part matching instead of startsWith, preventing human names like "Claude Smith" or "Devin Johnson" from being classified as AI-assisted - Add logger.warn when fetchItemsLimit is reached in getCommitHistory, matching the pattern used by getDeployments and getCommitShasBetween - Add logger.warn when defaultBranchRef is null (empty/misconfigured repo) - Remove explicit `: string` return type from getProviderId() to match sibling GithubOpenPRsProvider convention - Add comprehensive GithubClient.getCommitHistory tests covering single-page, multi-page pagination, fetchItemsLimit truncation, repository-not-found error, null node handling, and null defaultBranchRef - Add test for human-name false-positive prevention - Document AI adoption rate metrics in module README (IDs, thresholds, schedule configuration) - Update scorecard-backend README Available Metric Providers table - Add GithubAiAdoptionProvider to providers.md example list Addresses review feedback on #4580
|
🤖 Review · ❌ Terminated · Started 4:59 PM UTC · Ended 5:20 PM UTC Commit: |
|
Superseded by updated review
| history: true, | ||
| })); | ||
| } | ||
|
|
There was a problem hiding this comment.
[medium] naming-convention
Metric IDs github.aiAdoptionRate[7d/30d/90d] introduce bracket-parameterized suffixes not used by any existing metrics. Brackets could interact unexpectedly with downstream config paths, annotation keys, and translation key lookup.
Suggested fix: Consider plain CamelCase IDs (e.g., github.aiAdoptionRate7d) or verify downstream handling of brackets.
|
|
||
| export type GithubCommit = { | ||
| message: string; | ||
| committedDate: string; |
There was a problem hiding this comment.
[low] pattern-violation
GithubCommitHistoryQueryResponse uses optional-property markers (?) where existing response types use non-optional with | null. The codebase is already mixed on this pattern.
| repository: { | ||
| defaultBranchRef?: { | ||
| target?: { | ||
| history?: { |
There was a problem hiding this comment.
[low] code-organization
The nodes array element type is defined inline, duplicating the GithubCommit type declared five lines above.
Suggested fix: Reference GithubCommit: nodes: Array<GithubCommit | null>.
| @@ -0,0 +1,234 @@ | |||
| /* | |||
There was a problem hiding this comment.
[low] naming-convention
Class named GithubAiAdoptionProvider with lowercase i. Existing codebase preserves uppercase acronyms (GithubOpenPRsProvider, OpenSSFMetricProvider).
Suggested fix: Rename to GithubAIAdoptionProvider and file to GithubAIAdoptionProvider.ts.
| for (const line of lines) { | ||
| const trimmed = line.trim(); | ||
| const lower = trimmed.toLowerCase(); | ||
| let value: string | undefined; |
There was a problem hiding this comment.
[low] edge-case
isAiAssistedCommit scans every line for trailer patterns, not just the trailing block. Quoted trailers in revert descriptions could produce false positives (extremely rare in practice).
fullsend-ai-review
Bot
added
the
requires-manual-review
|
🤖 Finished Review · ✅ Success · Started 4:59 PM UTC · Completed 5:20 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Cost: $8.43 |
1 participant
Add GithubAiAdoptionMetricProvider to the scorecard GitHub module. The provider calculates AI adoption rates for 7d, 30d, and 90d time ranges by analyzing commit trailers (Co-Authored-By, Co-authored-by, Assisted-by) for known AI tool identifiers (Claude, Cursor, Copilot, Codeium, Cody, Tabnine, Gemini, Amazon Q, Windsurf, Devin, Aider).
Changes:
github.aiAdoptionRate[7d], [30d], [90d]
to fetch commit messages from the default branch
filtering, time range calculation, and edge cases
Closes #4579
Post-script verification
agent/4579-github-ai-adoption-metric)57947b408632c1ceba96ef8aa10f7efa903d3a83..HEAD)