Scorecard is a backstage plugin and supports multiple metric, grouped as modules in folder workspaces/scorecard/plugins.
Extend the existing GitHub plugin/module under plugins/github with a new GithubAiAdoptionMetricProvider.
-
getProviderDatasourceId: "github"
-
getProviderId: "github.aiAdoption"
-
getMetrics: 3 metrics, for the AI adoption rate for the last 7 days, 30 days and 90 days. ID should be calculated from this.getProviderId() + "Rate[7d]" (etc.). The threshold for all 3 metrics should be defined in the provider with:
export const AI_ADOPTION_RATE_TIME_RANGES = ["7d", "30d", "90d"];
export const AI_ADOPTION_RATE_THRESHOLD: ThresholdConfig = {
rules: [
{ key: 'success', expression: '>=0.2' },
{ key: 'warning', expression: '>=0.1' },
{ key: 'error', expression: '>=0' },
],
};
-
catalogFilter similar to the other GitHub providers:
getCatalogFilter(): Record<string, string | symbol | (string | symbol)[]> {
return {
'metadata.annotations.github.com/project-slug': CATALOG_FILTER_EXISTS,
};
}
-
calculateMetrics should request the commit history via GraphQL from GitHub for the biggest time range that is defined (currently 90 days).
Then it should take a look into all received commit messages and check if they was AI assisted or not. It could probably ignore merge commits.
It should check commit message lines that starts Assisted-by: or Co-Authored-By: (or Co-authored-by: ). When the value starts with a known AI tool, like Claude, Cursor it should count this message as AI assisted. Add extra conditions that identifies clearly AI assistents.
Log at the end how many commits was analyied, how many commits was ignored, marked as ai assisted commits and NOT marked as ai assisted commits.
Finally return the Map with the metric Ids and the ratio (0-1) per time range.
Scorecard is a backstage plugin and supports multiple metric, grouped as modules in folder
workspaces/scorecard/plugins.Extend the existing GitHub plugin/module under plugins/github with a new
GithubAiAdoptionMetricProvider.getProviderDatasourceId: "github"getProviderId: "github.aiAdoption"getMetrics: 3 metrics, for the AI adoption rate for the last 7 days, 30 days and 90 days. ID should be calculated from this.getProviderId() + "Rate[7d]" (etc.). The threshold for all 3 metrics should be defined in the provider with:catalogFiltersimilar to the other GitHub providers:calculateMetricsshould request the commit history via GraphQL from GitHub for the biggest time range that is defined (currently 90 days).Then it should take a look into all received commit messages and check if they was AI assisted or not. It could probably ignore merge commits.
It should check commit message lines that starts
Assisted-by:orCo-Authored-By:(orCo-authored-by:). When the value starts with a known AI tool, like Claude, Cursor it should count this message as AI assisted. Add extra conditions that identifies clearly AI assistents.Log at the end how many commits was analyied, how many commits was ignored, marked as ai assisted commits and NOT marked as ai assisted commits.
Finally return the Map with the metric Ids and the ratio (0-1) per time range.