Metrics API is_custom_model property

[monil-patel]

Describe the bug
I was staring at the response from the /copilot/metrics for our organization. We have deployed a model but the responses from the API show

      "is_custom_model": false,

Is this just an issue with the telemetry response? Is it possible to verify that people are correctly using our custom model?

longer snippet from the response

"copilot_ide_code_completions": {
      "editors": [
        {
          "name": "vscode",
          "models": [
            {
              "name": "default",
              "languages": [
                {
                  "name": "cpp",
                  "total_engaged_users": 1,
                  "total_code_acceptances": 8,
                  "total_code_suggestions": 39,
                  "total_code_lines_accepted": 5,
                  "total_code_lines_suggested": 44
                },
                {
                  "name": "powershell",
                  "total_engaged_users": 1,
                  "total_code_acceptances": 1,
                  "total_code_suggestions": 3,
                  "total_code_lines_accepted": 0,
                  "total_code_lines_suggested": 5
                }
              ],
              "is_custom_model": false,
              "total_engaged_users": 2
            }
          ],
          "total_engaged_users": 2
        },

Our org: https://github.com/FineTunedModelDogfoodersEMU

[monil-patel]

semi related - there seems to be a off by 1 on the "engaged" user property compared to the (older?) /usage endpoint?

left - /metrics
right - /usage

total_engaged for xml shows 0 on the right, which seemed odd

[kimyen]

Hello @monil-patel , please see the copilot/metrics API doc here: https://docs.github.com/en/rest/copilot/copilot-metrics?apiVersion=2022-11-28

In the example response, a custom model section should look like this:

          "models": [
            {
              "name": "a-custom-model",
              "is_custom_model": true,
              "custom_model_training_date": "2024-02-01",

This section shows that a model is a custom model, the name is not default and it has a model training date.

The snippet from the response you share is for a default model, not a custom model.

Is it possible to verify that people are correctly using our custom model?

Unfortunately, Custom Models response is not currently included in the copilot/metrics response. We are working on adding that back.

Our internal dashboard shows that your custom models are being used by 30-100 users per day.

For your question regarding total_engaged_users in Copilot Metrics API and active_users in Copilot Usage API, I will forward your question to the right team.

[monil-patel]

@kimyen thanks for the quick reply.

Our internal dashboard shows that your custom models are being used by 30-100 users per day.

I am a little confused then - When I run this from the terminal

 gh api -H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28" /orgs/FineTunedModelDogfoodersEMU/copilot/metrics

All of the responses have is_custom_model: false. Its slightly making me paranoid if our custom model is being used or if we are doing analysis on the right data as we try to gauge how well the model is working for us

[kimyen]

@monil-patel

All of the responses have is_custom_model: false.

Sorry for the confusion! As I mentioned above, the Copilot Metrics API currently does not include Custom Models in the response. The team is working on adding that back.

if we are doing analysis on the right data as we try to gauge how well the model is working for us

I would wait until the custom models information is returned by the Copilot Metrics API to do more analysis on Custom Models.

[monil-patel]

@kimyen sorry I'm getting tripped up a little bit:

So to clarify the Metrics API is only going to return the default model telemetry today? We had setup this organization specifically to have people onboarded onto the custom model so if they are getting a seat from this org why would they be using the default model?

Would the /usage endpoint have the same issue?

[kimyen]

@monil-patel

So to clarify the Metrics API is only going to return the default model telemetry today?

This is correct.

We had setup this organization specifically to have people onboarded onto the custom model so if they are getting a seat from this org why would they be using the default model?

There are a couple reasons why a user may be using a default model while they are assigned a seat from the org with Custom Models:

• They explicitly requests the default model by setting this up in the IDE advanced settings
• They request the Custom Models, but failed to authenticated their requests to the Custom Model for any reason, and is routed to the default model

With that said, since the data you are currently getting back only includes default model usage, it only reports these outliners. It's not a good data to draw any conclusions on Custom Models.

Would the /usage endpoint have the same issue?

I will forward this question to the right team as well.

[monil-patel]

Thanks for the info on why users still may be using the base model. Separately I would be interested in seeing if we can understand how common the auth issues are and how we can resolve it. But first would like to keep digging into this analysis question:

@kimyen I was staring more at the data between /metrics and /usage endpoints for the same day

PS C:\Users\mopatel> gh api -H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28" /orgs/FineTunedModelDogfoodersEMU/copilot/metrics > finetune_metrics.json
PS C:\Users\mopatel> gh api -H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28" /orgs/FineTunedModelDogfoodersEMU/copilot/usage > finetune_usage.json

Left - /metrics
Right - /usage

The language breakdown #s appear to match across both endpoints. Since you have confirmed that the /metrics endpoint only returns data for the default model. Then this leads me to believe that /usage is matching that as well.

If that is true, then I think there is some big misunderstandings across custom model pilot users that may be using this guidance which tells us to use /usage endpoint (separately the docs recommend upgrading to the /metrics endpoint, which is what led me down this rabbit hole :))

https://docs.github.com/en/enterprise-cloud@latest/copilot/customizing-copilot/creating-a-custom-model-for-github-copilot#assess-the-effectiveness-of-a-custom-model

Use the GitHub API to assess the usage of GitHub Copilot. See REST API endpoints for GitHub Copilot usage metrics.

cc @thejoebourneidentity

[kimyen]

If that is true, then I think there is some big misunderstandings across custom model pilot users that may be using this guidance which tells us to use /usage endpoint (separately the docs recommend upgrading to the /metrics endpoint, which is what led me down this rabbit hole :))

@monil-patel
It is the intention for the Copilot Metrics API to support custom model data. The custom model data were removed from the endpoint recently. And the team is working on adding that back. We will update our guidance accordingly.

Regarding you questions about total_engaged_users in Copilot Metrics API and active_users in Copilot Usage API, active users captures all users with any activity while total_engaged_users only count users who has accepted a suggestion.

[monil-patel]

It is the intention for the Copilot Metrics API to support custom model data. The custom model data were removed from the endpoint recently. And the team is working on adding that back. We will update our guidance accordingly.

@kimyen Thanks for bearing with me here I really appreciate the help

• Is there someone we can add to clarify the /usage response and if it includes the custom model? We have reports being built using that as a source of truth and i would like to get some confirmation if that data is accurately giving telemetry about the custom model or base model.

• Is there any ETA on when /metrics will include custom models information?

Thanks for the clarification on total_engaged_users vs active_users

[kimyen]

@monil-patel
I spotted check the data and learned that I made a wrong assumption about how the custom model information is "missing" from the response.

• What I thought (and is wrong): the response omitted the custom model activities all together and only report the default model model activities.
• What actually happens: the response includes all usage for your org, but does not separated the usage by model. All usage are under "default" model.

Is there someone we can add to clarify the /usage response and if it includes the custom model? We have reports being built using that as a source of truth and i would like to get some confirmation if that data is accurately giving telemetry about the custom model or base model.

The /usage API and the /metrics API reports activities for all models, including the custom model.

Is there any ETA on when /metrics will include custom models information?

The team is planning to re-introduce the separation of the default model and custom models in the /metrics API within the next month.

[monil-patel]

@kimyen Thank you for digging into this!

The team is planning to re-introduce the separation of the default model and custom models in the /metrics API within the next month.
Thank's will keep an eye out!

While we wait for that is there any good way at the moment to understand what % of people in this custom model org would be hitting the default model due to auth issues?

[kimyen]

While we wait for that is there any good way at the moment to understand what % of people in this custom model org would be hitting the default model due to auth issues?

In the past 30d, we only see 1 suggestion that is shown to the users in your org from the default model.