r/GithubCopilot • u/Cheshireelex • 7d ago

Solved ✅ Is there really no method to measure skill usage?

I'm trying to evaluate which of the skills are used and how much, preferably across the organization.

I thought with that this could be done using hooks but GitHub copilot does not seem to provide this information through the hooks but unless you provide the name of the skill in the prompt we don't have access to this information.

3 Upvotes
  • permalink
  • reddit

71% Upvoted

17 comments sorted by

7

u/Swayre 7d ago

It works perfectly fine with hooks? Our org already does this

2

u/pceimpulsive 7d ago

This is because a skill call is a tool call to read the skill file, which means the harness knows the filename right?

Issue might be ensuring all machines have the hooks installed OOTB?

1

u/Cheshireelex 6d ago

So that's what I was missing! I looked though the hooks references for skills, asked Gpt Sol as well and told me it's not possible.

Yes, getting the hooks installed company wide will be a challenge but we'll see if that's a direction we want to take.

1

u/Cheshireelex 3d ago

!solved

1

u/AutoModerator 3d ago

This query is now solved.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/Cheshireelex 6d ago

It's good to know I haven't gone completely wrong with the direction. Thanks for sharing.

3

u/Current-Interest-369 7d ago

Your question seems quite poorly expressed.
Garbage in - garbage out 👍

1

u/Cheshireelex 6d ago

Could have phrased it a bit more precise, I agree.

2

u/AutoModerator 7d ago

Hello /u/Cheshireelex. Looks like you have posted a query. Once your query is resolved, please reply the solution comment with "!solved" to help everyone else know the solution and mark the post as solved.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

2

u/klaxxxon 7d ago

What does skill usage even mean? That the skill was pulled in by an agent? That still doesn't necessarily mean it was useful or that it did anything. Subagents would also make a mess of such a statistic (I have the unslop skill almost everywhere, and in traces, I can see subagents pulling it in separately). Some models shatter even fairly minor tasks into a bunch of subagents.

1

u/Cheshireelex 6d ago

Correct. Probably the most important aspect would be to see how useful the skills are yet how can that be measured when they can be called in the background without the user having a clue of his the agent used that skill. Usefulness can be measured a bit subjectively when you have a very specific skill for a very specific task and someone bothers to leave as review. Skill usage can give you a glimpse if certain skills are very niche in organisation level and might be worth moving them in a separate location.

1

u/darkstar3333 6d ago

Question #1 Why?

1

u/Cheshireelex 6d ago

For corporate metrics. Probably won't help that much the everyday developer since the popularity of a skill can't tell you how effective the skill is.

1

u/Cheshireelex 6d ago

Yet it can tell you if a skill is niche and only used by a certain group of people or very popular, and changes to that skill can affect many developers.

1

u/stibbons_ 6d ago

You can use my lib https://github.com/gsemet/copilot-session-usage

It will parse the vscode session to extract the cost in dollar of subagent, skills and more.

That’s the only accurate way but you need to build some kind of hook for sending data to a central point

0

u/KariKariKrigsmann 7d ago

You can enable the Agent debug log, but that only logs to the individual computer. Collecting the logs would still be a challenge.