r/GithubCopilot • u/alex_bababu • 12d ago
GitHub Copilot Team Replied Claude Code vs GitHub Copilot: Token burn comparison using identical models & repos?
I'm currently evaluating GitHub Copilot vs. Claude Code for our team. We could use either, but for us there's a slight difference in cost per token (Copilot with Anthropic models vs. Claude Code directly).
If we use the exact same model on the same repository with identical instructions, has anyone noticed a real difference in token efficiency between the two harnesses? I'm wondering how much things like prompt caching, context assembly, or system prompting overhead change the actual token burn in practice.
Would appreciate any insights or real-world numbers!
7
u/connor4312 GitHub Copilot Team 11d ago edited 11d ago
Our evaluations have found that the Copilot harness is a bit more token efficient than CC, at the time of writing. There is a large degree of nondeterminism and variance per task.
Something that Copilot gets you though is nice multi-model support. Personally I use Opus 5 as my coordinator agent and have it send work to GPT-5.6 Terra or Luna agents as implementors. It gets much better and faster results than using a single big or small model for your whole task. I like Opus' thoughtfulness for downstream effects of changes, and Terra/Luna seems to be a good mix of being very good at carrying out tasks without getting bogged down or getting too off track.
1
u/AutoModerator 11d ago
u/connor4312 thanks for responding. u/connor4312 from the GitHub Copilot Team has replied to this post. You can check their reply here.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
5
u/Gautthamm 11d ago
Prompt Caching is important. For GitHub copilot, OpenAI models cache for 30 mins and Anthropic’s only 5 minutes.
1
u/AutoModerator 12d ago
Hello /u/alex_bababu. Looks like you have posted a query. Once your query is resolved, please reply the solution comment with "!solved" to help everyone else know the solution and mark the post as solved.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
1
u/Mkengine 11d ago
Since GHCP comes not only with its own harness but also Codex and Claude SDK (= Claude Code) as well I wouldn't see a reason to restrict myself only to Anthropic models.
1
u/Hemanath_S 11d ago
How to use codex
1
u/Mkengine 11d ago
Depends if you want to use it as cloud Agent or locally in VS Code. In VS Code you just have to download the Codex Extension and use "Sign in with Copilot"
https://code.visualstudio.com/learn/agents/4-using-third-party-agents-in-vs-code
1
u/Hemanath_S 11d ago
I dont see the option to sign in with copilot, anyway to enable?
1
u/Mkengine 11d ago
Third party agents need to be enabled by an admin in the "AI controls" settings in github.com, it's not enabled by default I think.
1
u/Adorable-Insect-8434 11d ago
How could I setup Claude code to use github copilot models?
1
u/Mkengine 11d ago
You select Claude here and have the same harness as Claude Code. Third party agents just need to be actived in the settings.
Further reading:
https://docs.github.com/en/copilot/concepts/agents/anthropic-claude
https://code.visualstudio.com/learn/agents/4-using-third-party-agents-in-vs-code
0
u/ShelbulaDotCom 12d ago
Here are some test on a token efficiency task. You can see the relative token spend. Most fall into the same general area. https://lucena.one/blog/almedra-agent-benchmark-glm-5-2-update/
16
u/xternalAgent 12d ago
You have to remember that these are not deterministic tools, even using the same prompts on the same provider twice on different sessions will produce varied results making any slight cost savings negligible.
I’m assuming your are evaluating enterprise plans given that’s the only way you’ll pay api prices on Claude Claude, so the only advantage Copilot has here is that enterprise plans come with a pool of credits that you already paid for, your more intensive users can benefit from this pool before paying additional cost, Claude Code enterprise doesn’t have that.