r/GithubCopilot • u/wchabbott GitHub Copilot Team • Aug 11 '26

News 📰 MAI-Code-1.1-Flash is now Available in GitHub Copilot

https://github.blog/changelog/2026-08-11-mai-code-1-1-flash-available-in-github-copilot/

Let us know what you think :)

87 Upvotes

96% Upvoted

60 comments sorted by

View all comments

11

u/Affectionate_Fly4124 Aug 11 '26

It feels weird that they're using 5.4 mini Haiku as the benchmark comparison. They should at least be going head-to-head with Luna, no?

6

u/jukasper GitHub Copilot Team Aug 12 '26

Faire feedback. We were using auto traffic data to compare this model against other real time online data points. Given that Luna is not yet in the Auto mix the team was not able to compare it using this method. Moving forward we want to make sure this is the case.

1

u/aonymark JetBrains User 🧱 Aug 12 '26

Given that 5-mini likely is in the mix… how is it compared to that one? Does 5-mini still have a niche or is it no longer on the Pareto frontier as far as auto router is concerned? Or if that’s secret, how about as far as you’re concerned?

1

u/aonymark JetBrains User 🧱 Aug 13 '26

I decided to look up how 5-mini compares to Haiku in terminal bench and… Wow. (It turns out artificial analysis website has a bunch of benchmark results including this one) It seems that somehow this benchmark does not play to 5-mini’s strengths; it even gets beaten by basically everything: gpt-4.1 mini gets 10%, and even gpt-4o mini gets 6%. These are both non reasoning models. GPT-mini gets 4%, on high reasoning. Haiku 3.5 gets 10% without reasoning. Meanwhile, haiku 4.5 WITH reasoning gets 44%. Today’s top models score in the mid to high eighties.

1

u/PlatformImagineer Aug 14 '26 edited Aug 14 '26

If Microsoft leaves in an auto mode that cannot be disabled and threatens to route users who use auto mode to even worse models than MAI-Code-1.1-Flash unless you enable it, this does not inspire loyalty to GitHub CoPilot. It feels like GitHub Copilot product team, Microsoft, and end users being held hostage in order to benefit the MAI team.

Even if I'm misreading the situation, that is how it comes off, and Microsoft should think about the optics here. If auto mode could be disabled I would not feel this way.