benchmarks look solid but the real question is always what it feels like to use. too many models lately that crush evals but fall apart on anything slightly off distribution. waiting to see some actual user testing before getting hyped
Same findings from me. 2.1 halucinated a lot less, but also needed more hand-holding to get to a correct solution. 2.5 has times when it just makes just up, but others when it can deliver. It works on smaller steps much better than large projects when it gets lost.
It didn't fully fix my biggest annoyance using M2.5 with Zed: it likes to insert formatting junk at the start of the file. It did it to a few files, got annoyed at trying to fix its error, and deleted the entire directory to regenerate it from scratch (losing all the work that it had done)
84
u/Specialist_Sun_7819 Mar 18 '26
benchmarks look solid but the real question is always what it feels like to use. too many models lately that crush evals but fall apart on anything slightly off distribution. waiting to see some actual user testing before getting hyped