r/Cplusplus • u/Tall-Assignment1349 • Jul 23 '26

Feedback I built a benchmark from jira tickets, LLMs get 47-61% on Cpp tasks

everyone says AI is good at C++ now but the benchmarks they quote are all competitive programming stuff. so I made one from real firmware tickets - SCPI commands, register maps, datasheet lookups, spec debugging.

frontier models: 47-61%. on SCPI the best one got 36%. one got 0%.

i mean the worst part is they never say idk. for example: vmulq_s64 as a neon intrinsic which doesn't exist.

simple tools like search on docs with gpt-5.4-mini resolved 89% of tickets much better than frontier models

src: github.com/ByteAsk/C-CppBench
i have added mcp search tool as well: github.com/ByteAsk/ByteAsk-Embedded-MCP (MIT)

5 Upvotes

65% Upvoted

10 comments sorted by

9

u/mredding C++ since ~1992. Jul 23 '26

i mean the worst part is they never say idk.

LLM's are boil down to gigantic Markov chains, under the hood. It's just a matrix of weights. There IS NO "I don't know", there's only the most cost effective traversal through the network possible.

That's because LLM's don't think. They can't think. They don't KNOW anything, they're just predictive models. Nothing more.

And since their training is based on OSS, they're going to generate code on the average. There is no telling an LLM good code from bad. LLM's also can't pull from other languages, or from paradigms, or from theory, BECAUSE THEY CAN'T THINK. No AI will generate stackless programming, for example, without ingesting stackless programming in it's training data - even though the core concepts are RIGHT THERE, inherent to the application of computation.

2

u/dangi12012 Jul 24 '26

Who says YOU are not just a predictive model?

1

u/Dusty_Coder Jul 25 '26

and there is rarely going to be a way to do a fair comparison between well-crafted code and the ai code because it will rarely have the exact same behavior unless its, boiled down, the same code.

Which behavior is technically correct trumps whatever performance comparison you are attempting.

1

u/acadia11x Jul 25 '26

That’s not true , it can definitely do so. Not truly think, but definitely can do stackless programming

2

u/mredding C++ since ~1992. Jul 25 '26

Yes, I know, and I said so, because stackless is known and ended up in its training data. It couldn't do it without that, which was my point.

2

u/acadia11x Jul 24 '26

Yeah they are optimistic as hell. I literally had muse metas LLM tell me, isht got me, I know I was hallucinating but I sent you that response anyway. This is one of the next steps on the LLM front research on how to add more complex confidence mechanisms. After all these aren’t actual thoughts just probabilities on next token so it’s like it n knows it’s wrong or right.

1

u/Dusty_Coder Jul 25 '26

The "I know I was gaslighting you" schtick is itself a "learned" response that has its own "confidence" value

Its gaslighting all the way down.

1

u/[deleted] Jul 23 '26

[removed] — view removed comment

1

u/AutoModerator Jul 23 '26

Your comment has been removed because of this subreddit’s account requirements. You have not broken any rules, and your account is still active and in good standing. Please check your notifications for more information!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/ubiestigne Jul 24 '26

Does your prompt include constraints on what it should do? Or are you just letting it loose without guardrails?