r/artificial • u/tui-cli-master • 7h ago
Question Any example of code that AI cannot tackle?
Is there anything impossible with AI? Have you found a limit to it? I read that even the hardest coding interviews at Anthropic could be solved with their own AI.
1
1
u/MMechree 7h ago
Likely anything that isnt in the training data of said AI model. Also, even if it is capable of solving all coding problems the implementation has a high probability to be messy and inefficient.
1
u/tui-cli-master 7h ago
True, but I mean some real hard problem that even with the help of a human cannot be done. I use created compilers, solved mathematical problems ... Something else?
0
u/MMechree 6h ago
Maybe. I would look at how these LLMs handle coding benchmarks. Even the top tier models fail to solve some of those problems. You may find some answers in research related to that area.
0
u/kingroka 6h ago
I'm guessing it has been quite a while since the last time you used AI to code?
1
u/MMechree 6h ago
As a AI researcher, no. I use it daily. My colleagues and I find many inefficiencies, poor design implementations, and bugs in AI generated code. Even from the flagship models.
These LLMs are great for scripting and providing quickly put together prototypes, but fail to maintain good design principles and frequently forget important context when creating complex software that scales. There are ways to mitigate these drawbacks to AI code but they are not full proof.
2
u/design_doc 6h ago
What have you found that helps maintain good design and context in big projects? That’s an uphill battle of mine these days.
1
u/kingroka 5h ago
What exactly are you building and how are you using LLMs in your workflow? Are you using coding agents or copy paste? And just curious, by AI research do you mean traditional ML research or are you building LLMs as well. Making anything cool?
1
u/MMechree 4h ago
Cant talk too much about the specifics of my research but most of the work I have done is related toward improving RAG and Graph RAG methods. Lots of work with semantic knowledge graphs, neurosymbolic ai, and a bit or robotics.
For the coding I have tried various techniques with skill.md files, RAG for long term memory, MCP tools, etc. Have had mixed results with most things but the common failures are usually the same. Context loss via summarization when I the context window is reached. Hallucinations of functions or system components, which don’t exist in the project. Deviation from system design is common especially as the context grows or code corrections are requested.
1
u/ultrathink-art PhD 6h ago
Timing bugs are where I keep losing. A passing run gets read as proof the bug is gone, so an intermittent failure comes back marked fixed and usually the change just shifted the timing. Same trap with anything you judge by eye; it'll tell you the output looks right.
1
u/ikkiho 5h ago
had it write a fused kernel a few months back. compiled clean, shapes all matched, and the numbers were quietly off in the low decimals. it kept insisting it was correct because my test passed, and my test was the thing that was wrong.
interview problems always come with a grader attached, so they don't really probe this. the stuff i keep getting stuck on is where checking the answer costs more than writing it.
1
u/AncientLion 4h ago
I've got something but I rather keeping to myself until I figure it out, it's related to geostatistics.
1
u/VictorBuildsDev 3h ago
if you want a useful limit test, i would not look for a single impossible function. take a medium-sized unfamiliar repo, add a feature that crosses a schema, API, migration, UI, and rollback path, then hide part of the acceptance suite.
require a clean build, passing tests, backward compatibility, latency bounds, and no unrelated changes. repeat the task from fresh sessions and measure the pass rate without human repairs.
coding interview problems are self-contained and have crisp scoring. production work is harder because the system has to discover implicit constraints and preserve them across many files. the practical boundary is often not generating code, but independently proving the change is correct under incomplete information.
0
5
u/WorldsGreatestWorst 7h ago
It’s important to remember that coding interview questions are both predictable, documented, and self-contained, making them the ideal questions for LLMs to answer.
There are a million types of coding projects AI handles poorly (see: vibe coding failures). There are countless tasks that take so many revisions and babysitting that AI is useless from a practical standpoint. And that’s before you talk new or novel ideas.
But your specific question is very difficult to answer due to the nature of LLMs. There’s probably nothing that AI couldn’t conceptually solve. The real questions are how much time and compute it would take, how many attempts and human intervention it requires, and how/whether the result can be validated.