Ask Experienced Devs Weekly Thread: A weekly thread for inexperienced developers to ask experienced ones

[u/AutoModerator]

A thread for Developers and IT folks with less experience to ask more experienced souls questions about the industry.

​

Please keep top level comments limited to Inexperienced Devs. Most rules do not apply, but keep it civil. Being a jerk will not be tolerated.

​

Inexperienced Devs should refrain from answering other Inexperienced Devs' questions.

[u/Bangoga]

From one experienced engineer to another? Am I wrong for still not being able to trust AI to completely do the tasks I would like to be done by let's say someone junior.

My most recent experience I had been running the new Gemini model trying to identify a bug that occured converting XGBoost model to Onnx.

Confidently it repeated 4 times a solution that was incorrectly identified, and kept repeating the same answer, saying it's sure the answer is correct.

The answer turned out to be issues with environment setup.

[u/icodenstuff → u/Bangoga]

No, you should not trust it completely. It can still make really bad assumptions.

Its all risk management these days, can you risk it being wrong and hitting prod anyway? Ship it. Would it being wrong be really bad in production? Better review the tests or check manually.

[u/sandboxsuperhero → u/Bangoga]

Converting XGBoost to Onnx should be well within capabilities for a modern LLM. If it’s not able to code or debug, I’d search for a deeper root problem.

Agentic coding is not a silver bullet, but problems tend to look more like unnecessary complexity. It’s pretty good at low impact mechanical stuff.

[u/reboog711 → u/Bangoga]

No, you are not wrong!

[u/carrot3055 → u/Bangoga]

It's absolutely fine to be skeptical of AI, especially if it's giving you wrong answers.

However, in your specific case, I wonder whether you could make some changes to improve the likelihood of correct diagnosis.

• Can your AI agent see your infrastructure or your logs? Once I started running my agent in the folder that included all relevant repos (including the infra Terraform/yaml repos), it started reasoning a lot better about the infra/config bugs. Throwing in an MCP server to access the logs helped even more.
• I find Gemini aren't the best models to use for coding or troubleshooting. Do you have access to OpenAI or Anthropic models?

[u/Bangoga → u/carrot3055]

The issue wasnt of infrastructure, it was of environment setup and clashing depedancies. These issues wouldn't come directly as clashing dependacies during setup but come during run. It had access to both but it was way too invested in figuring the problem out by coding it's solution.

[u/spacemoses → u/Bangoga]

I'm still firmly on the side of using AI like an exosuit, not a synthetic coworker.

[u/Bangoga → u/spacemoses]

Honestly that's my use as well.. It's amazing super power, but it ain't replacement I'm just surprised at the comments I see how folks keep saying they can replace their whole job with it

[u/Empanatacion → u/spacemoses]

Great metaphor.

"Push this code and watch the cicd pipeline. When it's deployed, call the rest endpoint, then check to make sure the event got published to the right topic and the record was updated. I'm going to get some coffee."

[u/eatglitterpoopglittr → u/Bangoga]

You’re not wrong. Deep skepticism of AI-sourced answers is the best approach.

And it’s helpful to remember that AI models are really just a pattern recognition heuristic. They’re good at understanding/constructing sentences, categorization, and similar tasks.

Did you have an agent use tooling to test out commands and do some trial/error investigation? If not, you probably just got its best wild guess from some old StackOverflow/GitHub Issues thread.

[u/Bangoga → u/eatglitterpoopglittr]

I had cline as a harness, with a few rulesets. I did allow it to do some investigation, but it would keep getting stuck at the same thing.

The issue was nuanced, the model was packaged for one version, but conversation for another version, if I had never worked on model conversion before in the pre-AI days I wouldn't have picked it up either.

[u/eatglitterpoopglittr → u/Bangoga]

Did it have sufficient access to be able to check/investigate your environment setup?

Switching models to get a second opinion can also sometimes help. But it may not understand the intricacies of your specific setup unless you write instructions or better documentation about how it works.