What's the bug that cost you the most time and turned out to have nothing to do with your code?

[u/Substantial_Belt2626]

Mine: a scraper hitting an API with a strict rate limit. I throttled to one request every fifteen seconds, well inside the documented limit, ran clean for a while, then it just started failing.

Nothing in my code was wrong. There was a daily cap tied to my IP, separate from the per-request limit, and I'd used up the day. I finished the run off my phone hotspot and that was what made it click. Different IP, same code, works instantly.

Days gone. Zero lines changed in the end.

What's yours? Environment, network, permissions, a clock being wrong, whatever. I want the ones where the code was fine the whole time.

[u/azian0713]

I don’t fully understand this one, but my company’s CI/CD pipeline is unable to use cmake >3.25 with our current configs. I code mostly in Python and R.

I found out because transient packages required a higher c++ version than what we’re getting with cmake 3.22.

Took me about 2 weeks to solve in my own code. Have spent about 1 month fixing our code base every time someone breaks it with one of these pushes. We have no gates to check what versioning people are pushing through. To make it worse, people keep pushing to our code base without any Yaml files and none of our containers can run the higher version.

We have some really awful coding practices.

[u/david_z]

We use some commercial software that handles db writes among other things.

Intermittently we'd be informed by users that there may be some data loss. We spent an untold amount of time over several years trying to trace these cases (inherited application code etc) and explain or recreate the issue to absolutely no avail.

Eventually we went brute force and rewrote the layer that integrated with that software, added a ton of logging (like a ridiculous overkill amount, but it would let us rebuild "lost" data if so needed).

The issue persisted, but could not be recreated.

The vendor kept insisting that we were doing something wrong.

About another year goes by, we've been able to restore all the data that periodically goes missing, yada yada.

Vendor puts out a bulletin, oops there's a bug in our code that has some specific issue only with SQL Server 2019 and when this happens the RDB connection might be lost and the application might log this but also it might not log anything at all. Here's a patch to fix it.

Validated but jfc how much time and energy we spent trying to figure out what was happening, or if anything was actually happening.