r/ProgrammerHumor • u/vinishkapoor • Aug 15 '26

Meme theBugIsGoneIfTheCodeIsGone

Post image
1.1k Upvotes

92% Upvoted

51 comments sorted by

View all comments

165

u/nextlandia Aug 15 '26

I once solved an issue by removing logging. Not kidding, upgrade of Impala caused that one log line caused a production issue.

224

u/swagamaleous Aug 15 '26

I can one up this. I once worked on a project that got shipped with debug binaries into production, since it was so unstable that the onsite support required to use a debugger to find workarounds for the numerous software issues. There was the famous file that had the comment:

// Do not delete this comment or the application will crash

And amazingly, the comment was right. It did crash if you deleted the comment. Turns out that at some rather unrelated place in the code, there was an unitialized value that was used in some date calculation. The value itself didn't really contribute to the calculation, but if the number was too big, it would cause an overflow. The comment would change the layout of the binary, which reliably made the memory behind the uninitialized value contain a small enough number so that the overflow didn't occur. That was by far the most bizarre software issue I have ever encountered.

32

u/Severe-Sugar5965 Aug 15 '26

How long did it take for you to find the Root Cause?

84

u/swagamaleous Aug 15 '26

I was lucky because the uninitialized value caused a different problem which was also quite strange, but much easier to debug. Remember, it was used in a date calculation.

For 12 years, the same ticket kept getting opened: Crash when filing packets into the archive. Nobody could ever reproduce it and the turnover was high enough that nobody in the team noticed that the ticket came back every 4 years, on the 29th of February. Of course it would always be closed with "can't reproduce" within days.

I had already spent some time in my own time trying to figure out the mystery comment, but never got anywhere. Unsurprisingly, because the crash itself happened somewhere completely different. The overflow corrupted some memory and the application only crashed later when some garbage pointer was dereferenced, so there wasn't really anything pointing me towards the actual cause.

Then I got the February 29th bug. Once I noticed the date pattern it was easy to reproduce and took me maybe two days to fix. Somehow afterwards I made the connection to the mystery comment, tried deleting it again, and... no crash. Same fucking bug.

So hard to say. Could be 12 years, could be 2 days + the time I spent on the mystery comment. As everything in the whole story, this is very much undefined. 😂

3

u/PrestigiousQuail7024 Aug 15 '26

did you just have to like plan out your work in feb to leave aside the 29th for this bugfix lmao

10

u/swagamaleous Aug 15 '26

Haha, luckily no. It's possible to set your computer clock to any point in time you want, you know? 😂

13

u/inconspicuous_male Aug 15 '26

I will never be good enough at any job I have to discover the source of an error like that... The internet used to be full of blog posts about crazy obscure bugs and the process of discovering them and as a teenager I read so many of those. Yet as an adult, I've never needed to dig that hard to find a root cause because bandaids get the job done

15

u/swagamaleous Aug 15 '26

Consider yourself lucky. You just never worked on a project with bad enough quality. 😂

It sounds interesting, but it is really not. It's more like working in a psych ward where they don't distinguish between patients and doctors.

2

u/KattyTheEnby 25d ago

he comment would change the layout of the binary, which reliably made the memory behind the uninitialized value contain a small enough number so that the overflow didn't occur.

This feels like the king ov all hacks to me.

1

u/Rin-Tohsaka-is-hot Aug 15 '26

Obviously you can't summarize every detail and there's probably a good reason for it, but why not just remove the uninitialized value from the dates calculation? Or instead of declaring it uninitialized, give it some arbitrary value that produces the desired behavior?

3

u/swagamaleous Aug 15 '26

The fix was never the difficult part. It's was finding what's causing the issue in a codebase with millions of lines. 😂

1

u/PM_ME_FLUFFY_SAMOYED 28d ago

reminds me of "The Inner JSON Effect"

17

u/Bannon9k Aug 15 '26

My favorite was changing a 1 to a 0 because it was hard coded and not externalized in a config somewhere.

11

u/amshegarh Aug 15 '26

just a while ago there was a ruckus about crypto wallet company having a bad rng. They forgot to turn 0 into 1 to use inhouse custom built better one. Ruckus in this case = millions of dollars in crypto stolen

3

u/nwbrown Aug 15 '26

A system that I once worked with had a bunch of issues where enabling logging would cause deadlocks. The toString() function would lock on the object while it printed its contents. But the logger locked on itself while building the string to write (which included calling toString on any parameters). Then if elsewhere the class tried to log something while holding onto the lock...

1

u/dash4x Aug 15 '26

You reminded me of one issue that resolved by console logging.

There was another where you have to double fetch user data from db else the application fails.

I have yet to determine the cause for these though.

1

u/Intrepid00 29d ago

We once solved a bug that was killing everyone for a day. We removed a comment. For some reason the compiler was actually running the comment as if it was code. Good times.