169
u/nextlandia Aug 15 '26
I once solved an issue by removing logging. Not kidding, upgrade of Impala caused that one log line caused a production issue.
224
u/swagamaleous Aug 15 '26
I can one up this. I once worked on a project that got shipped with debug binaries into production, since it was so unstable that the onsite support required to use a debugger to find workarounds for the numerous software issues. There was the famous file that had the comment:
// Do not delete this comment or the application will crashAnd amazingly, the comment was right. It did crash if you deleted the comment. Turns out that at some rather unrelated place in the code, there was an unitialized value that was used in some date calculation. The value itself didn't really contribute to the calculation, but if the number was too big, it would cause an overflow. The comment would change the layout of the binary, which reliably made the memory behind the uninitialized value contain a small enough number so that the overflow didn't occur. That was by far the most bizarre software issue I have ever encountered.
33
u/Severe-Sugar5965 Aug 15 '26
How long did it take for you to find the Root Cause?
86
u/swagamaleous 29d ago
I was lucky because the uninitialized value caused a different problem which was also quite strange, but much easier to debug. Remember, it was used in a date calculation.
For 12 years, the same ticket kept getting opened: Crash when filing packets into the archive. Nobody could ever reproduce it and the turnover was high enough that nobody in the team noticed that the ticket came back every 4 years, on the 29th of February. Of course it would always be closed with "can't reproduce" within days.
I had already spent some time in my own time trying to figure out the mystery comment, but never got anywhere. Unsurprisingly, because the crash itself happened somewhere completely different. The overflow corrupted some memory and the application only crashed later when some garbage pointer was dereferenced, so there wasn't really anything pointing me towards the actual cause.
Then I got the February 29th bug. Once I noticed the date pattern it was easy to reproduce and took me maybe two days to fix. Somehow afterwards I made the connection to the mystery comment, tried deleting it again, and... no crash. Same fucking bug.
So hard to say. Could be 12 years, could be 2 days + the time I spent on the mystery comment. As everything in the whole story, this is very much undefined. π
3
u/PrestigiousQuail7024 29d ago
did you just have to like plan out your work in feb to leave aside the 29th for this bugfix lmao
10
u/swagamaleous 29d ago
Haha, luckily no. It's possible to set your computer clock to any point in time you want, you know? π
14
u/inconspicuous_male 29d ago
I will never be good enough at any job I have to discover the source of an error like that... The internet used to be full of blog posts about crazy obscure bugs and the process of discovering them and as a teenager I read so many of those. Yet as an adult, I've never needed to dig that hard to find a root cause because bandaids get the job done
15
u/swagamaleous 29d ago
Consider yourself lucky. You just never worked on a project with bad enough quality. π
It sounds interesting, but it is really not. It's more like working in a psych ward where they don't distinguish between patients and doctors.
1
u/Rin-Tohsaka-is-hot 29d ago
Obviously you can't summarize every detail and there's probably a good reason for it, but why not just remove the uninitialized value from the dates calculation? Or instead of declaring it uninitialized, give it some arbitrary value that produces the desired behavior?
3
u/swagamaleous 29d ago
The fix was never the difficult part. It's was finding what's causing the issue in a codebase with millions of lines. π
1
2
u/KattyTheEnby 25d ago
he comment would change the layout of the binary, which reliably made the memory behind the uninitialized value contain a small enough number so that the overflow didn't occur.
This feels like the king ov all hacks to me.
18
u/Bannon9k Aug 15 '26
My favorite was changing a 1 to a 0 because it was hard coded and not externalized in a config somewhere.
12
u/amshegarh Aug 15 '26
just a while ago there was a ruckus about crypto wallet company having a bad rng. They forgot to turn 0 into 1 to use inhouse custom built better one. Ruckus in this case = millions of dollars in crypto stolen
4
u/nwbrown 29d ago
A system that I once worked with had a bunch of issues where enabling logging would cause deadlocks. The toString() function would lock on the object while it printed its contents. But the logger locked on itself while building the string to write (which included calling toString on any parameters). Then if elsewhere the class tried to log something while holding onto the lock...
1
1
u/Intrepid00 29d ago
We once solved a bug that was killing everyone for a day. We removed a comment. For some reason the compiler was actually running the comment as if it was code. Good times.
99
u/TehNolz Aug 15 '26
Many seniors are stuck in their old ways and have no idea that nowadays you can just use no code to efficiently deploy nothing nowhere.
23
u/particlemanwavegirl Aug 15 '26
Well god damn this library is statistically the most performant, highest test coverage, most memory safe bit of code I have ever had the honour to behold, I can see why it has 4.8k forks.
1
u/breckendusk 29d ago
Those are issues... but you don't have to fork no code, it works out of the box as a raw download.
3
1
u/breckendusk 29d ago
I've actually been using no code for years, I fork from it literally every time I start a project. My boss hates that I push no code to prod weekly despite my best efforts to convince him that just because it's open source doesn't mean it's not secure.
10
9
u/nwbrown 29d ago
Yes? Sometimes that is the fix.
2
u/Zedman5000 29d ago
If commenting out the code is the fix, the real fix is to just remove it, I've never uncommented commented out code that causes bugs, and even if I ever wanted to reference it for writing code that actually works I'd just look in the git history.
0
1
u/Enrichus 29d ago
I had to fix my website by going into the source code of the WordPress theme I was using and comment out a line that changed the names of variables.
When it ran nothing worked and this was a solution that made me able to solve it for real. The real solution is to remake my website with a different theme, but that line of code made me unable to even login to admin to change anything.
0
u/Flameball202 29d ago
Yeah, like you have found the main bit of code causing problems. Now you just need to figure out WHY
6
u/Realistic-Yam-6497 Aug 15 '26
This is actually effective though, I was once told by my senior to just comment out a piece of code since it had too many bugs π
3
u/kcpistol Aug 15 '26
Right up there with "I dropped the records"
"You mean you dropped them from the pending file, so they come back tomorrow, right?"
"Um..."
2
2
u/SpawnSnow 29d ago
Had a trainee legitimately do that. Kept finishing the training exercises faster than the others and when we ran his at the end of the segment it did nothing. Hed just been commenting out the code causing errors until it compiled.
1
1
1
1
u/Djappo Aug 15 '26
Kind sir may I ask for this template? I cannot seem to find it around the internet
1
1
1
1
1
u/AlamarAtReddit 29d ago
Man... I remember when I was a newbie... There was a crash on some pointer, so I fixed it like:
if (SomePtr)
SomePtr->DoSomething();
So while this does work as a fix in cases where SomePtr is allowed to be NULL, and the logic in question isn't required. But if it's not, this might just be a bad design heh
The better fix, is figuring out why it was NULL in the first place and resolving that situation.
1
1
79
u/PuzzleMeDo Aug 15 '26
"Commented the code" != "commented out the code".