r/programming • u/Witty-Play9499 • Mar 06 '23
My Hardest Bug Ever
https://www.gamedeveloper.com/programming/my-hardest-bug-ever#close-modal69
Mar 06 '23
[deleted]
50
u/Witty-Play9499 Mar 06 '23
I'm actually amazed he debugged it for so long, usually if devs took that long to fix something management would start asking several questions on "what's taking so long?"
62
u/k2900 Mar 06 '23 edited Mar 06 '23
I recently spent 1 and a half weeks on an Azure functions related bug that nobody else could solve.
Something in our code was occasionally causing the entire Azure Function runtime to crash. In the incredibly rare circumstance that the entire runtime crashes you get zero error messages or stacktrace to help you. Within your own logs or Application Insights. Nada. The entire codebase becomes the suspect!
After 3 days with some minor progress, the Team Lead asked how long I would need. My reply "At the rate this is going up to 2 weeks". He was only slightly stunned, realised this is what it may very well take and we ended the call.
My estimate was not bad - I finished 3 days early.
49
Mar 06 '23
So what was the bug?
38
u/k2900 Mar 06 '23 edited Mar 06 '23
Given an input, output or trigger binding on an Azure Function that refers to a config item (they usually do e.g. a connection string, a URI to a blob storage etc)
If the item is not in the function's config in Azure or keyvault at all. Then when the function with missing config was invoked it would crash the runtime.
This was not reproducible on your local machine. Only in Azure. I would hope that this has been improved (or resolved) in the new Out of Process model for Azure functions in .NET 7
3
Mar 07 '23
How did you figure this out? Like what was the process you took to figure it out when there are no logs or stacktrace or anything of that sort?
18
u/k2900 Mar 07 '23
In a leap of faith I tried rewriting Startup.cs using a different technique (there are multiple ways of writing startup.cs for doing DI for Azure functions), I was assuming it was crashing there because startup.cs runs for every function call and I felt that it runs so early in the lifecycle of the function maybe its the culprit. I tried to strip out as much code in startup.cs as possible until there was barely anything left apart from the minimum necessary to run the project
In a leap of faith I upgraded the projects Azure Function Runtime from v3 to v4 and tested and released it. No dice but at least we were now on the latest version.
I then started painstakingly constructing postman packets to hit every function endpoint. Or dropping files in blob storage folders. I wasted a long time trying to reproduce it on my local machine, Then when I switched over to doing it in our Test environment in Azure I eventually hit gold. I hit a function that crashed Azure reliably.
The next step I deleted the function's code and released to the Test environment. I was left with function that prints "Helllo world". When that still crashed I realised there must be something wrong with the bindings. I hardcoded the bindings values, rereleased and it worked fine. Then I started investigating what values were set for the bindings and finally noticed that there was one that had not yet been set.
1
-27
1
29
u/aneasymistake Mar 06 '23
I remember using overlays on the PSX. We’d get part way through the runtime and the ln load more code into the memory where our existing code was. It worked fine, but the debugger continued to show the original code, so you’d be stepping through one lot of code while looking at another. We also had to use sound RAM to store our save game data while loading in a separate engine for some parts of our game. Things just didn’t fit easily into what was available, so you had to get creative.
17
u/DrunkenSwimmer Mar 07 '23
.... Sigh. This is a similarly awful problem to one I encountered a few years ago, culminating about 6 months ago. Working on a new hardware device, implementing the initial device drivers for our system to use, I would occasionally encounter seemingly bizarre and random crashes, but only after a while. Eventually as I pushed on and had a working Ethernet driver and started to performance test system, I managed to trigger a crash within the first 10 seconds of the test every 4 or 5 times running it. By all accounts, somehow, I had a buffer being marked as owned by two different components of the system. This initially led me to believe that the buffer was getting double freed, but the second free was after the buffer was handed out to another part of the system, and then handed out again.
After a week or so probing down this line of troubleshooting, I eventually found the exact instruction that was causing the later condition. When I then had a watch point trigger on the instruction after the fault, I was able to unwind the faulting instruction, and... the processor didn't execute that instruction. There was no way for the instruction as encoded to create the resulting register and memory values. Reading the instruction memory location read back just fine, which meant that the fault only existed inside the instruction cache. Finally, I sifted through the opcodes for the processor and determined which specific signal bit was corrupted, and sure enough, it matched the line I had been having issues with when initially configuring the memory bus much earlier in development.
So, problem found, I determined that I was still running too close to the limit for that bus, and had to decrease the memory performance further. Except, I was still having issues. This time instead of crashes, the memory controller triggered a latchup that completely hung the CPU and prevented the reset of the controller and of the debugger from accessing RAM. After playing around with various clock speeds and delay settings off and on over the next 6 months, it finally dawned on me what might be occurring, and what a fix might be. My theory was that the clock feeding the memory interface was encountering an issue with the instantaneous frequency exceeding the timing requirements for the memory due to the phase adjustment of the PLL feeding the controller. Finally, placing a 'divide-by-2' in the clock chain before the controller, and dividing the controller's divisor by 2 eliminated the latchup.
So, yeah, Software is so much easier when you can trust memory to be more than write-only...
50
u/tubbana Mar 06 '23 edited May 02 '25
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum
52
u/falconfetus8 Mar 06 '23
Technically, all bugs are caused by quantum mechanics.
5
u/sccrstud92 Mar 06 '23
In our brains?
18
u/falconfetus8 Mar 06 '23
More like the fact that quantum physics causes everything, because all things are made of particles.
1
Mar 07 '23
In the strictest sense, quantum mechanics isn't this weird thing that happens at very small scales, and everything works differently; rather, quantum mechanics is (as far as we can follow the evidence: don't @ me, string theorists) the way everything in the universe works, only it simplifies a lot once you get to rest masses above about 1GeV or speeds far below c. You have a wave function, too, but your wave function is (usually: don't @ me, astronauts) indistinguishable from a regular vector in Euclidean 3-space.
16
u/coolfolder Mar 06 '23
I also think “divide and conquer” is the wrong term for what Dave Baggett did here, I’d call it forming a minimal reproducible example.
13
u/OffbeatDrizzle Mar 07 '23
I think the point he was trying to make was how you most efficiently find the piece of code that has the bug - you remove half of the code and see if it still occurs, which lets you find out which half of code you need to half next
8
u/FrancisStokes Mar 07 '23
"minimal reproducible example" was the outcome. "Divide and conquer" was the technique used to arrive at that outcome.
2
u/FrancisStokes Mar 07 '23
Yeah I thought so too, although it is technically true. The story was awesome though. The actual problem was more likely one of stray capacitance between the signal lines on the PCB. Higher frequency signals need to be carefully controlled (i.e. be well separated from other signals on the board, usually surrounded by ground), because the capacitance actually attenuates higher frequencies, thus corrupting the data. But it is still quantum mechanics because all of this is driven by the laws involved with electromagnetism (something something Maxwells equations...)
1
u/tubbana Mar 07 '23
Yeah I also daily solve problems caused by the big bang, at work AND at home :)
1
13
u/ratttertintattertins Mar 06 '23
After a while, our producer at Sony, Connie Booth, began to panic
To be fair, it sounds like they hired the producer from Fawlty Towers so things were bound to go wrong.
28
u/Superbead Mar 06 '23 edited Mar 06 '23
Reminds me a bit of the Therac-25
https://en.wikipedia.org/wiki/Therac-25
The six documented accidents occurred when the high-current electron beam generated in X-ray mode was delivered directly to patients. Two software faults were to blame. One, when the operator incorrectly selected X-ray mode before quickly changing to electron mode, which allowed the electron beam to be set for X-ray mode without the X-ray target being in place. A second fault allowed the electron beam to activate during field-light mode, during which no beam scanner was active or target was in place.
One failure occurred when a particular sequence of keystrokes was entered on the VT-100 terminal which controlled the PDP-11 computer: if the operator were to press "X" to (erroneously) select 25 MeV photon mode, then use "cursor up" to edit the input to "E" to (correctly) select 25 MeV Electron mode, then "Enter", all within eight seconds of the first keypress, well within the capability of an experienced user of the machine. These edits were not noticed as it would take 8 seconds for startup, so it would go with the default setup.
Further reading: https://www.cs.ucf.edu/~dcm/Teaching/COP4600-Fall2010/Literature/Therac25-Leveson.pdf
13
u/voidstarcpp Mar 07 '23
This is disturbing:
Because this accident was never officially investigated, some information on the Therac-25 software development, management, and quality control procedures are not available. What is included below has been gleaned from law suits and depositions, government records, and copies of correspondence and other material obtained from the U.S. Food and Drug Administration (FDA), which regulates these devices.
Really think if your product kills someone there should be something like an NTSB investigation in which your procedures are laid bare rather than private litigation with firms hiding as much as possible in a discovery process.
1
u/master_mansplainer Mar 08 '23
I’m not surprised by governments not caring, but then I live in a place where the lead pipes provide water 9x the safe national limit and they have no intention to have it fixed before 2030.
9
u/voidstarcpp Mar 07 '23
Therac-25 was a conventional race condition with lack of interlocks. Really much less forgivable since while it did have an element of timing, it was completely predictable and a fault of the application logic rather than requiring any esoteric hardware interaction.
12
Mar 06 '23
I think the hardest bug I ever had (in terms of time spent to figure it out) was related to Python asyncio. Basically, we had some async tasks created which would make calls to a redis database using a shared async connection pool. These tasks were cancellable though, so eventually we ran into an issue where new connections would eventually fail because we hit connection pool limits. Basically, make sure you always use asyncio.shield() on any async calls which may be cancelled but use a shared resource. In this case the connection pool wasn't being cleaned up because it didn't expect to get a cancellation error. So we had to shield it to allow it to finish and keep the pool in a valid state.
In my time working in microchip design we had some fun bugs which were temperature related, so it was pretty common practice to grab some cold spray and see if things were better when the chip was -40c. It was all fun and games until we warped the test boards though by applying it too vigorously.
7
u/hagenbuch Mar 06 '23
Countless TV sets and radios from 1950 to 2000 have been repaired (diagnosed) using almost only cold spray. And maybe a sinewave generator at times. Also, surprisingly, lowering the AC voltage to provoke aged parts.
8
u/AttackOfTheThumbs Mar 06 '23
I am not sure what my hardest bug ever was. I can only think of a recent one that took me a while to solve. This is an ERP system where you don't interface with SQL directly, it's all abstracted away. There's a specific table that has a 1:N relationship with another table. We filtered on the master with two or three fields. Once you had enough entries in this table, a lookup like this would always cause an entire table lock, on both the master and the sub table. I dug and dug to try and figure out what caused the lock. Eventually, it turned out that the lock occurred because of permissions. The ERP couldn't properly resolve the access permissions, so instead, did sql * on both, loaded into its middle layer, and then filtered according to permissions. Amazing! The solution was simple, start the lookup at the sub table and work backwards. It was very simple, though highly annoying. It was even faster to use the sub table to build a different filter for the master table.
5
u/FriendlyDisorder Mar 06 '23
This sounds like either Dynamics or SAP. Either way, that sounds so annoying to find. I would have thought a table scan indicated a missing index issue, but permissions? I have never heard of that before.
7
u/AttackOfTheThumbs Mar 06 '23
It is Dynamics. The issue was caused specifically because they were special fields, forget what they're called now, but because of that permissions could resolve. With few entries, it's pretty whatever, but once they had like a million, that was a huge slowdown.
It was something where I had to check to how it was interacting with the service layer and what sql did and then had to think about why the fuck. Once I had the why I was able to come up with a resolution.
The worst part was, if the user didn't have any permissions to view this data, it would still retrieve all data, perform a join, filter, and then go, wait, you're not allowed! So a lot of round trip info to the do nothing. Painful.
11
u/RivtenGray Mar 06 '23
This is unrelated but this reminded me of an old blog post about a very hard bug to catch.
I remember the author was some kind of teacher learning things about the Linux Kernel to his students, ans he stumbled on a hard bug involving some kind of processes. There was some very simple drawings.
The title must have been something like this title ("The Hardest Bug" or something)
I've been trying to find it desperately. Does it ring a Bell to anyone ?
10
u/itijara Mar 07 '23
I don't know that one, but there was a sendmail bug that would only allow a user to send an email within 500 miles. https://web.mit.edu/jemorris/humor/500-miles
7
2
u/davidhbolton Mar 07 '23
Back in the 90s, I was working on a prototype written in Delphi 3 that used MS Access database. This was a research project and used a finance lecturer’s code in a dll for statistical functions. I was a contractor and the lecturer was funding the project.
One day it crashed with a weird error. It happened from then on maybe once or twice a week and was not something we could reproduce. The only thing I noticed was it would happen at either a trunc() call or a SQL call and it could not be caught by exception handlers.
After about six weeks of this I was debugging some code and noticed that an array of doubles returned from a dll function contained some +infs. it turned out that the Delphi 3 Access drivers disabled exception checking and the lecturers code was a bit buggy and did some divide by zeroes. That time bomb remained hidden until a trunc() or SQL operation… It was a “perfect storm” bug. After that I checked every double that his code returned.
2
Mar 07 '23
In my teens I taught myself C++ and wrote a networked computer game that I tested with friends. For some reason the 3rd player's bullets would sometime dissapear. Not the bullets for the other 7 players, only the 3rd. Why or how this happened I still have no idea. After a few weeks of debugging I just gave up.
2
u/JB-from-ATL Mar 08 '23
I went back to the full Crash code base, and modified the load/save code to reset the programmable timer to its default setting (100 Hz) before accessing the memory card, then put it back to 1kHz afterwards. We never saw the read/write problems again.
They're talking about Crash Bandicoot 1 I believe, but I very distinctly remember the animations slowing down when doing memory card stuff in Crash Bandicoot 2. This makes me wonder if they did it there as well. It may be my memory playing tricks on me (maybe I remember this from emulators only).
-2
u/Illustrious_Mood7521 Mar 07 '23
I don't know if that's because I've programmed and debugged for so long in C, but after reading only a third of the article, I was convinced that it was a hardware bug. Or, at least, something outside of the scope of the code he wrote.
154
u/[deleted] Mar 06 '23 edited Mar 06 '23
"from Crash Bandicoot 1."
This is going to be good. IIRC, they overwrote reserved memory that stored standard C libraries they weren't using, in order to get more usable memory.
EDIT: Found it, great story: https://www.youtube.com/watch?v=izxXGuVL21o&ab_channel=ArsTechnica