Absolutely the worst. Ten years ago I encountered a bug in a wireless networking stack that would crash the whole network of nodes. It was in the worst possible location that was highly dependent on timed sending where every nanosecond mattered. This meant i could only use leds to indicate what was going on, as a print or anything the like would screw the timers enough to break the network. Can’t actually remember what caused it in the end.
It took me 6 weeks to debug and fix. I did nothing but debugging at the most primitive level. Probably could have done it more efficiently, but I was young and inexperienced.
Worst and most boring 6 weeks of my life caused by concurrent debugging.
About 10 years ago there was a bug in driver to the Intel wireless card, it crashed when it received 802.11n packet. That bug I remeber well, because it was responsible for me stunning Ubuntu. They knew about that bug (it was reported when they were getting ready to release) and still went forward with the release. I'm wondering if that was related.
Coupling can be a bitch. Worked in a shop once that had GPFS (shared filesystem) everywhere. Their rule was "no swap space on any node ever". Weird. Why not? Because apparently slow thrashing on one node will crush GPFS performance on all nodes, without ejecting the failing node. Ugh.
It took me 6 weeks to debug and fix. I did nothing but debugging at the most primitive level. Probably could have done it more efficiently, but I was young and inexperienced.
Worst and most boring 6 weeks of my life caused by concurrent debugging.