Tech/IT people of reddit! What is the biggest mistake you've made at work?

I work for a company that does industry certifications for a specific industry and we have a ton of servers, like, a ridiculous amount. We do updates on them monthly to keep everything up to date and my predecessor left a complete list of the servers on our network for me before he left.

When I started with the company I went through and remote desktoped into every one of them just to make sure I could get in and just scope out what role each server had. There was one however that's name was an exact copy of the previous line except with two letters in the name swapped and when I attempted to connect to it, it would never connect. So I chalked it up to just being a typo and moving on, always ignoring that one on the list when it came time for updates.

Until a few months ago (to note, I have been with the company for about 4 years now) when I was going through and doing updates and noticed that the first version of the name actually had a Hyper-V Manager installed on it, I have no idea how I missed it when I first did my scope out of the network but I somehow did.

When I connected to the manager I saw that stupid server just sitting there with an "up-time" of literally hundreds of days. I logged into it and found the waiting fuck-ton of updates (of course, luckily many were superseded or no longer needed.) So, to try and catch up I ran the updates on the machine and after waiting for a few hours they finally finished and the reboot was prompted.

I initiated the reboot and then all hell broke loose. The moment the server tried to come back up it would instantly blue screen and in my joy of finding and starting the updates I made no Save State for the VM prior to initiating the reboot or the updates. Well at this point it was about 3 AM, I still had an hour drive home, and was still expected to be back in to work by 8. So I shot off an e-mail to my boss explaining that we had a server that didn't come back up but that due to how long I had already been there (about 16 hours) I would tackle it in the morning and I would probably be an hour or two late just to try and get some extra sleep.

When I get back in the next day, I find out that the entire backend for a set of courses was no longer working and none of the company's customers could access their tests for that program. I went into panic mode because that was the day I learned that the server I had just inadvertently nuked, was the server that hosted the program.

It took me 2 more days of basically non-stop work to rebuild the server and get everything back to working because the system used a license that my predecessor never left for me and I had to track down and contact the company that made it, and then figure out how to verify our company's license, without any order information or prior information on the system. But it was still a giant pain in my ass.

TL;DR- Due to a typo I ignored a server for a few years until the accumulated updates broke it.

/r/AskReddit Thread