On Monday, October 4th, Facebook experienced a colossal outage that affected all 3 billion users across the planet. Lots of people depend on Facebook for their livelihood and some of us are just plain addicted and we deserve to know what happened. Okay, we may not deserve anything but I was super curious over how a supremely technical company could be taken down so astonishingly hard. The story is interesting from a technical standpoint if you’re a nerd. However, it also gives some insight into how the internet works, so that may be interesting to everyone else.
Everything connected to the internet has an Internet Protocol (IP) address. An IP address is usually a string of numbers such as 220.127.116.11. That’s the IP address of my company’s website. But numbers are hard to remember at cocktail parties so we invented something called Domain Name Service (“DNS” - think of it as a phonebook for the internet) that allows us to associate a name to these long numbers so we can remember them. In this case, it is ExcelAL.com. You can access my website by typing those numbers or those letters thanks to DNS. When you type those letters, the DNS “routes” your computer to the proper website address.
The Facebook outage involves more than just DNS, but explaining BGP records will put all of you to sleep. None of you even care, so “DNS” will suffice.
Unlike everyone else, Facebook owns their own DNS phonebook servers located deep in the bowels of Facebook headquarters. These servers aren’t available to the rest of the world. Turns out some goober in the big, huge server room at Facebook headquarters implemented an ordinary update to the DNS records and, well, Mark Zuckerberg (Facebook CEO) was suddenly 6 billion dollars poorer.
Ordinarily, this would be no big deal. Any good technician would simply “undo” or restore from a backup. But this was different. During the pandemic, a lot of technicians work from remote locations. Once the update broke the DNS, all the technicians were blocked from remote access. No big deal, they can just go on-site and work, right? Nope. When Facebook servers went down, so did all the key cards that grant physical access to the server rooms. No big deal, the technicians will simply send a message to a security guard to come manually unlock the door. Welp, the messaging service runs through the same equipment so a big NOPE there, too. Isn’t that funny? It took them 6 long hours to sort it all out. I’ve had days like that and they are not fun.
All this comes on the heels of an unflattering whistleblower report aired by 60 Minutes the night before. I know it’s fun to think there was a relation between the two, but there is no evidence of any connection, whatsoever. Conspiracy theorists are abuzz about it because that’s what conspiracy theorists do.
Jim Fisher owns Excel Computer Services in Florence. Reach him at www.ExcelAL.com