Facebook Down? Angry Emoji

Confusion swept the nation last week.

People everywhere were forced to resort to primitive, caveman-esque pastimes such as finding patterns in the clouds and watching Hulu.

For six seemingly endless hours, society teetered on the brink of collapse as we collectively held our breath for good news.

Perhaps this hope brought us together? Healing old wounds as we discovered we all, when it comes down to it, worship the same deity. This is the epic tale of when Facebook was down for maybe half a day or so.

While it's a lot of fun to make light of situations like this one, Facebook's reach in 2021 goes quite a lot further than simply hosting photos from your vacation. This most recent outage the company suffered not only took out their main site, but also prevented WhatsApp, Messenger, Instagram and their Oculus services from working too.

I know I use at least two of those platforms to communicate with friends and family, and while news of any importance better befits a phone call, it was inconvenient to have messages remain unsent.

As it happens, the downtime also affected Facebook's own employees use of messaging tools internally.

So what happened?

Well, while it's not scandalous, the cause is still quite interesting. To help you be as captivated as I am, first let's take a moment to think about how the internet works. The web we all enjoy is simply a network of networks. Facebook, Netflix, Amazon - anything we visit online - exists on its own smaller network which we connect to when we want to access its content. The journey this data takes back and forth is called a route, and routing is a very important part of what makes the internet work as it does.

With me so far?

These routes need to be smart, as it takes time for data to travel from place to place.

What's to stop the movie you're streaming from traveling around the world three times before ending up on your TV?

Enter: Border Gateway Protocol, or BGP.

Think of BGP as the roadmap of the internet; it allows devices to exchange information to ascertain the quickest route for your data to get from its source to your device.

We don't hear about it very regularly, because it's quite robust and more often than not it simply works silently in the background without fuss. In the case of Facebook's most recent disappearance, not so much.

Around 9:30 a.m. on Oct. 4, it was the lack of this needed BGP communication which resulted in a loss of service.

Without knowing how to properly direct web traffic to Facebook and its related services, the result was errors and hourglasses.

Due to the nature of this failure, it meant that engineers couldn't use the tools they'd normally employ to troubleshoot it.

No route, no communication, no fixing it remotely.

Engineers had to physically travel to the vast datacenters required to run such widely used services to perform the fix.

Datacenters, which I'm sure you can imagine, have physical and technological security which is no small task to circumvent, even in legitimate situations like this one.

The cause of this headache? A simple digital oil change.

Routine maintenance gone wrong.

During what should have been a simple connectivity check, an engineer used a command which inadvertently broke Facebook's connection to the internet.

A bug in the system which audits these commands to prevent mishaps like this allowed it to be run and the resulting whoops cost Facebook themselves around $80 million in ad revenue.

This is in addition to what it cost the businesses actually paying for the ads.

If ever we needed a "dislike" button...

Richard Noble is the founder of Want For Tech, an IT company based in Glasgow.