How Facebook Vanished from the Internet
Around noon Eastern time on 4 October 2021, Facebook, Instagram, and WhatsApp suddenly vanished from the Internet and remained offline for much of the day. While no one outside Facebook yet knows why it happened, Internet infrastructure company Cloudflare has an explanation of what went wrong. It all traces back to an obscure but vital networking protocol called the Border Gateway Protocol (BGP).
BGP enables big networks to tell each other how to route packets. Early on 4 October 2021, BGP messages were sent that withdrew routes to Facebook’s DNS servers, effectively taking them offline. DNS is short for Domain Name System, and it’s the Internet protocol that lets you type tidbits.com to visit our site rather than having to know and enter our IP address of 172.67.213.57. Without global DNS servers knowing how to match facebook.com to an appropriate IP address, there was no way to connect to any Facebook server.
Throughout the day, many other intermittent outages were reported, like Apple Pay, Gmail, and Twitter, along with cellular carriers, as you can see in the Downdetector screenshot below. That likely happened because both apps and users kept trying to connect to Facebook, which strained the DNS servers and led to collateral issues all over the Internet. Facebook resolved the routing and lookup problems for its servers after about 5 hours, and Instagram came back an hour or so later.
Cloudflare’s blog post neatly summarizes the lesson from the day:
Today’s events are a gentle reminder that the Internet is a very complex and interdependent system of millions of systems and protocols working together. That trust, standardization, and cooperation between entities are at the center of making it work for almost five billion active users worldwide.
Too bad it wasn’t permanently!
This explains why for a time when trying to access replies sent to me in Instagram I just kept getting “Can’t refresh feed…”
Some really good memes are here:
https://www.fastcompany.com/90683082/how-twitterers-celebrated-the-great-facebook-whatsapp-and-instagram-shutdown-of-2021
At least there haven’t been any reports of hysterical people flooding 911 call centers, or user data being compromised. I’d like to believe that some Facebook users went to actual news sites instead, but I haven’t read a word about traffic shifting in this direction.
And it wasn’t like the response of many to Orson Wells’ 1938 “War Of The Worlds” radio broadcast:
I had reason to call T-Mobile for support yesterday, and I was greeted not with their usual “We’re having high call volumes and will call you back” message (which works really well) but with “We’re experiencing technical difficulties. Please try again later.”
Then today, when I called again, I asked the support rep about it. He didn’t know about the different system message but said that they were absolutely flooded with calls yesterday, many about the problems with Facebook. That jived with the Downdetector screenshot we published in the ExtraBIT showing the cellular carriers as receiving a lot of down reports as well. Lots of people must see the carrier support departments as handling anything that happens on their phones.
Last night (Monday, 4 Oct 21) on the news reports about the outage, it was said that the timing was interesting as it followed Sunday’s “60 Minutes” piece on the Facebook whistleblower.
Much more likely that a carrier would be having problems rather than Facebook.com
Yes, but as you can see in the graphic I referenced, Spectrum, T-Mobile, Verizon, and AT&T all had significant down reports spiking at the same time as Facebook. It’s clearly connected.
I was speaking as the typical consumer would have seen it, addressing why the support departments were overwhelmed. Carriers outages, usually regional, are much more common from the consumer viewpoint. It’s doubtful Joe User could determine there was a different cause until the news hit the national media.
All reports indicate that this was caused by someone injecting bad data into the global BGP routing network.
For those unfamiliar, the Internet is a “network of networks”. Various sites (large and small companies) run their own networks that are, more or less, self-contained. The links that connect these networks are typically leased from the telcos (although some big companies own their own as well).
The BGP protocol is used to distribute information globally about the connections between these networks. It is critical to the Internet, since the servers you access are almost always located on somebody else’s network and BGP is how the Internet knows how to move your packets from your location to the destination.
The problem here is that BGP has only minimal security. Someone injecting bad data into a BGP router (whether by accident or maliciously) can create all kinds of problems, usually resulting in network outages for various segments of the Internet until the correct data can be re-distributed to replace the bad data.
In this case, bad data was injected, making Facebook’s DNS server unreachable, effectively knocking them off line. Whether this was an accident or a deliberate attack, and the source of the bad data has yet to be determined.
According to Facebook, it appears that the BGP misconfiguration originated from Facebook’s own network.
I don’t believe the conspiracy theory that Facebook did it deliberately for some reason. If they wanted to silence a story, they can just censor it, the way they’ve done for countless other stories, without taking the rest of their system down.
Outages like this do happen from time to time, so it is definitely possible that this was an accident. Someone accidentally clicks “delete” instead of “update” (yes, an oversimplification) on a network management console can do this, and even if the problem is discovered immediately it can take a few hours for the corrected data to propagate to the rest of the Internet.
On the other hand, any time a major site goes down, you should never rule out sabotage. I’m sure FB’s IT people are investigating. Hopefully they’ll let the rest of us know what they discover, but I’m not counting on that.
The supposed panic response to War of the Worlds is an urban legend, whipped up at the time by newspapers jealous of the popularity of radio.
From what I always herd from my parents, grandparents and those of other friends, family and work colleagues was that although there weren’t stampedes of people in NYC and suburbs, as well as relatives in Chicago, there were some people some screaming in the streets. It was not hordes of people, just a few. My grandparents said that they saw a few people up on the roof of a neighboring apartment building to check for space aliens.
The disturbing thing is that not just Facebook went down - the problem also took down the associated WhatsApp and Instagram systems. That is simply poor structure. Those systems should be quite separate and have internal redundancy. The only reason to skimp on this is to save money. My suspicion is that the actual problem was due to some configuration that tried to more closely link these services so they could share hardware resources - what could possibly go wrong! It’s hard to come up with a good analogy - since I have a medical background perhaps: it would be as if 3 operating rooms tried to share one big set of instruments. We always have known that this group is ethically corrupt, but they are also technically bankrupt.
Am I the only one here who had never heard of BGP before yesterday?
Nope. I never did.
Selling cross platform and cross device advertising packages is what keeps Facebook’s profits growing. Here are a few examples:
More details have emerged:
Not sabotage. Just a colossal screw-up combined with bugs in the audit software designed to detect such mistakes.
The more I learn about Facebook, the less I like them. That’s all I’m going to say, for now.