so uh. what on earth happened to the internet while we were away most of today
-F
@Felthry Yeah, *apparently* some kind of routine upgrade work screwed up their DNS registrations on a very deep level, and it screwed up all the sites that depend on Facebook.
I saw some Birdsite giggling that, apparently, part of what made it hard to fix things is tech teams use Facebook-connected services to coordinate activity and now that was gone.
Also *allegedly* some of their techs were locked out of buildings and sections by Internet-connected door locks that routed through Facebook.
@Austin_Dern what the fuck why are their door locks connected to facebook
why are they connected to the internet anyway and not some local-only network
-F
@Felthry I don't know, and I don't know whether I believe it. Any situation like this there's a lot of people putting (often funny) jokes out but it's not always obvious what's true and false.
Does seem like a contributing factor to the problem lasting was many people working remotely, so they were short on people who could actually touch a for-real router or server or whatever.
@Felthry @Austin_Dern they're not connected to Facebook the software but they probably route their web requests through a domain owned by Facebook (if not a subdomain of facebook.com)
@Felthry @Austin_Dern I read that they had to use an angle grinder to reach the servers
@Austin_Dern @Felthry it's the kind of thing that makes me wanna become ops
@Felthry seems like someone at Facebook pushed a bad BGP config that ended up making all the world's computers forget what servers "facebook.com" should direct to, including their own internal systems. This resulted in everything that relied on Facebook - from WhatsApp and Oculus to (for some bizzare reason) the smart door locks in FB's buildings - ceasing to function, and then a bit later everything that would ping FB for stuff started essentially DDoSing DNS services as they tried desperately to connect.
@starseeker What's BGP?
-F
@Felthry It stands for "Border Gateway Protocol"; I'm not *super* certain on what exactly it is, but it seems to be part of the system that tells the world what IP address a URL refers to. It's also a system that can take a bad config uploaded in one place and propagate it to the entire world, such as what happened in 2008 when Pakistan tried to block domestic traffic to Youtube:
https://abcnews.go.com/Technology/story?id=4344105
i'm seeing people mocking facebook for going down and somehow that took down half the internet with it??
-F