Angry work venting [1/?]
All of the websites I'm responsible for are down right now and there's nothing I can do about it.
Our load balancer started dropping connections. We put in a ticket; no response for 2 hours. Now the LB is failing to load our most popular ecom site, so I call in and try to get someone to do something about the problem.
Shortly after that, all of our sites went down, hard.
It's been 6 hours since we reported this and the problem is now WORSE THAN WHEN WE REPORTED IT.
Angry work venting [2/?]
The only ticket update I've gotten that wasn't "I've escalated this" or "I will get back to you soon" was the following, verbatim:
"We still investigating. We have determinate that the Loadbalancer is working correctly. We need to check if there is an issue with the VMs."
We tested that. Extensively. I swiched hosts to point directly at the webservers and they all work great! Point at the LB, they all time out. I pointed this out earlier in the ticket, too.
Angry work venting [3/?]
I sent an email in explaining that I'd already verified the issue was not with the VMs, but with either the LB or the networking between the LB and the VMs.
Half an hour goes by without a peep from anyone, so I call and get a scared sounding person on the phone with a thick accent who does his very best to get me off the phone without doing anything at all to help me.... but I wasn't havin' that.
Angry work venting [4/4]
@mawr Round-robin DNS will solve that in the short-term. It's not as efficient as a LB, but it will get you around having to have one and distribute load around all your webservers instead of putting it all on one that might go down.
Angry work venting [4/4]
@literorrery For the time being we've got enough horsepower on each of these webservers to handle 2x average peak traffic. I'm pretty sure we'll be alright, but round-robin is a solution we might look into if that changes. Thanks for the advice. ♥️
Angry work venting [4/4]
I already contacted our infrastructure team and gotten them to repoint DNS to the webservers directly, bypassing the hosting provider's shitty LB. A single server might not handle the traffic by itself, but at least it will handle _some_ of the traffic while our Hosting provider figures out how to get their thumbs out of their assholes.
I can't even begin to describe how angry I am with our current provider. This isn't even the worst thing they've done to us.