Angry work venting [1/?]
All of the websites I'm responsible for are down right now and there's nothing I can do about it.
Our load balancer started dropping connections. We put in a ticket; no response for 2 hours. Now the LB is failing to load our most popular ecom site, so I call in and try to get someone to do something about the problem.
Shortly after that, all of our sites went down, hard.
It's been 6 hours since we reported this and the problem is now WORSE THAN WHEN WE REPORTED IT.
Angry work venting [2/?]
The only ticket update I've gotten that wasn't "I've escalated this" or "I will get back to you soon" was the following, verbatim:
"We still investigating. We have determinate that the Loadbalancer is working correctly. We need to check if there is an issue with the VMs."
We tested that. Extensively. I swiched hosts to point directly at the webservers and they all work great! Point at the LB, they all time out. I pointed this out earlier in the ticket, too.
Angry work venting [4/4]
I already contacted our infrastructure team and gotten them to repoint DNS to the webservers directly, bypassing the hosting provider's shitty LB. A single server might not handle the traffic by itself, but at least it will handle _some_ of the traffic while our Hosting provider figures out how to get their thumbs out of their assholes.
I can't even begin to describe how angry I am with our current provider. This isn't even the worst thing they've done to us.
Angry work venting [2/?]
@mawr Agggh.
This is starting to look like "escalate to your own management and start charging back the time you're down as a factor of lost sales."
Angry work venting [2/?]
@literorrery If I were the person writing the checks, I'd be doing that.
Hell, if I were two levels above me right now, I'd be forcing us to move to a new hosting provider because this is... I can't even begin to understand how they stay in business.
Angry work venting [4/4]
@mawr Round-robin DNS will solve that in the short-term. It's not as efficient as a LB, but it will get you around having to have one and distribute load around all your webservers instead of putting it all on one that might go down.
Angry work venting [4/4]
@literorrery For the time being we've got enough horsepower on each of these webservers to handle 2x average peak traffic. I'm pretty sure we'll be alright, but round-robin is a solution we might look into if that changes. Thanks for the advice. ♥️
Angry work venting [2/?]
@mawr rrrfg that's terrible :(
Hope you're not taking the heat for this from your company v_v
Angry work venting [2/?]
@vahnj Nah, I over-communicate with my team and my boss is rad. He communicates clearly with his bosses.
They all know our hosting provider are the gotdamned worst. We'd have moved already but there are some major expensive and fragile projects in place and they don't want to make things more difficult for those teams
Angry work venting [2/?]
@mawr this feels like the point where it's more difficult than if you chosen otherwise
At least your teams on your side!
Angry work venting [2/?]
@vahnj That's how I see it, too. I wish the others involved felt the same way.
I did nothing today except try to troubleshoot this problem for our hosting provider because they weren't being responsive via phone or email.
I just. I can't.
Angry work venting [3/?]
I sent an email in explaining that I'd already verified the issue was not with the VMs, but with either the LB or the networking between the LB and the VMs.
Half an hour goes by without a peep from anyone, so I call and get a scared sounding person on the phone with a thick accent who does his very best to get me off the phone without doing anything at all to help me.... but I wasn't havin' that.