Angry work venting [1/?]
All of the websites I'm responsible for are down right now and there's nothing I can do about it.
Our load balancer started dropping connections. We put in a ticket; no response for 2 hours. Now the LB is failing to load our most popular ecom site, so I call in and try to get someone to do something about the problem.
Shortly after that, all of our sites went down, hard.
It's been 6 hours since we reported this and the problem is now WORSE THAN WHEN WE REPORTED IT.
Angry work venting [2/?]
The only ticket update I've gotten that wasn't "I've escalated this" or "I will get back to you soon" was the following, verbatim:
"We still investigating. We have determinate that the Loadbalancer is working correctly. We need to check if there is an issue with the VMs."
We tested that. Extensively. I swiched hosts to point directly at the webservers and they all work great! Point at the LB, they all time out. I pointed this out earlier in the ticket, too.
Angry work venting [2/?]
@mawr rrrfg that's terrible :(
Hope you're not taking the heat for this from your company v_v
Angry work venting [2/?]
@vahnj Nah, I over-communicate with my team and my boss is rad. He communicates clearly with his bosses.
They all know our hosting provider are the gotdamned worst. We'd have moved already but there are some major expensive and fragile projects in place and they don't want to make things more difficult for those teams
Angry work venting [2/?]
@mawr this feels like the point where it's more difficult than if you chosen otherwise
At least your teams on your side!
Angry work venting [2/?]
@vahnj That's how I see it, too. I wish the others involved felt the same way.
I did nothing today except try to troubleshoot this problem for our hosting provider because they weren't being responsive via phone or email.
I just. I can't.