Work Vent 

Oh look a critical service with no redundancy went down in a not 24/7 datacenter, taking one of our flagship products down completely in that region.

This is the 3rd time this week it is happened.

9 months ago, the first time it happened.. I wrote remediation recommendations for the product's team.

THE BIG ONE WAS MOVING THIS SERVICE TO MODERN HARDWARE WITH IPMI AND HOT SPARE!

Guess what they just did...?
FINALLY SPIN THE SERVICE UP ON ONE OUR MODERN COMPUTE BOXES... and decide to stop using the old and busted.

Work Vent 

@wobblewuffess
Why is multihomed redundancy such a hard concept for some folks? Gah.

Follow

re: Work Vent 

@kistaro

I don't know... but it drives me nuts. This particular product requires a service that doesn't play nice with other instances of it running in the same region.. but there is literally no reason not to have a easily spun up hot spare ready to go especially when the hardware is in place already.

Sign in to participate in the conversation
Awoo Space

Awoo.space is a Mastodon instance where members can rely on a team of moderators to help resolve conflict, and limits federation with other instances using a specific access list to minimize abuse.

While mature content is allowed here, we strongly believe in being able to choose to engage with content on your own terms, so please make sure to put mature and potentially sensitive content behind the CW feature with enough description that people know what it's about.

Before signing up, please read our community guidelines. While it's a very broad swath of topics it covers, please do your best! We believe that as long as you're putting forth genuine effort to limit harm you might cause – even if you haven't read the document – you'll be okay!