The Mighty Buzzard writes:
Yeah, so, failure to babysit the db node that was scheduled for a reboot on the 5th resulted in a bit of database FUBAR that left us temporarily losing everything from then to now. Fortunately we had a backup less than six hours old, restored from it, and appear to be copacetic now. Except for the missing five hours and change.
I'd usually make some sort of dumb joke here but it was already four hours past my bedtime when I found out about the problem. My brain is no work good anymore. Fill in whatever dad joke or snark about getting a do-over for a change strikes your fancy.
(Score: 2) by gawdonblue on Monday August 10 2020, @02:40AM (1 child)
Yeah, in the last 3 years we've had to restart the DB at work twice because of "high-availability" clustering getting out of sync. These are the only fatal DB software failures that we have had.
Seems the more dependencies you add the more brittle things become.
(Score: 2) by The Mighty Buzzard on Monday August 10 2020, @04:58AM
Yeah, I'm sure there must be cluster ninjas out there that know every pitfall ahead of time and never have these problems but there aren't any on staff here.
My rights don't end where your fear begins.