The Mighty Buzzard writes:
Yeah, so, failure to babysit the db node that was scheduled for a reboot on the 5th resulted in a bit of database FUBAR that left us temporarily losing everything from then to now. Fortunately we had a backup less than six hours old, restored from it, and appear to be copacetic now. Except for the missing five hours and change.
I'd usually make some sort of dumb joke here but it was already four hours past my bedtime when I found out about the problem. My brain is no work good anymore. Fill in whatever dad joke or snark about getting a do-over for a change strikes your fancy.
(Score: 3, Funny) by The Mighty Buzzard on Sunday August 09 2020, @01:27PM (1 child)
Test our backups? He's a witch! Burn him!
Seriously though, Linode had the most recent (and easy to restore from as well) backup and it was a snapshot of the entire DB server VM that's made daily, so testing them really isn't viable unless you folks want to fork over the money for another server for backup testing and to pay someone for work as tedious as the testing of the three VMs we have them make daily backups for.
My rights don't end where your fear begins.
(Score: 2) by Reziac on Monday August 24 2020, @12:47PM
"I'm not a production environment, I'm a basement."
-- TechHole Productions
And there is no Alkibiades to come back and save us from ourselves.