SoylentNews is people
SoylentNews
from the who-needs-QA-when-we-can-test-it-on-production dept.
At around 9:15 UTC [17 May] Salesforce pushed a database script update that was intended to add modify all permissions to a specific internal profile used by their Pardot service. Due to a scripting error View and Modify All Objects Permission was granted to all user profiles for all organizations that ever had the Pardot product, including public facing community instances. This was of course a security nightmare for customers, especially those in the Financial and Health sectors, and an emergency change was pushed around 10:00 UTC to revoke all permissions to all profiles except for administrators. No announcement was made on their status sites due to the potential for bad actors to take advantage of the security issue that was introduced until the databases could be locked down. Further action was taken around 11:00 UTC to take down PODS completely, likely to further mitigate access risk which effectively expanded the outage to customers that never used Pardot but shared an instance with customers who did.
Salesforce is holding hourly calls, and recently admitted that the script had run both in their production PODS and also in the Passive Disaster Recovery Instances, complicating the ability to recover from the issue. There is currently no ETA for recovery, though it is still their hope that they will not have any data loss. They are beginning to bring back up instances, but only administrators will have access initially and it will require additional time before administrators will be able to modify permissions and rebuild profiles and there will be a longer wait yet before profile settings can be restored from backup.
Coverage at: Geekwire, The Register, and reddit
(Score: 5, Insightful) by Farkus888 on Sunday May 19 2019, @02:52AM (3 children)
2 kinds of people and thinking. I haven't quite determined what to call them. Negotiable and non-negotiable interactions with the real world make the difference. Say an engineer and a welder build a steel bridge and it falls down. One of them messed up, no getting out of it. Either the welds were bad or the design was bad. The MBAs at Salesforce aren't playing that same game, they can always blame the techs. Understaffed or under skilled techs due to MBA incompetence doesn't matter, the tech still actually pushed the button so the MBA never gets punished. MBA fires the tech and hires a replacement who is even more under skilled and cheaper speeding the next disaster, then pats himself on the back for a job well done.
(Score: 0) by Anonymous Coward on Sunday May 19 2019, @07:07AM (1 child)
Aye, but you also forgot that for the tech who gets fired, this is a black mark as far as his future (if any) career in IT is concerned, the MBA?, it not just his self congratulatory back patting that happens, he also gets fucking manglement brownie points for firing the poor sod, thus guaranteeing a future internal promotion and/or a better paid position in another organisation.
I've seen this happen far too many times, and not just in IT.
(Score: 2) by Farkus888 on Sunday May 19 2019, @11:02AM
Didn't forget, just got bored of writing and felt I had the core of my idea covered. That is the real problem though. The MBA and his fellow MBA buddies aren't trying to mislead you, they truly believe the version of the story that blames the tech. People who personally know psychics say they start knowing they are just cold reading, but slowly start to believe they have real powers. If they xray the collapsed bridge and the welds were cold, there is no other story for the welder to tell. People who spend all their time in domains with built in weasel opportunities are different than people who primarily spend their time in no wiggle domains.
(Score: 2) by Bot on Sunday May 19 2019, @07:17AM
In fact, the internal structure of your IT with QA and stuff is not going to save you from disaster, if everybody is overworked and trying to meet completely made up deadlines who are simply a way for management to prove they exist.
Overuse of meatbags (still a less demeaning term than HR) should be treated like going around in a car with a windshield so dirty that you barely see through it. The inevitable eventual becomes your fault and it's aggravated by your complete awareness of the situation and lack of rectifying it.
Maximum de-facto working hours should be enforced more than minimum wage, because it's far more important to society.
Account abandoned.