SoylentNews is people
SoylentNews
from the who-needs-QA-when-we-can-test-it-on-production dept.
At around 9:15 UTC [17 May] Salesforce pushed a database script update that was intended to add modify all permissions to a specific internal profile used by their Pardot service. Due to a scripting error View and Modify All Objects Permission was granted to all user profiles for all organizations that ever had the Pardot product, including public facing community instances. This was of course a security nightmare for customers, especially those in the Financial and Health sectors, and an emergency change was pushed around 10:00 UTC to revoke all permissions to all profiles except for administrators. No announcement was made on their status sites due to the potential for bad actors to take advantage of the security issue that was introduced until the databases could be locked down. Further action was taken around 11:00 UTC to take down PODS completely, likely to further mitigate access risk which effectively expanded the outage to customers that never used Pardot but shared an instance with customers who did.
Salesforce is holding hourly calls, and recently admitted that the script had run both in their production PODS and also in the Passive Disaster Recovery Instances, complicating the ability to recover from the issue. There is currently no ETA for recovery, though it is still their hope that they will not have any data loss. They are beginning to bring back up instances, but only administrators will have access initially and it will require additional time before administrators will be able to modify permissions and rebuild profiles and there will be a longer wait yet before profile settings can be restored from backup.
Coverage at: Geekwire, The Register, and reddit
(Score: 1, Informative) by Anonymous Coward on Sunday May 19 2019, @02:52AM (4 children)
They're like the Apple of Google. A whole shitload of expensive overhyped services that have no end consumer value but make corporate look good.
(Score: 0) by Anonymous Coward on Sunday May 19 2019, @06:39PM
Metrics! SEO! Analytics! Ad views! User retention!
Looks like it is gonna be a while longer before people realize that centralized services are often more of a liability than the convenience is worth. Some of the prices I've heard for these managed service solutions are so insane that they could hire 1+ full time employees and better hardwsre.
(Score: 1) by Ethanol-fueled on Sunday May 19 2019, @07:59PM (2 children)
Absolutely this. When a former employer of mine "upgraded" their CRM system from Access to Salesforce, regular verbal arguments were breaking out over lots of lost work because Salesforce didn't implement record-locking. You would go to a record, input shit-tons of data as part of a very detailed inspection process, then when you hit the "save" button it would be all like, "LOL sorry, somebody else is viewing this record, and there is no way to save all of your information you just painstakingly entered." This was back around 2015ish. Losing work in that fashion really demotivates and knocks the wind out of your sails, whether it is writing a novel or recording music, or getting job stuff done. We had to develop a whole separate informal system where people would shout across the room that they were editing a record, or ask if others were editing.
Fucking inexcusable. Don't believe the hype. It's the kind of idiocy that only San Francisco could produce.
(Score: 1) by fustakrakich on Monday May 20 2019, @12:22AM
You would go to a record, input shit-tons of data as part of a very detailed inspection process, then when you hit the "save" button it would be all like, "LOL sorry, somebody else is viewing this record, and there is no way to save all of your information you just painstakingly entered."
You work at Wikipedia?
La politica e i criminali sono la stessa cosa..
(Score: 2) by datapharmer on Monday May 20 2019, @02:21PM
Remember that the core of Salesforce goes back 20 years now, so it has a lot of legacy cruft. Although this permissions blunder is in a field of its own, they do know where they've got gaps and need to play catch up. They actually have "lightning live records" on the roadmap to solve this exact problem. Cache invalidation is harder than it sounds.