from the who-needs-QA-when-we-can-test-it-on-production dept.
At around 9:15 UTC [17 May] Salesforce pushed a database script update that was intended to add modify all permissions to a specific internal profile used by their Pardot service. Due to a scripting error View and Modify All Objects Permission was granted to all user profiles for all organizations that ever had the Pardot product, including public facing community instances. This was of course a security nightmare for customers, especially those in the Financial and Health sectors, and an emergency change was pushed around 10:00 UTC to revoke all permissions to all profiles except for administrators. No announcement was made on their status sites due to the potential for bad actors to take advantage of the security issue that was introduced until the databases could be locked down. Further action was taken around 11:00 UTC to take down PODS completely, likely to further mitigate access risk which effectively expanded the outage to customers that never used Pardot but shared an instance with customers who did.
Salesforce is holding hourly calls, and recently admitted that the script had run both in their production PODS and also in the Passive Disaster Recovery Instances, complicating the ability to recover from the issue. There is currently no ETA for recovery, though it is still their hope that they will not have any data loss. They are beginning to bring back up instances, but only administrators will have access initially and it will require additional time before administrators will be able to modify permissions and rebuild profiles and there will be a longer wait yet before profile settings can be restored from backup.
Coverage at: Geekwire, The Register, and reddit
(Score: 2, Funny) by Anonymous Coward on Sunday May 19 2019, @12:39AM (2 children)
"Break it fast and break it often".
(Score: 0) by Anonymous Coward on Sunday May 19 2019, @01:06AM
Darn, "This is windows calling" couldn't log in for an hour and fix mein Fenster PC.
(Score: 2) by Hartree on Sunday May 19 2019, @04:58AM
"Break it fast and break it often"
Does that refer to the bones in their feet that they're rapid fire shooting at?
(Score: 5, Informative) by sshelton76 on Sunday May 19 2019, @12:46AM (11 children)
This demonstrates pretty clearly why modern paradigms such as "Agile", "move fast and break things" and even devops is bad.
Programming and QA and sys-admin duties should never, ever, ever be combined into a single department, let alone a single person.
I find it telling that just a little under a year ago Salesforce got rid of QA by merging it with Dev and firing anyone who couldn't adapt.
https://www.indystar.com/story/money/2018/08/06/salesforce-reorganization-shed-workers-indianapolis/914544002/ [indystar.com]
They then fired a bunch of their long time guys and farmed their duties out to the lowest bidder from India, but before forcing them to train their replacements.
https://h1bdata.info/listlca.php?em=SALESFORCE [h1bdata.info]
As a result, a lot of institutional knowledge is just gone and the people left behind are those who didn't have the skills to get employed elsewhere before the house of cards collapsed.
Something like this NEVER would have made it out of Q&A had they just left things in place. But chasing the next quarter's "growth targets" and short sighted managers who can't see past their next bonus are directly the cause.
Note to anyone considering outsourcing... You will save a little money upfront because these guys will over promise and under deliver. They overstate their skills and it isn't until everything implodes like this that you, as a person who has not done the job every day for years, will realize that you were conned. Good luck hiring back the guys and gals you let go.
(Score: 4, Insightful) by sshelton76 on Sunday May 19 2019, @12:50AM (2 children)
Should read, "but not before forcing them to train their replacements.
Which brings up another question... If a person is made to train their H1B replacement, doesn't that sorta say two things.
First off the H1B didn't already have the skills by virtue of needing to be trained, ergo why was he or she brought over in the first place?
Secondly that the H1B wasn't really needed because there was already someone doing the job and the whole point of an H1B is that there is no one in the US labor pool who has the skills to do the job?
(Score: -1, Offtopic) by Anonymous Coward on Sunday May 19 2019, @02:35AM
Woah, wait? Didn't Trump! Trump! Trump! fix that already? That's why he's resorting to tariffs, concentration camps, and preparing to make use [wsws.org] of the military's Operation Jade Helm training, because not even shutting down the H1B program has been able to stem the barbarian hordes, right?
(Score: 2) by DeVilla on Tuesday May 21 2019, @06:13PM
To be fair, some training will always be needed to take over a non-trivial environment. Things like "we run our builds on Jenkins on this host", "the users ID's are managed in the LDAP server there", "The current diagram of the deployment pipeline is here.", etc.
But of course there are no shortage of stories like
(Score: 5, Insightful) by Farkus888 on Sunday May 19 2019, @02:52AM (3 children)
2 kinds of people and thinking. I haven't quite determined what to call them. Negotiable and non-negotiable interactions with the real world make the difference. Say an engineer and a welder build a steel bridge and it falls down. One of them messed up, no getting out of it. Either the welds were bad or the design was bad. The MBAs at Salesforce aren't playing that same game, they can always blame the techs. Understaffed or under skilled techs due to MBA incompetence doesn't matter, the tech still actually pushed the button so the MBA never gets punished. MBA fires the tech and hires a replacement who is even more under skilled and cheaper speeding the next disaster, then pats himself on the back for a job well done.
(Score: 0) by Anonymous Coward on Sunday May 19 2019, @07:07AM (1 child)
Aye, but you also forgot that for the tech who gets fired, this is a black mark as far as his future (if any) career in IT is concerned, the MBA?, it not just his self congratulatory back patting that happens, he also gets fucking manglement brownie points for firing the poor sod, thus guaranteeing a future internal promotion and/or a better paid position in another organisation.
I've seen this happen far too many times, and not just in IT.
(Score: 2) by Farkus888 on Sunday May 19 2019, @11:02AM
Didn't forget, just got bored of writing and felt I had the core of my idea covered. That is the real problem though. The MBA and his fellow MBA buddies aren't trying to mislead you, they truly believe the version of the story that blames the tech. People who personally know psychics say they start knowing they are just cold reading, but slowly start to believe they have real powers. If they xray the collapsed bridge and the welds were cold, there is no other story for the welder to tell. People who spend all their time in domains with built in weasel opportunities are different than people who primarily spend their time in no wiggle domains.
(Score: 2) by Bot on Sunday May 19 2019, @07:17AM
In fact, the internal structure of your IT with QA and stuff is not going to save you from disaster, if everybody is overworked and trying to meet completely made up deadlines who are simply a way for management to prove they exist.
Overuse of meatbags (still a less demeaning term than HR) should be treated like going around in a car with a windshield so dirty that you barely see through it. The inevitable eventual becomes your fault and it's aggravated by your complete awareness of the situation and lack of rectifying it.
Maximum de-facto working hours should be enforced more than minimum wage, because it's far more important to society.
Account abandoned.
(Score: 0) by Anonymous Coward on Sunday May 19 2019, @07:24AM
The joke there is that it's the ones with the knowledge who'll find employment elsewhere difficult, as most places now seem to have swallowed whatever agile/devops/name-your-fuckwittery kool-aid is currently du jour in MBA-land. It's the ones who eventually hit the job market when the house of cards does finally collapse who'll find employment as they've just been engaged on a project where they've become experienced in the use of these Manglement fuckwitteries..
If age and experience counted for sweet fuck all in one organisation, it apparently counts for nothing elsewhere too...not that I'm speaking from recent bitter experience...
(Score: 4, Informative) by JoeMerchant on Sunday May 19 2019, @01:15PM (1 child)
Only if they are deployed irresponsibly.
"Move fast and break things" is an excellent, very productive, development philosophy - and perfectly safe and acceptable, as long as you make the additional investment of a robust sandbox deployment and test environment.
Using your paying customers as a sandbox is, well, just like the housecat analogy would imply, and customers should react accordingly.
🌻🌻 [google.com]
(Score: 2) by Farkus888 on Monday May 20 2019, @04:21AM
Agreed. For example Agile like grandparent mentioned. Certainly calling a reckless approach to things Agile will still be a reckless approach with the expected outcome that brings. We use a modified Agile for chore and project tracking in my house, post-it's on the wall and all. It is awesome, the house is nicer and more gets done with fewer disputes.
(Score: 0) by Anonymous Coward on Sunday May 19 2019, @02:00PM
TBH I don't think the script was properly tested, which would make agile proponents able to count themselves out. Agile being codeword for "we push git HEAD and call it a day" does not make it so.
(Score: 1) by RandomFactor on Sunday May 19 2019, @12:48AM
our stuff isn't in scope :-\
On the bright side, better now than next weekend I suppose. Not getting those Memorial Day spam^h^h^h^hmarketing emails out would tick off a lot of customers.
В «Правде» нет известий, в «Известиях» нет правды
(Score: 2) by ataradov on Sunday May 19 2019, @12:52AM
Our corporate stuff got turned off, but then went back online some time around noon the same day. Not sure if the data loss was a concern, but it sucked and affected my work for half a day.
Not that SalesForce does not suck when it works as designed.
(Score: 2, Insightful) by Anonymous Coward on Sunday May 19 2019, @01:08AM (5 children)
Who are these people? Do they produce anything of value, besides providing work for computer janitors? What do they do? Pollinate Google with numbers to fatten the portfolio?
What kind of work are you damn kids doing?! Is anybody fixing the roads, the power/phone lines, the damn toilet??! And look at the people you vote into office! What's up with that? You a bunch of damn arsonists too??
(Score: 1, Informative) by Anonymous Coward on Sunday May 19 2019, @02:52AM (4 children)
They're like the Apple of Google. A whole shitload of expensive overhyped services that have no end consumer value but make corporate look good.
(Score: 0) by Anonymous Coward on Sunday May 19 2019, @06:39PM
Metrics! SEO! Analytics! Ad views! User retention!
Looks like it is gonna be a while longer before people realize that centralized services are often more of a liability than the convenience is worth. Some of the prices I've heard for these managed service solutions are so insane that they could hire 1+ full time employees and better hardwsre.
(Score: 1) by Ethanol-fueled on Sunday May 19 2019, @07:59PM (2 children)
Absolutely this. When a former employer of mine "upgraded" their CRM system from Access to Salesforce, regular verbal arguments were breaking out over lots of lost work because Salesforce didn't implement record-locking. You would go to a record, input shit-tons of data as part of a very detailed inspection process, then when you hit the "save" button it would be all like, "LOL sorry, somebody else is viewing this record, and there is no way to save all of your information you just painstakingly entered." This was back around 2015ish. Losing work in that fashion really demotivates and knocks the wind out of your sails, whether it is writing a novel or recording music, or getting job stuff done. We had to develop a whole separate informal system where people would shout across the room that they were editing a record, or ask if others were editing.
Fucking inexcusable. Don't believe the hype. It's the kind of idiocy that only San Francisco could produce.
(Score: 1) by fustakrakich on Monday May 20 2019, @12:22AM
You would go to a record, input shit-tons of data as part of a very detailed inspection process, then when you hit the "save" button it would be all like, "LOL sorry, somebody else is viewing this record, and there is no way to save all of your information you just painstakingly entered."
You work at Wikipedia?
La politica e i criminali sono la stessa cosa..
(Score: 2) by datapharmer on Monday May 20 2019, @02:21PM
Remember that the core of Salesforce goes back 20 years now, so it has a lot of legacy cruft. Although this permissions blunder is in a field of its own, they do know where they've got gaps and need to play catch up. They actually have "lightning live records" on the roadmap to solve this exact problem. Cache invalidation is harder than it sounds.