SoylentNews is people
SoylentNews
from the who-needs-QA-when-we-can-test-it-on-production dept.
At around 9:15 UTC [17 May] Salesforce pushed a database script update that was intended to add modify all permissions to a specific internal profile used by their Pardot service. Due to a scripting error View and Modify All Objects Permission was granted to all user profiles for all organizations that ever had the Pardot product, including public facing community instances. This was of course a security nightmare for customers, especially those in the Financial and Health sectors, and an emergency change was pushed around 10:00 UTC to revoke all permissions to all profiles except for administrators. No announcement was made on their status sites due to the potential for bad actors to take advantage of the security issue that was introduced until the databases could be locked down. Further action was taken around 11:00 UTC to take down PODS completely, likely to further mitigate access risk which effectively expanded the outage to customers that never used Pardot but shared an instance with customers who did.
Salesforce is holding hourly calls, and recently admitted that the script had run both in their production PODS and also in the Passive Disaster Recovery Instances, complicating the ability to recover from the issue. There is currently no ETA for recovery, though it is still their hope that they will not have any data loss. They are beginning to bring back up instances, but only administrators will have access initially and it will require additional time before administrators will be able to modify permissions and rebuild profiles and there will be a longer wait yet before profile settings can be restored from backup.
Coverage at: Geekwire, The Register, and reddit
(Score: 5, Informative) by sshelton76 on Sunday May 19 2019, @12:46AM (11 children)
This demonstrates pretty clearly why modern paradigms such as "Agile", "move fast and break things" and even devops is bad.
Programming and QA and sys-admin duties should never, ever, ever be combined into a single department, let alone a single person.
I find it telling that just a little under a year ago Salesforce got rid of QA by merging it with Dev and firing anyone who couldn't adapt.
https://www.indystar.com/story/money/2018/08/06/salesforce-reorganization-shed-workers-indianapolis/914544002/ [indystar.com]
They then fired a bunch of their long time guys and farmed their duties out to the lowest bidder from India, but before forcing them to train their replacements.
https://h1bdata.info/listlca.php?em=SALESFORCE [h1bdata.info]
As a result, a lot of institutional knowledge is just gone and the people left behind are those who didn't have the skills to get employed elsewhere before the house of cards collapsed.
Something like this NEVER would have made it out of Q&A had they just left things in place. But chasing the next quarter's "growth targets" and short sighted managers who can't see past their next bonus are directly the cause.
Note to anyone considering outsourcing... You will save a little money upfront because these guys will over promise and under deliver. They overstate their skills and it isn't until everything implodes like this that you, as a person who has not done the job every day for years, will realize that you were conned. Good luck hiring back the guys and gals you let go.
(Score: 4, Insightful) by sshelton76 on Sunday May 19 2019, @12:50AM (2 children)
Should read, "but not before forcing them to train their replacements.
Which brings up another question... If a person is made to train their H1B replacement, doesn't that sorta say two things.
First off the H1B didn't already have the skills by virtue of needing to be trained, ergo why was he or she brought over in the first place?
Secondly that the H1B wasn't really needed because there was already someone doing the job and the whole point of an H1B is that there is no one in the US labor pool who has the skills to do the job?
(Score: -1, Offtopic) by Anonymous Coward on Sunday May 19 2019, @02:35AM
Woah, wait? Didn't Trump! Trump! Trump! fix that already? That's why he's resorting to tariffs, concentration camps, and preparing to make use [wsws.org] of the military's Operation Jade Helm training, because not even shutting down the H1B program has been able to stem the barbarian hordes, right?
(Score: 2) by DeVilla on Tuesday May 21 2019, @06:13PM
To be fair, some training will always be needed to take over a non-trivial environment. Things like "we run our builds on Jenkins on this host", "the users ID's are managed in the LDAP server there", "The current diagram of the deployment pipeline is here.", etc.
But of course there are no shortage of stories like
(Score: 5, Insightful) by Farkus888 on Sunday May 19 2019, @02:52AM (3 children)
2 kinds of people and thinking. I haven't quite determined what to call them. Negotiable and non-negotiable interactions with the real world make the difference. Say an engineer and a welder build a steel bridge and it falls down. One of them messed up, no getting out of it. Either the welds were bad or the design was bad. The MBAs at Salesforce aren't playing that same game, they can always blame the techs. Understaffed or under skilled techs due to MBA incompetence doesn't matter, the tech still actually pushed the button so the MBA never gets punished. MBA fires the tech and hires a replacement who is even more under skilled and cheaper speeding the next disaster, then pats himself on the back for a job well done.
(Score: 0) by Anonymous Coward on Sunday May 19 2019, @07:07AM (1 child)
Aye, but you also forgot that for the tech who gets fired, this is a black mark as far as his future (if any) career in IT is concerned, the MBA?, it not just his self congratulatory back patting that happens, he also gets fucking manglement brownie points for firing the poor sod, thus guaranteeing a future internal promotion and/or a better paid position in another organisation.
I've seen this happen far too many times, and not just in IT.
(Score: 2) by Farkus888 on Sunday May 19 2019, @11:02AM
Didn't forget, just got bored of writing and felt I had the core of my idea covered. That is the real problem though. The MBA and his fellow MBA buddies aren't trying to mislead you, they truly believe the version of the story that blames the tech. People who personally know psychics say they start knowing they are just cold reading, but slowly start to believe they have real powers. If they xray the collapsed bridge and the welds were cold, there is no other story for the welder to tell. People who spend all their time in domains with built in weasel opportunities are different than people who primarily spend their time in no wiggle domains.
(Score: 2) by Bot on Sunday May 19 2019, @07:17AM
In fact, the internal structure of your IT with QA and stuff is not going to save you from disaster, if everybody is overworked and trying to meet completely made up deadlines who are simply a way for management to prove they exist.
Overuse of meatbags (still a less demeaning term than HR) should be treated like going around in a car with a windshield so dirty that you barely see through it. The inevitable eventual becomes your fault and it's aggravated by your complete awareness of the situation and lack of rectifying it.
Maximum de-facto working hours should be enforced more than minimum wage, because it's far more important to society.
Account abandoned.
(Score: 0) by Anonymous Coward on Sunday May 19 2019, @07:24AM
The joke there is that it's the ones with the knowledge who'll find employment elsewhere difficult, as most places now seem to have swallowed whatever agile/devops/name-your-fuckwittery kool-aid is currently du jour in MBA-land. It's the ones who eventually hit the job market when the house of cards does finally collapse who'll find employment as they've just been engaged on a project where they've become experienced in the use of these Manglement fuckwitteries..
If age and experience counted for sweet fuck all in one organisation, it apparently counts for nothing elsewhere too...not that I'm speaking from recent bitter experience...
(Score: 4, Informative) by JoeMerchant on Sunday May 19 2019, @01:15PM (1 child)
Only if they are deployed irresponsibly.
"Move fast and break things" is an excellent, very productive, development philosophy - and perfectly safe and acceptable, as long as you make the additional investment of a robust sandbox deployment and test environment.
Using your paying customers as a sandbox is, well, just like the housecat analogy would imply, and customers should react accordingly.
Україна досі не є частиною Росії Слава Україні🌻 https://news.stanford.edu/2023/02/17/will-russia-ukraine-war-end
(Score: 2) by Farkus888 on Monday May 20 2019, @04:21AM
Agreed. For example Agile like grandparent mentioned. Certainly calling a reckless approach to things Agile will still be a reckless approach with the expected outcome that brings. We use a modified Agile for chore and project tracking in my house, post-it's on the wall and all. It is awesome, the house is nicer and more gets done with fewer disputes.
(Score: 0) by Anonymous Coward on Sunday May 19 2019, @02:00PM
TBH I don't think the script was properly tested, which would make agile proponents able to count themselves out. Agile being codeword for "we push git HEAD and call it a day" does not make it so.