SoylentNews is people
SoylentNews
A man appears to have deleted his entire company with one mistaken piece of code.
By accidentally telling his computer to delete everything in his servers, hosting provider Marco Marsala has seemingly removed all trace of his company and the websites that he looks after for his customers.
Mr Marsala wrote on a forum for server experts called Server Fault that he was now stuck after having accidentally run destructive code on his own computers. But far from advising them how to fix it, most experts informed him that he had just accidentally deleted the data of his company and its clients, and in so doing had probably destroyed his entire company with just one line of code.
The problem command was "rm -rf": a basic piece of code that will delete everything it is told to. The "rm" tells the computer to remove; the r deletes everything within a given directory; and the f stands for "force", telling the computer to ignore the usual warnings that come when deleting files.
His backups were also mounted at the time. That's a nightmare scenario, right there.
(Score: 3, Interesting) by KilroySmith on Friday April 15 2016, @09:57PM
Did that once - in my first week at a new job. The other developers started asking "Are you having problems with the server", and I stared at my command line...and the horror slowly rose.
We did, however, have backups....
(Score: 4, Funny) by Blightbow on Friday April 15 2016, @11:17PM
My favorite was a server being reported as "not working" by a developer who had complained loud enough to get root on a system. Sure enough, it was tanked.
Booted to an emergency image, did a full audit, and the command history looked something like this:
cd /some/tomcat
rm -r /lib
rm -r -9 /lib
man rm
rm -r -f /lib
The "rm -r -9" was my favorite. "It's not working! -9 means do it harder, right?!"
Once the server exploded, he rebooted and played innocent. Bad gamble, that...
(Score: 2) by butthurt on Friday April 15 2016, @11:40PM
I lost a file I'd not backed up, by using the ">" operator rather than the intended ">>" to append.
(Score: 2) by richtopia on Saturday April 16 2016, @12:00AM
I did something similar on my personal server. The issue was I has a symbolic link in the folder I was removing. I realized after 30 seconds that it was taking too long, but still lost 300GB of data before mashing ctrl-c.
(Score: 0) by Anonymous Coward on Saturday April 16 2016, @05:32AM
That's how users teach noob admins not to nuke home dirs that way.
(Score: 3, Interesting) by Whoever on Saturday April 16 2016, @05:57AM
A IT contractor for a small company had been asked to add another hard drive to the company file server.
The server has 2 drives: /dev/hda and /dev/hdb.
He adds the hard drive and assumes that the new drive is /dev/hdc. Proceeds to format hdc, which was in fact the old drive with the entire network files on it, while the new drive was now /dev/hdb.
(Score: 0) by Anonymous Coward on Saturday April 16 2016, @04:27PM
For people wondering why: that wasn't that uncommon back in the day. On some distros, /dev/hd[a-d] were commonly assigned based on which pata port it was attached to. Therefore, 1 disk on the primary master was /dev/hda, the second drive on the secondary master was /dev/hdc. However, others would assign them based on the order they were attached. Still others would assign them in a somewhat random order based on a couple of factors like spin up time, other kernel activity, etc. That is why many distros now use various drive ID rules, like UUID, to make sure you get the same disk regardless of other factors.
(Score: 5, Insightful) by takyon on Friday April 15 2016, @09:58PM
The code was just doing its job.
[SIG] 10/28/2017: Soylent Upgrade v14 [soylentnews.org]
(Score: 5, Insightful) by mhajicek on Friday April 15 2016, @10:04PM
All of his backups were mounted? Even the offsite backups that any responsible hosting service would have?
The spacelike surfaces of time foliations can have a cusp at the surface of discontinuity. - P. Hajicek
(Score: 0) by Anonymous Coward on Friday April 15 2016, @11:30PM
They were a company that researches quantum entanglement and were developing a "Quantum Entanglement Datastore (QED)". I guess it worked ...
(Score: 0) by Anonymous Coward on Saturday April 16 2016, @06:21PM
consider yourself modded +6 insightful (it doesn't go higher than 5)
(Score: 4, Funny) by GungnirSniper on Friday April 15 2016, @10:05PM
Garbage in, zeros out.
Tips for better submissions to help our site grow. [soylentnews.org]
(Score: 5, Informative) by Anonymous Coward on Friday April 15 2016, @10:12PM
There are too many clues this is a troll. The first is that a person on that level knows enough to write the scripts and run the VPSes for thousands of customers so far without incident, so probably not an intro level person. Second is that he had backups mounted while making system changes. Third is that in response to instructions to make a copy of the disks with dd, his response is that he swapped if and of (which wouldn't actually do any damage in this case). Fourth is that his given script, it wouldn't actually result in deleting everything like he said due to --preserve-root being the default now. Fifth is that he seemed to have trouble with other commands, screwing them up just enough to look like an accident but not enough for outright suspicion.
And finally, he states in a later comment that he recovered everything (without specifying how), so this isn't news anyway.
(Score: 2, Informative) by Blightbow on Friday April 15 2016, @11:08PM
It was in fact a troll.
(Score: 2) by Pino P on Friday April 15 2016, @11:17PM
it wouldn't actually result in deleting everything like he said due to --preserve-root being the default now.
Even on non-GNU implementations of rm, such as those in BusyBox and FreeBSD? One of my Twitter followers thinks --preserve-root is not the answer [twitter.com] as much as set -u [davidpashley.com]. POSIX sh supports a strict mode enabled by set -u, which protects programs other than rm from unintended behavior due to uninitialized variables. It's the shell equivalent of the warning-to-exception snippet in the PHP manual [php.net].
(Score: 0) by Anonymous Coward on Friday April 15 2016, @11:50PM
I think all shell scripts should start with "set -ueo pipefail" for protection from all sorts of errors and sysadmins should consider having that be the default in general.
While true that I assumed he was using Linux and specifically GNU userland on Linux, I think that is a fair assumption given the relative market share in the server space.
(Score: 0) by Anonymous Coward on Friday April 15 2016, @11:57PM
I think all shell scripts should start with "set -ueo pipefail"
I think all shell scripts should start with: sudo make me a sandwich -now -please
(Score: 2) by darkfeline on Saturday April 16 2016, @02:42AM
I think that shell scripts that would need to start with "set -ueo pipefail" should be considered for rewriting in a "real" scripting language, like Perl, Python, or Ruby, or perhaps even a language like Go, which provides static type checking, goroutines, and channels (which are superb for running different commands piped to different other commands asynchronously).
Join the SDF Public Access UNIX System today!
(Score: 1) by Francis on Saturday April 16 2016, @12:45AM
I don't see any reference to that sort of protection on my FreeBSD install, I'm using ZFS, so I could relatively easily unmount and disconnect the non-ZFS volumes and take one last snapshot before executing the command, but I'm too lazy to purposefully nuke my system knowing that rm -rf works the way that it's designed to work.
One of the reasons why I'm on FreeBSD is because it trusts me to know what the fuck I'm doing. The Linux "fix" at best keeps you from deleting the whole system, it doesn't protect you against executing that command from inside of /home where most or all of the files that you're likely to care about are located.
(Score: 3, Informative) by frojack on Saturday April 16 2016, @07:19AM
Yes it was a troll
http://www.repubblica.it/tecnologia/2016/04/15/news/cancella_l_azienda_per_sbaglio_la_disavventura_tecnologica_di_marco_marsala-137693154/ [repubblica.it]
And its amazing just how widely the troll has spread.
No, you are mistaken. I've always had this sig.
(Score: 2) by turgid on Saturday April 16 2016, @09:19AM
And look at the syntax of his rm command. No $ on the {foo} or {bar}.
I refuse to engage in a battle of wits with an unarmed opponent [wikipedia.org].
(Score: 1, Interesting) by Anonymous Coward on Friday April 15 2016, @10:30PM
The novice user fired up some long-forgotten text editor on one of his documents and typed 'edit'.
e - selects everything
d - deletes current selection
i - enter insert mode
t - the letter 't'
(Score: 2) by devlux on Friday April 15 2016, @10:33PM
I have in fact had this happen to me.
Completely wiped out a text document while trying to type.
Not really fun when it's your thesis, and everything is stored on floppies.
Guessing it was harder to delete punch card backups though? Anyone have experience with that one?
(Score: 2, Interesting) by Anonymous Coward on Friday April 15 2016, @11:35PM
Guessing it was harder to delete punch card backups though? Anyone have experience with that one?
We have to schedule time to run our programs... My time has finally come to compile (onto tape), and I'm carrying several stack of freshly punched cards. An intern carrying coffee takes a corner and slams into me, cards explode into the air in slow motion I see weeks of my life flashing before my eyes. My glazed expression is broken by the visibly shaking intern's squeak, "Oh crap! I'm so sooooo sorry." He frantically begins picking up the cards. I slowly turn and walk away, still holding a few handfuls of coffee stained punched cards. "Meet me at my desk. You're going to learn how to punch cards. Don't bring coffee."
The guys who were running their program ahead of me turn look at each other. I hear a whispered, "YES!" They must have needed more CPU time. I begin to suspect I've been set up... The entire time the intern is with me sorting the numbered cards and punching new ones for the next few days I speak only in metaphorical double entendre concerning "what occurred". I conclude he's either extremely stupid, or a very convincing spy.
(Score: 2) by Joe Desertrat on Saturday April 16 2016, @01:12AM
Guessing it was harder to delete punch card backups though? Anyone have experience with that one?
Watched one time as someone loaded a huge stack of punch cards into the feeder. About halfway through it started shooting cards into the air instead of through the feeder. Being a new student at the time, I made the mistake of asking him why he didn't just sort them out rather then dumping the whole thing in the garbage. Learned something that day.
(Score: 2) by butthurt on Friday April 15 2016, @11:24PM
It's easier in a GUI:
(Score: 2) by Marand on Saturday April 16 2016, @12:27AM
It's easier in a GUI:
With emacs, select-all is C-x h , so you probably won't be doing it by mistake. Also, typing doesn't replace selection by default*, so even if you accidentally C-x h and then type it won't replace anything unless you explicitly press something like delete or backspace immediately after selection. </smug emacs user>
* If you actually want that behaviour for some reason, though, you can get it by adding (delete-selection-mode 1) to your emacs config. Or temporarily by typing it into a buffer and using M-x eval-last-sexp on the line.
(Score: 2, Interesting) by anubi on Saturday April 16 2016, @02:22AM
I had something similar happen to me in aerospace.
New managers came in... they just had to change everything. You know the routine ... they have to show they are spending a lot of money so their salaries look small by comparison. Investors just seem to love that kind of thing. So do governments.
So I end up with another machine while the one I had ( which I had configured exactly to my liking ) was deemed too old to keep. Knowing there were a lot of programs I really needed, I saved them off to floppies, knowing management would soon call men wearing badgehats to take my machine away.
Sure enough, men came and carted off my old machine, which was extremely well tuned and working perfectly, and replaced it with some new machine that to me worked like a pile of crap. It had on it what at that time was a brand new program... Windows 2.0. I was being upgraded from a lowly 286 to a 386SX. Now, mind you, I had dual hard drives and 16 MEG of RAM in that old 286 configured as a ramdisk, which I used mostly for SPICE simulations. I also knew the guy who had taken the original Berkeley SPICE 3G5 and rewrote it from Fortran to Borland C++ to run it on PC's... and he had helped me optimize that old 286 for SPICE runs. That was the main thing I used my machine for - and oh yes, I did have copies of Mathcad 2, Eureka, Borland C++, GWBasic, Turbo Assembler, and a host of other goodies on that machine. Technical goodies. But I did not do fonts or prettyprint presentation materials - and that's what Management needed.... presentation. These guys were too high up to look at system font and make any sense of it.
The new machine seemed awful slow - I mean excruciatingly slow. It seemed it was taking forever to do a spice run. I was used to seconds... the new machine was taking hours! Well, one thing - I did not have my ramdisk anymore - and the hard disk light stayed on constantly, and the drive constantly churned between the source file, print file, and library file cylinders - which did not happen in my dual hard drive + ramdisk 286. Judging by the amount of churning, I figured I must have a terribly fragmented hard drive.
Anyone remember that old DOS file "Compress.exe"? It was a DOS disk defragmenter. PCTools, I believe, put it out. I ran it.
Then all hell broke loose. Nothing worked.
I managed to reformat the hard drive and get my DOS stuff running again, but the Windows stuff was all gone forever.
That day, I learned about a whole new disk technology called "double-space".
That day was also my pivot point to my fall from grace in the aerospace industry.
I was laid off by that company, but I did end up getting my machine back through surplus sales. I reconfigured it just as I had it, and loaded it back up and continued using it at my house. I still have the case, keyboard, and power supply running, but its innards have been upgraded a Pentium. It still runs DOS and WIN95. I still do spice runs on it occasionally, and for years it did all my Futurenet and PADS-PCB circuit layout - however by now, 99% of my SPICE runs are on LTSpice and EAGLE for schematics and PCB - run from a cheap WalMart HP WIN7 laptop - which does everything I need it to do.
I do not know where so many people get the idea one has to spend so much money when it is so much more productive to know what you are doing.
"Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
(Score: 2) by Justin Case on Saturday April 16 2016, @02:40PM
I do not know where so many people get the idea one has to spend so much money when it is so much more productive to know what you are doing.
Because most people don't know what they're doing, but they do know how to spend money!
Sorry to hear your story, BTY. I too was bitten hard by "double-space". It was near the end of a burst of creativity and I hadn't run my tape backups yet. I still fondly remember some of that code, never recovered, clumsily and partially re-implemented.
I renamed dblspc.dll to dvlspc.dll where "dvl" was my abbreviation for "Devil". (Might not have been a .dll suffix, it was a while ago.)
(Score: 1) by anubi on Saturday April 16 2016, @10:19PM
It sure is a lot easier to spend money than it is to make it, eh?
That was my first experience with doublespace. Once I found out what was going on, and why my machine was running so poorly, I threw a fit. Right then and there in the conference room.
"Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
(Score: 3, Interesting) by MichaelDavidCrawford on Friday April 15 2016, @10:32PM
leave your backups unmounted unless you're specifically making a backup.
I expect there will be lots of lawsuits, doubtlessly he'll have to declare bankrupcy.
I lost the source code to a one-month project once. I was flying home two days later. Without telling anyone, I rewrote the whole thing - and did a better job - in just one day.
Yes I Have No Bananas. [gofundme.com]
(Score: 3, Insightful) by Dunbal on Friday April 15 2016, @11:03PM
leave your backups unmounted unless you're specifically making a backup.
And preferably in a different building, or even a different state.
(Score: 3, Insightful) by Fnord666 on Friday April 15 2016, @10:51PM
the r deletes everything within a given directory;
Incorrect. The -r option says to apply the action recursively in any subdirectories as well.
(Score: 0) by Anonymous Coward on Friday April 15 2016, @10:54PM
If your product and business prospects were toast, then torching the place might not do much, but maybe if you can arrange to get "hacked"...
(Score: 2) by butthurt on Friday April 15 2016, @11:35PM
This reminds me of "PyroFlipper [twicsy.com]."
(Score: 0) by Anonymous Coward on Friday April 15 2016, @10:57PM
reversed if and of and made it worse. Sounds more like a BS story to get attention.
(Score: 0) by Anonymous Coward on Friday April 15 2016, @11:11PM
Yeah, that certainly looks bad. Except, as mentioned above, swapping "if" and "of" would not result in any damage. The reason is that the input file is empty. This means that it wouldn't copy anything and all the data would be the same.
(Score: 2) by Dunbal on Friday April 15 2016, @11:01PM
This backup concept is as old as life itself.
(Score: 5, Informative) by Blightbow on Friday April 15 2016, @11:05PM
The original Q&A on Serverfault has since been locked and flagged as a hoax.
http://serverfault.com/questions/769357/recovering-from-a-rm-rf [serverfault.com]
The author fessed up here (Italian):
http://www.repubblica.it/tecnologia/2016/04/15/news/cancella_l_azienda_per_sbaglio_la_disavventura_tecnologica_di_marco_marsala-137693154/ [repubblica.it]
(Score: 0) by Anonymous Coward on Friday April 15 2016, @11:07PM
I suspect its a fake story to get attention, or clicks for cash..
(Score: 1) by ShadowSystems on Friday April 15 2016, @11:59PM
Must. Resist. Urge. To. Bwa Hahahahahahahahahahahahhahhaha...
*Sighs & wipes away the tears of mirth*
Damn, now that's funny.
Definitely glad it wasn't me. =-)p
(Score: 2) by coolgopher on Saturday April 16 2016, @12:57AM
You didn't mount tape. And if you rm'd your tape device nodes you were thankful for the opportunity to remember what all the various characters at the end of each device node did in terms of compression etc. Bah, kids these days... ;)
(Score: 0) by Anonymous Coward on Saturday April 16 2016, @01:55AM
SmartOS would simply report an error, see https://youtu.be/l6XQUciI-Sc?t=4864 [youtu.be] (That video is a fantastic interview by the way: it makes me want to work for Joyent). Just as Bryan Cantrill pointed out in the interview, this commonly happens due to bugs with unset variables, and is never what a user wants. Thus throwing an error is the right thing to do (So that is what SmartOS does). It just happens to be the case that its undefined behavior on POSIX anyway (it deletes the current working directory) so you can give the nice error and still be complaint, but the should not be important. As software engineers, we often forget adjusting software like rm to be nice to users is even an option.
(Score: 0) by Anonymous Coward on Saturday April 16 2016, @02:24AM
Sounds like SmartOS is trying to be smarter than the owner. Sure, on the whole, it may be an accurate assumption that the owner doesn't know what they're doing. However, doing such things inevitably leads to conflicts with owners who are indeed smarter than the dumb assumptions of operating systems.
You don't have to 'rm -rf' to screw up an operating system; 'chmod -R 000 /' is just about as much work to clean up.
(Score: 3, Informative) by butthurt on Saturday April 16 2016, @01:58AM
The newspaper account says that
Yet (had this happened) the advice to "turn off everything right now and hand your disks over to a reputable data recovery company" could work. The reason is that rm [oracle.com] "removes the directory entry," typically by calling unlink(2) [oracle.com]. The storage space is then deemed available, and can be overwritten when files are created or enlarged.
This is different from what we call "wiping" in which the contents of a file are overwritten by a command such as srm. The reasons it's desirable to turn off the power immediately are that an orderly shut-down involves some writing of files, and because file-systems are sometimes mounted asynchronously in which case there's a chance the unlinking might not have been fully written to disk.
(Score: 0) by Anonymous Coward on Saturday April 16 2016, @05:12AM
Everyone does that once.
In my case, I was helping a DEC field engineer test a new 6250 bpi tape drive on our VAX 11-750.
We dumped the root partition to tape, then extracted it into a test directory, then deleted what we had extracted.
Except ... oops! ... I was in the root directory when I did that.
My boss said, "That's what backups are for", and, since I was the guy who did the backups, we used the opportunity to upgrade from 4.2 BSD to 4.3 BSD, then reloaded everyone's home directories, configured the network interface (which tapped into an Ethernet cable as thick as my thumb), and we were good to go.
Apparently whomever did this was a beginner.
Seasoned pros run their commands with the '-n' switch to check syntax before they execute it ... and they anticipate worst-case scenarios, drawing upon their rich store of unhappy experiences. They add code to test for directories before they operate upon them. They unmount devices when they are through with them. They check filesystems after they unmount them - even when the unmount was error-free. Etc.
That's what employers get for hiring kids fresh out of college and deliberately snubbing older, wiser heads.
What's the value of a seasoned UNIX systems administrator?
Whatever it is, it's orders of magnitude less than the value of those ~1300 websites and the man-hours that went into building and maintaining them, the data contained within their databases, the transactions generated by their media, the business intelligence embedded within their middleware, the unique security tokens, etc.
Learning Ansible, Salt, Chef or Puppet is TRIVIAL ... compared to learning how to be diligent, and responsible, and patient.
But, hey, what do I know?
I've only been in this business longer than most recruiters and some venture capitalists have been alive.
These are the same people who want "rock star" sysadmins and developers ... but somehow forget that Keith Richards was born in 1943 ... Rod Stewart, and Pete Townshend, were born in 1945 ... Gene Simmons was born in 1949.
~childo
(Score: 0) by Anonymous Coward on Saturday April 16 2016, @05:18PM
It appears now it's a hoax:
http://www.telegraph.co.uk/technology/2016/04/16/did-it-guy-really-delete-entire-company-with-wrong-bit-of-code/ [telegraph.co.uk]