SoylentNews is people
SoylentNews
A Debian user has recently discovered that systemd prevents the skipping of fsck while booting:
With init, skipping a scheduled fsck during boot was easy, you just pressed Ctrl+c, it was obvious! Today I was late for an online conference. I got home, turned on my computer, and systemd decided it was time to run fsck on my 1TB hard drive. Ok, I just skip it, right? Well, Ctrl+c does not work, ESC does not work, nothing seems to work. I Googled for an answer on my phone but nothing. So, is there a mysterious set of commands they came up with to skip an fsck or is it yet another flaw?
One user chimed in with a hack to work around the flaw, but it involved specifying an argument on the kernel command line. Another user described this so-called "fix" as being "Pretty damn inconvenient and un-discoverable", while yet another pointed out that the "fix" merely prevents "systemd from running fsck in the first place", and it "does not let you cancel a systemd-initiated boot-time fsck which is already in progress."
Further investigation showed that this is a known bug with systemd that was first reported in mid-2011, and remains unfixed as of late December 2014. At least one other user has also fallen victim to this bug.
How could a severe bug of this nature even happen in the first place? How can it remain unfixed over three years after it was first reported?
(Score: 1) by tftp on Sunday December 21 2014, @05:41AM
Drives don't flip bits for no reason. Every data record (sector) has a checksum. If the sector reads OK but a bit is flipped, it's because you wrote it flipped. Don't do that.
One of primary reasons for the need of fsck in days of ext2 was absence of the journal. This means that abrupt reset of the computer could wreck the FS. The same was the norm in days of Windows 95 (FAT). However ext3/ext4 on Linux side (not even mentioning other journaling FS) and NTFS on Windows reduced the need of full scan because everything that one needs to know for recovery is, generally, in the journal. Forcing a check of the filesystem every so many boots is just demonstrating either extreme paranoia, or lack of trust in the FS code. Perhaps back then, 15 years ago, it was a reasonable measure, considering the state of the filesystems in Linux. It is strange to encounter such a thing today. How often does a modern Windows box force you to rescan the HDD? IIRC, it happens only if an error is detected during normal operation - and then rescan is scheduled on next reboot. And yes, you can cancel it :-)
(Score: 2) by sjames on Sunday December 21 2014, @06:37AM
They don't flip for no reason, they flip for a variety of reasons. Spike in vibration while writing an adjacent track, EMI in the drive cable, stray cosmic ray, power glitches, etc.
In SOME of those cases the checksum doesn't match, but you won't know that unless something like a full fsck comes along and detects it for you. In others, the checksum will have been generated AFTER the data got corrupted and so it will perfectly validate the incorrect metadata. Fsck can sanity check that for you.
Journaling in the file system is a great advancement, but it isn't a panacea. Paranoid? Perhaps, but we're talking about servers, not glorified solitaire and minesweeper machines :-)
But Linux does respect user choice. You can use the -c and -i options of tune2fs to change the fsck interval or disable periodic fsck entirely. As long as you avoid systemd you can cancel an fsck if it's not a good time. You can manually command it to do the fsck and reset the countdown.
(Score: 1) by tftp on Sunday December 21 2014, @06:55AM
I agree about servers; but I recall that the darned thing activated when I powered up the laptop at a meeting :-) Don't even remember what it was, SUSE or RedHat.
Checksums are calculated and checked by the hardware. The data rate is way too high to do it in a CPU. Besides, it has to be done atomically, on sector level, when HDD rewrites a sector. But sure, there is always a possibility to screw things up. It may be reasonable to be extra careful on servers. However desktops are just fine with lazy verification.
(Score: 2) by sjames on Sunday December 21 2014, @09:17AM
I agree about servers; but I recall that the darned thing activated when I powered up the laptop at a meeting :-) Don't even remember what it was, SUSE or RedHat.
An excellent example of why fsck might need to be cancelled.
You know which use case best applies to your machine, so you must configure it if you want lazy verification.
(Score: 2) by Immerman on Sunday December 21 2014, @06:45AM
>Drives don't flip bits for no reason.
Of course they do - hence the name "random error". Back in the day you could expect an error on average in one bit out of every 10^14 bits read. Roughly one byte in in 10 terabytes - not so bad when in the day when that was a a ferocious amount of data transfer for a 100MB drive at 50mb/s. Today though, it probably rears it's head a few times in the lifetime of a 1TB drive. And that's traditional, simple, crude even HD technology. Newer tech... well a lot o it hasn't even been out long enough to determine realistic real-world error rates. To say nothing of SSDs - where I just found numbers in the 1 in 10^8 range. That's 1 bit in every 100MB, or one error every few seconds if you're saturating the SATA bus. Seems ridiculous to me, so presumably there's a lot of error correction going on behind the scenes, but even that has it's limits.
(Score: 3, Interesting) by sjames on Sunday December 21 2014, @07:44AM
A simple read error is not a big deal except sometimes for performance, just read again until the checksum verifies the data. Often that happens at the hardware level. Write errors or bit flipping on the media IS a big deal since that is actually corrupt data. Worst of all is when the flipped bit happens prior to checksumming. That can cause silent corruption.
That's why BTRFS and ZFS do checksumming at the file system level and support raid-like storage. Unlike a device level RAID, they can decide which disk has the correct data in cases of bit flip and can then re-write the correct data to the other drive. Even on a single drive, btrfs likes to write two copies of the metadata.
(Score: 2) by FatPhil on Sunday December 21 2014, @08:44AM
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 2) by tonyPick on Sunday December 21 2014, @10:57AM
Also leads to (potentially) bigger problems if you use raid: http://www.zdnet.com/article/has-raid5-stopped-working/ [zdnet.com]
(Score: 2) by FatPhil on Sunday December 21 2014, @11:08AM
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 2) by cafebabe on Thursday December 25 2014, @08:33AM
A similar problem occurs with networking. For a few packets over a few segments, 16 bit checksums may be sufficient. However, 0.01% corruption of Jumbo Frames over 13 hops leads to silent corruption approximately once per hour. Worse links or small packets may lead to significantly higher rates of corruption.
Unfortunately not. One person's payload is another person's header. So, if you aren't processing the payload immediately, the corruption is silent. Even if you are processing the payload immediately, corruption may elude validation or parsing.
1702845791×2
(Score: 2) by FatPhil on Sunday December 21 2014, @08:27AM
In that case - turn of filesystem checking completely. Good luck, and enjoy your bitrot.
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 2) by cafebabe on Thursday December 25 2014, @08:31AM
That isn't an end-to-end checksum. Also, ATA drives autonomously substitute sectors when regenerative checksums exceed a threshold. What algorithm and threshold? That is proprietary and varies with each revision of firmware on each model of each manufacturer's drives.
After every Blue Screen Of Death.
1702845791×2