SoylentNews is people
SoylentNews
System adminsitrator Chris Siebenmann has found Modern versions of systemd can cause an unmount storm during shutdowns:
One of my discoveries about Ubuntu 20.04 is that my test machine can trigger the kernel's out of memory killing during shutdown. My test virtual machine has 4 GB of RAM and 1 GB of swap, but it also has 347 NFS[*] mounts, and after some investigation, what appears to be happening is that in the 20.04 version of systemd (systemd 245 plus whatever changes Ubuntu has made), systemd now seems to try to run umount for all of those filesystems all at once (which also starts a umount.nfs process for each one). On 20.04, this is apparently enough to OOM[**] my test machine.
[...] Unfortunately, so far I haven't found a way to control this in systemd. There appears to be no way to set limits on how many unmounts systemd will try to do at once (or in general how many units it will try to stop at once, even if that requires running programs). Nor can we readily modify the mount units, because all of our NFS mounts are done through shell scripts by directly calling
mount; they don't exist in/etc/fstabor as actual.mountunits.
[*] NFS: Network File System
[**] OOM Out of memory.
We've been here before and there is certainly more where that came from.
Previously:
(2020) Linux Home Directory Management is About to Undergo Major Change
(2019) System Down: A systemd-journald Exploit
(2017) Savaged by Systemd
(2017) Linux systemd Gives Root Privileges to Invalid Usernames
(2016) Systemd Crashing Bug
(2015) tmux Coders Asked to Add Special Code for systemd
(2016) SystemD Mounts EFI pseudo-fs RW, Facilitates Permanently Bricking Laptops, Closes Bug Invalid
(2015) A Technical Critique of Systemd
(2014) Devuan Developers Can Be Reached Via vua@debianfork.org
(2014) Systemd-resolved Subject to Cache Poisoning
(Score: 2) by rleigh on Monday May 11 2020, @06:08PM (1 child)
Competent software developers ensure that such situations can't arise in the first place. As well as competent developers, you also need a good specification and a good design, which also take these situations into account, as well as unit and integration testing that tests the extremes. It should all be documented, up front. Do you think the systemd developers did this, or did they just bash out some code and hope for the best? The most critical part of the system should not be written by dangerously incompetent arseholes.
Testing parallelism is hard, but several of the projects I work on have threading tests which are designed to provoke edge cases with locking and resource limitations, it can be done.
This isn't just the systemd developers. I could previously cause a kernel panic with LVM just by creating and deleting a few snapshots in parallel. Some kernel locking defect I assume. Back when I was testing Debian by rebuilding the whole archive on a 24 CPU system, I thought I'd test out ~30 parallel builds, and it repeatedly panicked the system within 5 minutes every time. No idea if it's fixed, I gave up on it at this stage, but that usage had clearly not been tested at all.
(Score: 0) by Anonymous Coward on Wednesday May 13 2020, @02:42PM
>Competent software developers
This should be corrected to say "Software developers with an excessive amount of time and fungible resources on their hands". It's easy to ask for these things, a lot harder to swallow when you get stuck with the bill.