Stories
Slash Boxes
Comments

SoylentNews is people

posted by Fnord666 on Sunday May 10 2020, @06:47PM   Printer-friendly
from the stormy-weather dept.

System adminsitrator Chris Siebenmann has found Modern versions of systemd can cause an unmount storm during shutdowns:

One of my discoveries about Ubuntu 20.04 is that my test machine can trigger the kernel's out of memory killing during shutdown. My test virtual machine has 4 GB of RAM and 1 GB of swap, but it also has 347 NFS[*] mounts, and after some investigation, what appears to be happening is that in the 20.04 version of systemd (systemd 245 plus whatever changes Ubuntu has made), systemd now seems to try to run umount for all of those filesystems all at once (which also starts a umount.nfs process for each one). On 20.04, this is apparently enough to OOM[**] my test machine.

[...] Unfortunately, so far I haven't found a way to control this in systemd. There appears to be no way to set limits on how many unmounts systemd will try to do at once (or in general how many units it will try to stop at once, even if that requires running programs). Nor can we readily modify the mount units, because all of our NFS mounts are done through shell scripts by directly calling mount; they don't exist in /etc/fstab or as actual .mount units.

[*] NFS: Network File System
[**] OOM Out of memory.

We've been here before and there is certainly more where that came from.

Previously:
(2020) Linux Home Directory Management is About to Undergo Major Change
(2019) System Down: A systemd-journald Exploit
(2017) Savaged by Systemd
(2017) Linux systemd Gives Root Privileges to Invalid Usernames
(2016) Systemd Crashing Bug
(2015) tmux Coders Asked to Add Special Code for systemd
(2016) SystemD Mounts EFI pseudo-fs RW, Facilitates Permanently Bricking Laptops, Closes Bug Invalid
(2015) A Technical Critique of Systemd
(2014) Devuan Developers Can Be Reached Via vua@debianfork.org
(2014) Systemd-resolved Subject to Cache Poisoning


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 3, Interesting) by DeVilla on Tuesday May 12 2020, @06:52AM

    by DeVilla (5354) on Tuesday May 12 2020, @06:52AM (#993267)

    If systemd had umount()ed serially or called `umount -a`, this would not have happened.

    Several people are arguing things about whether you should handle weird errors or not. That misses the point. Systemd took over the process of unmounting files systems and decided to do so in parallel. If you are doing parallel processing you monitor your resources. Systemd in not doing this.

    Systemd's allowing users to unknowingly to define a fork bomb without checking if the resource needed are available nor providing the user a mechanism to check or prevent it. The fact that it's a umount() is beside the point. If I have a large system (assumes the system has resource to run everything in steady state) can I trigger a fork bomb induced OMM by starting tons of small, independent services via unit files? If systemd is doing it's job correctly, then the answer should be no.

    Starting Score:    1  point
    Moderation   +1  
       Interesting=1, Total=1
    Extra 'Interesting' Modifier   0  
    Karma-Bonus Modifier   +1  

    Total Score:   3