Stories
Slash Boxes
Comments

SoylentNews is people

posted by Fnord666 on Sunday May 10 2020, @06:47PM   Printer-friendly
from the stormy-weather dept.

System adminsitrator Chris Siebenmann has found Modern versions of systemd can cause an unmount storm during shutdowns:

One of my discoveries about Ubuntu 20.04 is that my test machine can trigger the kernel's out of memory killing during shutdown. My test virtual machine has 4 GB of RAM and 1 GB of swap, but it also has 347 NFS[*] mounts, and after some investigation, what appears to be happening is that in the 20.04 version of systemd (systemd 245 plus whatever changes Ubuntu has made), systemd now seems to try to run umount for all of those filesystems all at once (which also starts a umount.nfs process for each one). On 20.04, this is apparently enough to OOM[**] my test machine.

[...] Unfortunately, so far I haven't found a way to control this in systemd. There appears to be no way to set limits on how many unmounts systemd will try to do at once (or in general how many units it will try to stop at once, even if that requires running programs). Nor can we readily modify the mount units, because all of our NFS mounts are done through shell scripts by directly calling mount; they don't exist in /etc/fstab or as actual .mount units.

[*] NFS: Network File System
[**] OOM Out of memory.

We've been here before and there is certainly more where that came from.

Previously:
(2020) Linux Home Directory Management is About to Undergo Major Change
(2019) System Down: A systemd-journald Exploit
(2017) Savaged by Systemd
(2017) Linux systemd Gives Root Privileges to Invalid Usernames
(2016) Systemd Crashing Bug
(2015) tmux Coders Asked to Add Special Code for systemd
(2016) SystemD Mounts EFI pseudo-fs RW, Facilitates Permanently Bricking Laptops, Closes Bug Invalid
(2015) A Technical Critique of Systemd
(2014) Devuan Developers Can Be Reached Via vua@debianfork.org
(2014) Systemd-resolved Subject to Cache Poisoning


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 3, Interesting) by driverless on Monday May 11 2020, @11:55AM (3 children)

    by driverless (4770) on Monday May 11 2020, @11:55AM (#992757)

    AFAIK there's no maximum for NFS mounts, so in theory your software is broken if it can't handle 6.022e23 NFS mounts. That's a problem with people writing standards, virtually none of them ever specify a valid range for something. It's astounding how much security code you can crash or even get code exec on simply by sending data with a range that's valid according to the spec but unexpected by the implementation. For example vast amounts of PKI has a notation '(1...MAX)' where MAX means "there's some sort of maximum somewhere but we're not going to tell you what it should be", so when you define MAX = 2^64 - 1 things break. You can do the same with TLS extensions, IPsec options, ...

    The other problem is that if your code does actually enforce sanity checks then there's always someone who comes along to complain that your idea of what constitutes a sane upper bound doesn't match theirs and therefore your code is broken and you need to fix it.

    Starting Score:    1  point
    Moderation   +1  
       Interesting=1, Total=1
    Extra 'Interesting' Modifier   0  
    Karma-Bonus Modifier   +1  

    Total Score:   3  
  • (Score: 4, Insightful) by canopic jug on Monday May 11 2020, @12:06PM

    by canopic jug (3949) Subscriber Badge on Monday May 11 2020, @12:06PM (#992766) Journal

    There is a maximum that it can unmount concurrently in parallel. It is dependent on memory and other factors. After that limit, the rest get queued up and then processed only as resources become available.

    --
    Money is not free speech. Elections should not be auctions.
  • (Score: 2, Insightful) by Anonymous Coward on Monday May 11 2020, @09:39PM (1 child)

    by Anonymous Coward on Monday May 11 2020, @09:39PM (#993091)

    in theory your software is broken if it can't handle 6.022e23 NFS mounts

    Not in theory, in practice that is broken. It doesn't need to handle them successfully; error handling is handling. If it is going to crap out after 10, 100, 1e6, whatever mounts, then it should fail gracefully.

    GP's comment about bounds shows insight. I have worked in QA and the very first thing I would do if a step could be done 0..N times, would be to try 0, 1, 2, 32, 256, 257, 1025, 65537, N-1, N, N+1 times.

    In this case, the tests might've included "mount multiple NFS" but the test set definitely wasn't well-planned if it never exhausted resources. What if NFS had sekrit internal workings using an 8-bit-byte as NFS UID? What if NFS has static buffers that I can overflow? The test set needs a "cause resource exhaustion and confirm error behaviour is sane and not 'corrupt everything'" and then if it starts to *not* error gracefully (eg. due to increased structure size, dynamic lists instead of static arrays, whatever) then it should fail, then, so that the test set can be updated with whatever new values are required to confirm behaviour in erroring conditions.

    tl;dr: failing to test the scenarios when something SHOULD fail means the testing is not thorough, and likely there are bugs.

    • (Score: 2) by driverless on Tuesday May 12 2020, @01:24AM

      by driverless (4770) on Tuesday May 12 2020, @01:24AM (#993185)

      tl;dr: failing to test the scenarios when something SHOULD fail means the testing is not thorough, and likely there are bugs.

      Bugs? In systemd? Naaaah, you're pulling my leg...