System adminsitrator Chris Siebenmann has found Modern versions of systemd can cause an unmount storm during shutdowns:
One of my discoveries about Ubuntu 20.04 is that my test machine can trigger the kernel's out of memory killing during shutdown. My test virtual machine has 4 GB of RAM and 1 GB of swap, but it also has 347 NFS[*] mounts, and after some investigation, what appears to be happening is that in the 20.04 version of systemd (systemd 245 plus whatever changes Ubuntu has made), systemd now seems to try to run umount for all of those filesystems all at once (which also starts a umount.nfs process for each one). On 20.04, this is apparently enough to OOM[**] my test machine.
[...] Unfortunately, so far I haven't found a way to control this in systemd. There appears to be no way to set limits on how many unmounts systemd will try to do at once (or in general how many units it will try to stop at once, even if that requires running programs). Nor can we readily modify the mount units, because all of our NFS mounts are done through shell scripts by directly calling
mount; they don't exist in/etc/fstabor as actual.mountunits.
[*] NFS: Network File System
[**] OOM Out of memory.
We've been here before and there is certainly more where that came from.
Previously:
(2020) Linux Home Directory Management is About to Undergo Major Change
(2019) System Down: A systemd-journald Exploit
(2017) Savaged by Systemd
(2017) Linux systemd Gives Root Privileges to Invalid Usernames
(2016) Systemd Crashing Bug
(2015) tmux Coders Asked to Add Special Code for systemd
(2016) SystemD Mounts EFI pseudo-fs RW, Facilitates Permanently Bricking Laptops, Closes Bug Invalid
(2015) A Technical Critique of Systemd
(2014) Devuan Developers Can Be Reached Via vua@debianfork.org
(2014) Systemd-resolved Subject to Cache Poisoning
Related Stories
Whether you're running systemd happily or begrudgingly, it's best if you disable systemd-resolved as your DNS resolver for the time being. Reported today at seclists is a new DNS cache poisoning bug in systemd-resolved.
At its simplest, an attacker triggers a query to a domain he controls via SMTP or SSH-login. Upon receipt of the question, he can just add any answer he wants to have cached to the legit answer he provides for the query, e.g. providing two answer RR's: One for the question asked and one for a question that has never been asked - even if the DNS server is not authoritative for this domain.
Systemd-resolved accepts both answers and caches them. There are no reports as to the affected versions or how widespread the problem may be. Comments over at Hacker News suggests that it might not be widespread, most users would still be running the backported 208-stable while the DNS resolver was committed in 213 and considered fairly complete in 216, but that is if they enabled systemd-resolved in /etc/nsswitch.config.
Devuan.org announces:
Devuan is spelled in Italian and it is pronounced just like "DevOne" in English.
[...]is it really a fork?
This is just the start of a process, as bold as it sounds to call it a fork of Debian.
[...]Devuan aims to be a base distribution whose mission is protect the freedom of its community of users and developers. Its priority is to enable diversity, interoperability and backward compatibility for existing Debian users and downstream distributions willing to preserve Init freedom.
Devuan will derive its own installer and package repositories from Debian, modifying them where necessary, with the first goal of removing systemd, still inheriting the Debian development workflow while continuing it on a different path: free from bloat as a minimalist base distro should be. Our objective for the spring of 2015 is that users will be able to switch from Debian 7 to Devuan 1 smoothly, as if they would dist-upgrade to Jessie, and start using our package repositories.
Devuan will make an effort to rebuild an infrastructure similar to Debian, but will also take the opportunity to innovate some of its practices. Devuan developers look at this project as a fresh new start for a community of interested people and do not intend to enforce the vexation hierarchy and bureaucracy beyond real cases of emergency. We are well conscious this is possible for us mostly because of starting small again; we will do our best to not repeat the same mistakes and we welcome all Debian Developers willing to join us on this route.
Structural and semantic deficiencies in the systemd architecture for real-world service management
This is a in-depth architectural critique of systemd. It claims to be the first purely technical review of systemd internals, and provides a detailed analysis of several components. It criticizes on the basis of ordering related failures, a difficult to predict execution model, non-determinism in boot-order, as well as several other points.
Though many users would perceive the long processing pipeline to increase reliability and be more "correct" than the simpler case, there is little to acknowledge this. For one thing, none of jobs, transactions, unit semantics or systemd-style dependencies map to the Unix process model, but rather are necessary complications to address issues in systemd being structured as an encapsulating object system for resources and processes (as opposed to a more well-defined process supervisor) and one accommodating for massive parallelism. Reliability gains would be difficult to measure, and that more primal toolkits like those of the daemontools family have been used in large-scale deployments for years would serve as a counterexample needing overview.
A number of users have reported that running "rm --no-preserve-root -rf /" not only deletes all their files (as expected), but also permanently bricks their computers (which is not). Tracing the issue revealed that the ultimate cause was that SystemD mounted the EFI pseudo-fs as read-write even when this FS was not listed in fstab, and deleting certain files in this pseudo-fs causes certain buggy, but very common, firmware not to POST anymore. A user reported this bug on SystemD's GitHub issue tracker, asking that the FS be mounted read-only instead of read-write, and said bug was immediately closed as invalid. The comment thread for the bug was locked shortly after. Discuss.
Links:
https://github.com/systemd/systemd/issues/2402
http://thenextweb.com/insider/2016/02/01/running-a-single-delete-command-can-permanently-brick-laptops-from-inside-linux/
The spreading of systemd continues, now actively pushed by themselves unto other projects, like tmux:
"With systemd 230 we switched to a default in which user processes started as part of a login session are terminated when the session exists (KillUserProcesses=yes).
[...] Unfortunately this means starting tmux in the usual way is not effective, because it will be killed upon logout."
It seems methods already in use (daemon, nohup) are not good for them, so handling of processes after logout has to change at their request and as how they say. They don't even engange into a discussion about the general issue, but just pop up with the "solution". And what's the "reason" all this started rolling? dbus & GNOME coders can't do a clean logout so it must be handled for them.
Just a "concidence" systemd came to the rescue and every other project like screen or wget will require changes too, or new shims like a nohup will need to be coded just in case you want to use with a non changed program. Users can probably burn all the now obsolete UNIX books. The systemd configuration becomes more like a fake option, as if you don't use it you run into the poorly programmed apps for the time being, and if they ever get fixed, the new policy has been forced into more targets.
Seen at lobsters 1 & 2 where some BSD people look pissed at best. Red Hat, please, just fork and do you own thing, leaving the rest of us in peace. Debian et al, wake up before RH signed RPMs become a hard dependency.
Security researcher and MateSSL founder, Andrew Ayer has uncovered a bug which will either crash or make systemd unstable (depending on who you talk to) on pretty much every linux distro. David Strauss posted a highly critical response to Ayer. In true pedantic nerd-fight fashion there is a bit of back and forth between them over the "true" severity of the issue and what not.
Nerd fights aside, how you feel about this bug, will probably largely depend on how you feel about systemd in general.
The following command, when run as any user, will crash systemd:
NOTIFY_SOCKET=/run/systemd/notify systemd-notify ""
After running this command, PID 1 is hung in the
pausesystem call. You can no longer start and stop daemons. inetd-style services no longer accept connections. You cannot cleanly reboot the system. The system feels generally unstable (e.g. ssh and su hang for 30 seconds since systemd is now integrated with the login system). All of this can be caused by a command that's short enough to fit in a Tweet.Edit (2016-09-28 21:34): Some people can only reproduce if they wrap the command in a
while trueloop. Yay non-determinism!
Submitted via IRC for TheMightyBuzzard
A bug in Linux's systemd init system causes root permissions to be given to services associated with invalid usernames, and while this could pose a security risk, exploitation is not an easy task.
A developer who uses the online moniker "mapleray" last week discovered a problem related to systemd unit files, the configuration files used to describe resources and their behavior. Mapleray noticed that a systemd unit file containing an invalid username – one that starts with a digit (e.g. "0day") – will initiate the targeted process with root privileges instead of regular user privileges.
Systemd is designed not to allow usernames that start with a numeric character, but Red Hat, CentOS and other Linux distributions do allow such usernames.
"It's systemd's parsing of the User= parameter that determines the naming doesn't follow a set of conventions, and decides to fall back to its default value, root," explained developer Mattias Geniar.
While this sounds like it could be leveraged to obtain root privileges on any Linux installation using systemd, exploiting the bug in an attack is not an easy task. Geniar pointed out that the attacker needs root privileges in the first place to edit the systemd unit file and use it.
[...] Systemd developers have classified this issue as "not-a-bug" and they apparently don't plan on fixing it. Linux users are divided on the matter – some believe this is a vulnerability that could pose a serious security risk, while others agree that a fix is not necessary.
See, this is why we can't have nice init systems.
Source: http://www.securityweek.com/linux-systemd-gives-root-privileges-invalid-usernames
The other day, Michael W Lucas, who is normally known for good technical literature, put up a wild experiment of a short story: Savaged by Systemd. It's erotica, sort of. It's computer erotica, to be specific. It's Linux sysadmin erotica, to be more specific. OK, fine, it's systemd erotica. Really. Anyway, despite the subject and the genre, and in spite of the combination of the two, the e-book is trending and rising in quite a few lists.
Hopefully he can still remain focused on Absolute FreeBSD and be able to get that finished by the next BSDCan.
[Ed note: Has anybody actually bought and read this short story? I wasn't going to spend $2.99 to see what the hubub was about. - cmn32480]
From TFA (the friendly article) at https://www.openwall.com/lists/oss-security/2019/01/09/3:
We discovered three vulnerabilities in systemd-journald (https://en.wikipedia.org/wiki/Systemd):
- CVE-2018-16864 and CVE-2018-16865, two memory corruptions (attacker-controlled alloca()s);
- CVE-2018-16866, an information leak (an out-of-bounds read).
CVE-2018-16864 was introduced in April 2013 (systemd v203) and became exploitable in February 2016 (systemd v230). We developed a proof of concept for CVE-2018-16864 that gains eip control on i386.
CVE-2018-16865 was introduced in December 2011 (systemd v38) and became exploitable in April 2013 (systemd v201). CVE-2018-16866 was introduced in June 2015 (systemd v221) and was inadvertently fixed in August 2018.
We developed an exploit for CVE-2018-16865 and CVE-2018-16866 that obtains a local root shell in 10 minutes on i386 and 70 minutes on amd64, on average. We will publish our exploit in the near future.
To the best of our knowledge, all systemd-based Linux distributions are vulnerable, but SUSE Linux Enterprise 15, openSUSE Leap 15.0, and Fedora 28 and 29 are not exploitable because their user space is compiled with GCC's -fstack-clash-protection.
This confirms https://grsecurity.net/an_ancient_kernel_hole_is_not_closed.php: "It should be clear that kernel-only attempts to solve [the Stack Clash] will necessarily always be incomplete, as the real issue lies in the lack of stack probing."
The article goes on with more detailed information on exploits.
<sarcasm>It's a good thing that systemd does not affect very many systems and no systems running anything important.</sarcasm>
Good News:
Linux home directory management is about to undergo major change:
With systemd 245 comes systemd-homed. Along with that, Linux admins will have to change the way they manage users and users' home directories.
[...] Prior to systemd every system and resource was managed by its own tool, which was clumsy and inefficient. Now? Controlling and managing systems on Linux is incredibly easy.
But one of the creators, Leannart Poettering, has always considered systemd to be incomplete. With the upcoming release of systemd 245, Poettering will take his system one step closer to completion. That step is by way of homed.
[...] let's take a look at the /home directory. This is a crucial directory in the Linux filesystem hierarchy, as it contains all user data and configurations. For some admins, this directory is so important, it is often placed on a separate partition or drive than the operating system. By doing this, user data is safe, even if the operating system were to implode.
However, the way /home is handled within the operating system makes migrating the /home directory not nearly as easy as it should be. Why? With the current iteration of systemd, user information (such as ID, full name, home directory, and shell) is stored in /etc/passwd and the password associated with that user is stored in /etc/shadow. The /etc/passwd file can be viewed by anyone, whereas /etc/shadow can only be viewed by those with admin or sudo privileges.
(Score: -1, Offtopic) by Anonymous Coward on Sunday May 10 2020, @06:55PM (2 children)
Natty Portman on my stove, she is dancing, dancing, dancing.
I woke up and she was gone, but she left a small playing card with the joker on it and SYSTEMD ATE MY BALLS!
(Score: 2) by DannyB on Monday May 11 2020, @05:21PM (1 child)
Rather than "the systemd that ate the world", I would call it "DannyB's systemd incompleteness theorem".
Basically stated: systemd can never be complete until it has assimilated the entire observable universe.
Therefore, even after systemd assimilates both the kernel and the bootloader, something still must be done about the UEFI firmware. All motherboards MUST have a systemd bootloader firmware. Ultimately all microprocessors and peripheral devices must have systemd.
The lower I set my standards the more accomplishments I have.
(Score: 0) by Anonymous Coward on Monday May 11 2020, @05:58PM
systemd-borg
(Score: 3, Insightful) by Anonymous Coward on Sunday May 10 2020, @06:58PM (23 children)
I'm shocked, just shocked. And you know what Poettering's reaction is going to be: "Fuck you, Chris Sibenmann! Will not fix."
(Score: 3, Touché) by DimestoreProstitute on Monday May 11 2020, @12:34AM (22 children)
Nah he'll just say NFS is archaic and mounts should be moved to his own protocol.
(Score: 5, Touché) by qzm on Monday May 11 2020, @01:09AM (20 children)
Exactly! He has never had this particular problem on HIS LAPTOP, and all SystemD work is based on what Pottering thinks would make HIS LAPTOP work better this week.
Should he ever hit this problem, it would of course indicate a problem with all existing other filesystems, and require everyone to switch to Systemd-FS, which would be modeled on how he things NTFS works.
(Score: 1, Funny) by Anonymous Coward on Monday May 11 2020, @03:10AM
SystemdFS?
Stop giving them ideas. Please.
(Score: 2) by driverless on Monday May 11 2020, @05:18AM (18 children)
Not wanting to defend systemd here (who has that much asbestos?) but this is a pretty weird situation, it's not that systemd is fatally broken... well OK, we can debate that elsewhere, but that presumably no-one working on it ever expected to run into a situation with three hundred and forty seven NFS mounts. I mean WTF is this guy doing, NFS mounting the planet? In any case now the systemd people know that something as outlandish as this is actually possible, they can do something to fix it.
(Score: 4, Insightful) by canopic jug on Monday May 11 2020, @08:28AM (5 children)
When writing programs, test ranges of input and handle extremes and exceptions. If you have an input that allows an minimum through a maximum, it should have an appropriate reponse when getting something out of range. That used to be common knowledge prior to Bill personally destroying comp sci across the country. Poettering seems to be of this new generation brought up under Bill's values and lack of knowledge. An out of memory event is not an appropriate response, the extra unmount requests need to be queued up instead and handled as memory becomes available.
Money is not free speech. Elections should not be auctions.
(Score: 3, Interesting) by driverless on Monday May 11 2020, @11:55AM (3 children)
AFAIK there's no maximum for NFS mounts, so in theory your software is broken if it can't handle 6.022e23 NFS mounts. That's a problem with people writing standards, virtually none of them ever specify a valid range for something. It's astounding how much security code you can crash or even get code exec on simply by sending data with a range that's valid according to the spec but unexpected by the implementation. For example vast amounts of PKI has a notation '(1...MAX)' where MAX means "there's some sort of maximum somewhere but we're not going to tell you what it should be", so when you define MAX = 2^64 - 1 things break. You can do the same with TLS extensions, IPsec options, ...
The other problem is that if your code does actually enforce sanity checks then there's always someone who comes along to complain that your idea of what constitutes a sane upper bound doesn't match theirs and therefore your code is broken and you need to fix it.
(Score: 4, Insightful) by canopic jug on Monday May 11 2020, @12:06PM
There is a maximum that it can unmount concurrently in parallel. It is dependent on memory and other factors. After that limit, the rest get queued up and then processed only as resources become available.
Money is not free speech. Elections should not be auctions.
(Score: 2, Insightful) by Anonymous Coward on Monday May 11 2020, @09:39PM (1 child)
Not in theory, in practice that is broken. It doesn't need to handle them successfully; error handling is handling. If it is going to crap out after 10, 100, 1e6, whatever mounts, then it should fail gracefully.
GP's comment about bounds shows insight. I have worked in QA and the very first thing I would do if a step could be done 0..N times, would be to try 0, 1, 2, 32, 256, 257, 1025, 65537, N-1, N, N+1 times.
In this case, the tests might've included "mount multiple NFS" but the test set definitely wasn't well-planned if it never exhausted resources. What if NFS had sekrit internal workings using an 8-bit-byte as NFS UID? What if NFS has static buffers that I can overflow? The test set needs a "cause resource exhaustion and confirm error behaviour is sane and not 'corrupt everything'" and then if it starts to *not* error gracefully (eg. due to increased structure size, dynamic lists instead of static arrays, whatever) then it should fail, then, so that the test set can be updated with whatever new values are required to confirm behaviour in erroring conditions.
tl;dr: failing to test the scenarios when something SHOULD fail means the testing is not thorough, and likely there are bugs.
(Score: 2) by driverless on Tuesday May 12 2020, @01:24AM
Bugs? In systemd? Naaaah, you're pulling my leg...
(Score: 0) by Anonymous Coward on Wednesday May 13 2020, @02:12PM
Are both unchecked for conditions in Firefox since just after they ousted the original developer of firefox as the project lead (not sure if he is still with Mozilla, I just remember they placed their own PHB as manager over the project, which then moved to XUL, causing memory bloat and performance regressions until the JIT version of their javascript engine got released.)
I forget what other applications but a *LOT* of applications on linux have no handling for either out of memory or out of disk errors, including for example java running on 32bit (just make it 64 bit and the problem goes away, because nobody will ever hit memory pressure on a 64 bit system, lol! Same was said during the 16->32 bit transition btw.)
(Score: 3, Interesting) by Thexalon on Monday May 11 2020, @12:42PM (9 children)
Good design accounts for weirdness, extremes, and unexpected user behavior.
For instance, any properly designed piece of software must ensure that if the process is interrupted (e.g. someone just pulled the plug from the computer), it can recover with minimal damage. That means that you have to leave files on disk in a state where they can function or be recovered at all times, and well-written software does just that.
This sounds like a case where Lennart decided to ignore the simple fact that every single call to malloc() can come back with a null pointer, and if you do that you create segfaults. It also sounds like he ignored the fact that if somebody can do a thing more than once, they can do them infinite numbers of times.
The best QA guy I ever worked with was absolutely fantastic at finding ways to interact with whatever I had built that nobody except sometimes me had planned for. I'd send him the use cases, he'd poke around and find a bunch more and see whether I'd considered what happened when users did unusual things (e.g. interact with your web-based thing without Javascript turned on). If Lennart were some kid in his parents' basement working on a hobby project I'd consider skipping that step to be excusable, but he's a professional programmer leading a critical project for a billion-dollar company who can definitely afford to have QA and should have somebody involved who thinks of this kind of thing.
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 2) by driverless on Monday May 11 2020, @01:00PM (8 children)
Good hindsight accounts for weirdness, extremes, and unexpected user behavior. The problem with weirdness, extremes, and unexpected user behavior is that it's weird and unexpected until the first time you encounter it, after which you fix your software to handle it the next time round. If it was expected behaviour then your software would be able to handle it already.
(Score: 2) by Thexalon on Monday May 11 2020, @01:59PM (7 children)
You seem to have missed the point of my post.
Good design assumes that anything that can happen will happen. Really really good design also assumes that anything that can't happen will happen too. There are some limits I'd expect to have to the ability to adapt, e.g. I don't expect software to be able to keep functioning if the server room is on fire, but handling edge cases gracefully is a hallmark of good software, and I expect software that is at the core of millions of systems to be held to a very high quality standard.
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 2) by driverless on Monday May 11 2020, @02:08PM (6 children)
I didn't miss the point, you're saying that theoretically perfect software should be able to handle all possible error and unexpected conditions. I'm saying that such software doesn't exist. The closest we've got is formally verified software, which averages to, from memory, something like $1,000 per line of code to develop. And it still sometimes fails to handle every possible condition.
(Score: 3, Informative) by Thexalon on Monday May 11 2020, @02:26PM
The Linux kernel doesn't have these kinds of problems on a regular basis. And a major reason for this is that Linus is notoriously ruthless about demanding contributors think about and address those weird conditions and edge cases before he'll even consider merging their code.
I'd expect systemd, with its aspirations to be at least as critical to Linux systems as the kernel is, to be held to similar standards. It's not, and the fact that it's not is a problem.
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 2) by rleigh on Monday May 11 2020, @05:52PM (3 children)
In the field I work in, every system requirement has to have an associated FMEA (failure modes effects analysis), which includes all of the hardware and software mitigations to take. It's tedious, but it ensures that all of the common and not-so-common failure modes have been thoroughly explored by a whole team of people, and that the appropriate mitigations have been implemented where appropriate.
Do you think the systemd developers have done this, or anything remotely like this? No, neither do I. They don't care about stuff like that.
And yet... having deliberately placed themselves in the most safety-critical part of the system, that's exactly what they should be doing.
Whenever you parallelise something, you've got to have an upper bound on the parallelisation. Often, that's a maximum bound, and you might want to lower it if the system can't cope. Look at how, e.g. ZFS balances I/O. It's continually monitoring the available bandwidth to each device and adjusting the I/O load on them to maximise throughput. It also cares about responsiveness. If you start e.g. a scrub, it will consume all available disk bandwidth, but it has a very clever algorithm which slowly ramps up the utilisation over several minutes, and it has a fast backoff if any other I/O requests come in. There's no reason that systemd couldn't be doing this on unmount. It doesn't need to parallelise everything, it can start slow, monitor the time each umount takes, the completion rate and system resources used, and it can back right off if things start stalling. But as mentioned elsewhere in the thread, this is a place where parallelisation is almost pointless. Sometimes the simple solution is the best solution. You can safely and reliably unmount everything in a one line shell script, so why can't systemd do something that simple?
(Score: 0) by Anonymous Coward on Monday May 11 2020, @08:51PM
They definitely didn't do anything near that. They have less than 50% function coverage and less than 38% statement coverage. I can't even imaging them trying to get anywhere near the truly required edge, branch, and condition coverage. Give me some MCDC, fuzzing, and property testing. But no, don't do any of that but then act surprised when bugs people predicted and warned you about crop up.
(Score: 0) by Anonymous Coward on Monday May 11 2020, @09:02PM
Shell script is anathema for systemd. Any admin can debug that with a plain editor, or even test interactively by copy & paste, then edit lines in shell history (basic REPL). Not in the systemd vision.
We already concluded that IBM (and RH before them) plan is to add complexity so only them can touch the steering wheel, and all the rest have to buy support contracts and training, and "enjoy" the ride. And as you said in the past, other projects bent over dreaming they will still matter in the future and everything will be roses by then, when Rome does not pay traitors (that is the part I add).
Free Software licenses like GPL require distribution of "source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities". Corporations found the loophole, the license does not say anything about making such big mess that people can make use of their freedoms. The mess is so big in some projects, that even small companies can not do it. Even less so, fork a project to try to get it back into sanity.
(Score: 2) by driverless on Tuesday May 12 2020, @01:29AM
Your mention of FMEA is exactly the point I was making about software that ends up costing $1,000 per line of code in it. I've worked on SIL 2 and 3 systems and the amount of effort required for even a simple system is insane, no standard commercial or open-source system could be developed that way. Sure, it's then highly reliable, but only if you're prepared to invest the massive amounts of time and money into doing it that way.
Anyway, as I mentioned earlier, not trying to defend systemd, but pointing out that just because it's in theory possible to build something to (say) SIL 3 doesn't mean it's practical for most software.
(Score: 0) by Anonymous Coward on Monday May 11 2020, @09:45PM
This issue here isn't some weird logic bug. This issue is a simple "cannot make call X more than N times."
That is not "all possible error and unexpected conditions" as you call it, that is "they failed to test atomic fault handling for a well known and commonly encountered fault" - making sure you check for nulls out of malloc() is 1990s bugs - and it is "their testing wasn't written to find and expose issues" because as Thex and I know very well, this is well within the kind of things that a good QA person tries.
(Score: 0) by Anonymous Coward on Monday May 11 2020, @08:36PM
> they can do something to fix it
"Works fine for me, won't fix" ~ Lennart Poettering, every damn time
(Score: 3, Interesting) by DeVilla on Tuesday May 12 2020, @06:52AM
If systemd had umount()ed serially or called `umount -a`, this would not have happened.
Several people are arguing things about whether you should handle weird errors or not. That misses the point. Systemd took over the process of unmounting files systems and decided to do so in parallel. If you are doing parallel processing you monitor your resources. Systemd in not doing this.
Systemd's allowing users to unknowingly to define a fork bomb without checking if the resource needed are available nor providing the user a mechanism to check or prevent it. The fact that it's a umount() is beside the point. If I have a large system (assumes the system has resource to run everything in steady state) can I trigger a fork bomb induced OMM by starting tons of small, independent services via unit files? If systemd is doing it's job correctly, then the answer should be no.
(Score: 2) by DeVilla on Tuesday May 12 2020, @06:30AM
No. Manually mounting files systems via direct calls to `mount` in script is not supported. It's horribly broken, doesn't work and never has worked. The filesystems should be mounted via unit files which has a declarative syntax that is clearly superior. (The ancient fstab sort of works too, but that's really for only backwards compatibility and should be avoided as well.)
(Score: 2) by gtomorrow on Sunday May 10 2020, @07:02PM (11 children)
<sarcasm>It's a feature!</sarcasm>
It's a good thing that my 18.4.4 LTS machine hasn't been
busting my chopsadvising me to upgrade....not like I'm running 347 NFS mounts, but you never know. Or is this Poettering's version of "640K is enough for anyone"?(Score: 2) by NCommander on Sunday May 10 2020, @07:31PM (10 children)
Canonical doesn't start pushing updates until the .1 release of an LTS which is a good thing because sometimes a few nasty bugs get through the RC process.
Still always moving
(Score: 3, Interesting) by bzipitidoo on Sunday May 10 2020, @07:41PM (9 children)
I'll say. I've been running Lubuntu 20.04 on an old machine with only (only, hah!) 3G RAM. I strongly advise staying with 18.04. 20.04 just isn't polished enough. One of the worst bugs, which they mention in the release notes, is that LibreOffice Export to PDF does not work. That's just plain unacceptable for office work. I've also run into a weird problem with printing. CUPS complains that the cert is expired, and refuses to print. Huh? My printer even has a cert? WTF are they talking about? Printing and Export to PDF both are fine in 18.04.
Lubuntu is supposed to be lightweight, but it rips through that 3G RAM like a starving wolf gulping down a mini sausage. Best keep it to one app at a time. No running Firefox and LibreOffice at the same time.
(Score: 2) by NCommander on Sunday May 10 2020, @07:49PM
Honestly, 18.04 wasn't a great release either. I'm running it on my laptop, and while most of the bugs have been SRUed out, I have run into oddities with the included systemd and journalctl. Most of the issues appeared to come from Debian and I found 7/8 both to be very unstable compared to what came before. I've been fairly dissolution with Linux distributions for the last several years, and my only solace is that Windows continues its steady march in becoming crapper with each Windows 10 update.
Still always moving
(Score: 2) by Snotnose on Monday May 11 2020, @02:12AM (6 children)
Is it Lubuntu, or systemd that is breaking LibreOffice. Cuz if systemd can break a LibreOffice Export to PDF, thats kind of A Big Deal (tm).
Seriously, systemd is a boot thing. Once you're up and running systemd should be moot. LibreOffice is a user mode thing. Having a boot thing break a user mode thing just, well, glad I retired about the time systemd poked it's head up.
FWIW, I got 100% anti-Pottering when he did something in user space that caused problems with the kernel (his audio thing with too many debug messages or something). When he said "not my problem, your problem", the name "Pottering" stuck in my head as avoid at all costs.
When the dust settled America realized it was saved by a porn star.
(Score: 3, Interesting) by bzipitidoo on Monday May 11 2020, @03:22AM (4 children)
No, it's Ubuntu 20.04. Nothing to do with systemd-- at least, I hope it can't possibly be related to systemd. The problem is that they didn't settle on fonts for LibreOffice's PDF generation. So, LibreOffice will produce a PDF and act like it was successful, but when you view the PDF, you'll see the pictures and diagrams if any, and no text.
I find that LibreOffice's handling of PDFs is not too good. A plain old PDF viewer will display a PDF correctly, with the correct fonts. But LibreOffice might not. It won't use the fonts in the PDF, instead looking for those fonts in the system and if it can't find them, doing a font substitution. The fonts might be right there in the PDF, but LibreOffice won't use them. Too often, the substitution throws all the positioning off.
Yes, I know, PDFs aren't meant to be edited. That's not the reality in the office. Businesses act like giving out a source document (usually a Word file) is giving away their secrets or their control, but that very same info in a PDF is okay. So a PDF might be the best you have to work with.
As for Poettering, the impression I've formed is that he's sloppy. Pulseaudio is a terrible resource hog. Might take 200M of RAM, just so audio from different apps can be blended. If you don't have a lot of RAM, devoting 200M of it to such a trivial thing is a problem. I can't imagine why it needs so much memory, and guess it shouldn't but that it's full of bugs, specifically, memory leaks. Maybe he doesn't care about that, as long as it does the job. To be fair, it might not be pulseaudio, it could be something else leaking like mad, perhaps Firefox. However, top reported pulseaudio as the memory hog. I simply uninstall pulseaudio, and let apps fight for ALSA. If you are also only running one app at a time, then there's no contention.
(Score: 2) by janrinok on Monday May 11 2020, @07:28AM
I'm running Ubuntu 20.04 MATE and I'm not seeing this fault - I don't know if the fonts are any different between MATE and Lubuntu?
One installation is a clean install from a 20.04 iso, and two others are updates from 19.10.
(Score: 0) by Anonymous Coward on Monday May 11 2020, @09:21AM (1 child)
ALSA's dmix is quite capable of allowing multiple applications to play audio at the same time, and is, or used to be, normally configured this way by default. Perhaps this has changed in Ubuntu now that they assume you'll be using PulseAudio exclusively, especially since the default PulseAudio configuration opens the ALSA output device directly, denying audio output to non-PulseAudio programs anyway. (This can be changed in PulseAudio's config.)
If you're not interested in per-application volume controls in the systray, networked audio or (so I've heard) Bluetooth audio, using plain ALSA without PulseAudio still works absolutely fine. (Except for Firefox, which seems to have dropped ALSA support, so you have to use a wrapper like apulse[1], or use a build that has ALSA support re-added. I use Void Linux, which comes with a build of Firefox that works just fine with ALSA "out of the box".)
[1] https://github.com/i-rinat/apulse [github.com]
(Score: 2) by bzipitidoo on Monday May 11 2020, @01:28PM
I daresay you are correct, and my somewhat hazy idea of what exactly PulseAudio does is a little off.
More generally, what has happened to our systems that 0.0625G RAM was once considered generous, and now, 4G is minimally adequate? Okay, yes, moving from ASCII to UTF-8 dramatically increased the memory needed for fonts, and library routines for handling text strings. And we are running much bigger screens-- 1280x1024 is kinda small by today's standards, and no one settles for just 256 colors any more. Wayland should be trimmer than XWindows, but dumping the 1980s graphics cruft that is now handled by monster graphics hardware doesn't seem to have gained back much. Moving from 32bit to 64bit was another memory eater. There are slimmer alternatives to glibc, such as uclibc and musl libc. And systemd, wouldn't surprise me if that has the largest footprint of all the init systems. Firefox too has grown, with ever more standards to follow. Really must have the Opus audio codec (added in Firefox 19), and now there's AV1 video in the works. Perhaps JPEG will at last be upgraded, to JPEG-XL. Somehow, now we can no longer completely control whether our browsers automatically play video, but back in the day there was a browser option to disable automatic downloading of mere images, to save bandwidth and all.
(Score: 0) by Anonymous Coward on Monday May 11 2020, @09:48PM
PDFs are meant to be edited. They're based on PS which was also meant to be edited.
(Score: 0) by Anonymous Coward on Monday May 11 2020, @03:51AM
Heretic! Stone him! Draw and quarter 'im! Crucify him! Burn his remains! Subject his followers and family to use Windows forever!
(Score: 0) by Anonymous Coward on Monday May 11 2020, @10:33PM
I don't use Ubuntu, but since it is based on Debian this might be available.
Just install zram-tools, and reboot. The default config will use 20% of your ram as compressed swap before hitting your regular swap on disk (you still need swap on disk too). On low ram machines (512MB VPS for me), it is a large improvement. Increasing swappiness to 100 will also help, so there is some free ram for buffers when needed.
tune settings in:
/etc/sysfs.d/zswap.conf
I even run it on machines with decent ram (but with %ram reduced) to reduce erase cycles to the ssd. And, on my netbook without aes-ni instruction, the compression / decompression is wayyyy faster than encryption/decryption to disk.
(Score: 5, Insightful) by Anonymous Coward on Sunday May 10 2020, @07:08PM
It just... sucks.
(Score: 5, Interesting) by NCommander on Sunday May 10 2020, @07:09PM (20 children)
Seriously, I'm honestly surprised no one has forked it to try and have a less crazy upstream. Then again, given that systemd and GNOME are more or less married at the hip, it's probably a lost cause. Part of why SN started migrating to Gentoo was because of the constant pain systemd brings and general opinions that Ubuntu QA had been suffering to the point it was time to abandon ship.
PulseAudio didn't stop having constant issues until upstream changed and actual bugs were fixed.
Still always moving
(Score: 5, Interesting) by canopic jug on Sunday May 10 2020, @07:22PM (14 children)
It's the concept of systemd which is flawed. Reimplementing the flawed design in a different code base will just bring the same flaws to a new sdet of coders, wasting everyone's time even more. If you're going to fork, it's going to have to be at the distro level. Devuan [devuan.org] is one fork and an example of how much work it has become. It is an excessive amount of work these days because systemd is pushing into the package as well. So various unrleated packages end up with systemd dependencies and entanglement for lots of excuses but no real technical reasons or benefits.
I figure it is that Red Hat, now IBM, has figured out how to carry out the strategies against GNU/Linux and Software Freedom outlined by M$ in the Halloween Documents [catb.org] around 22 years ago. Until systemd came along, distros were more or less interchangeable commodities. Now everything is tied more and more into RHEL. The code has gotten too complex to dig into, so just pay Red Hat for changes. The resulting software is even getting too complex to learn, so just pay Red Hat for training. The changes get more and more convoluted, so just pay Red Hat for support. And so on.
Money is not free speech. Elections should not be auctions.
(Score: 3, Informative) by NCommander on Sunday May 10 2020, @07:30PM (13 children)
upstart was the same basic idea (as an init/mount manager) that was considerably less of a problem. I do count mounting disks to be an init issue because it was an endless source of bug reports especially in cases where NFS was in play (SysV automount can go die in a fire). We had issues with it here on SN when we briefly tried NFS for site deployment; it was an utter mess (we used gluster, before just deciding to rsync the entire /srv/soylentnews.org folder which was a brute force approach)
If systemd was content to stay as an init process, I'd live with it. The fact that provides relatively little benefit and major pain has soured me on it. At least Ubuntu still keeps the text logs around in addition to journalctl.
Still always moving
(Score: 5, Insightful) by canopic jug on Sunday May 10 2020, @07:46PM (12 children)
I agree about Upstart being a better choice, but it is not an apt comparison. Upstart [darknedgy.net] was an init system. systemd is not an init system [darknedgy.net], though I can see where the confusion comes from. Poettering and his fanbois had been and continue be misrepresenting systemd as an init system.
In short, they lie about it. That's no surprise as the two other methods of promoting it have been appeals to novelty and personal attacks. The real puzzle is the Debian Technical Committee vote. The tie-breaking vote picked systemd even though it was fourth place and three more popular options were ahead of it. Then the whole project just folder and acquiesced without even any grumbling and, with that, hundreds of derivative distros fell to systemd. Then Poettering and his fanbois could add the false claim that systemd was popular, rather than a more accurate claim of being widely deployed because the downstream distros had little to no choice. Devuan and the few other free distros do a lot of clean up, that is a lot of effort not spent in other ways making the distros and packages better. Of those, I was most disappointed in the fact that there was neither pushback from Canonical over this move from Red Hat nor did they move their base from Debian to Devuan as they could have done. However, by that time Canonical already had "former" microsofters on staff. probably eager to throw systemd sand in the gears.
Money is not free speech. Elections should not be auctions.
(Score: 5, Informative) by NCommander on Sunday May 10 2020, @07:53PM (3 children)
The Technical Committee vote is a serious sore point for me, but I can't go into details as most of it was on debian-private. I can't blame Canonical for switching to systemd when Debian did due to the burden it would have caused, but I put more of the blame on Red Hat who have a long history of shoving shit down the communities throat and forcing people to accept it.
Red Hat was the reason we had to put up with Ulrich Drepper and the utter nightmare of glibc development until he was forked around with eglibc.
Disclaimer: I was employed by Canonical at the time, and was a Debian Developer.
Still always moving
(Score: 0) by Anonymous Coward on Monday May 11 2020, @12:52AM
Do you recommend an inquiry?
(Score: 2, Interesting) by Anonymous Coward on Monday May 11 2020, @09:16PM
If Debian forgot one of the main points to exist was caring about their users, maybe it is time for some whistleblowing. Not you, just dropping the idea.
Or the more polite enquire suggested above. But I guess transparency and accountability are just words in the dictionary, not to be applied in real world.
(Score: 1, Informative) by Anonymous Coward on Monday May 11 2020, @09:52PM
Thanks for the veiled insight. Consolations.
(Score: 5, Interesting) by Anonymous Coward on Sunday May 10 2020, @09:56PM (2 children)
On the Arch Linux forums when systemd was first brought up, the "official" maintainers (whatever that means) stated that they didn't wanna support multiple init systems so they'd be moving to it.
A handful of people volunteered to support the existing init system in parallel. They were directed to a separate subforum then banned.
That was my first warning sign that this systemd thing might be something I'd want to avoid.
(Score: 0) by Anonymous Coward on Thursday May 14 2020, @12:42AM
I was there when that happened, I can confirm. And it was what drove me away from Arch, even though I was very happy with it at the time. I'm glad I found Void, it makes for a great systemd-free alternative to Arch. A little drama these past few weeks but nothing the maintainers can't work around.
(Score: 0) by Anonymous Coward on Tuesday May 26 2020, @11:59AM
Seems similar to Poettering engineered the transition from consolekit to logind.
Get himself assigned maintainership, hand Gnome a patch set to support logind on a silver platter, then send out a terse email declaring consolekit deprecated and redirect all replies to /dev/null.
Even better is that systemd-logind overriding nohup came about because Gnome failed to clean up its session on logout.
Oh, and su is apparently a big nono with systemd. Instead you should use ssh to 127.0.0.1 if you need a root prompt.
I swear the guy has Apple envy, yet keep recreating Windows piece by piece. Do wonder how long before he joins Icaza at Redmond. Probably not long after Microsoft declares its own Linux distro with a Win32 compatibility layer.
(Score: 2) by https on Sunday May 10 2020, @10:24PM
When you suggest that
...clearly, you were not subscribed to debian-devel.
Never heard such a load of whining until it was shown that Trump is masturbating to burning copies of the constitution.
Offended and laughing about it.
(Score: 0) by Anonymous Coward on Monday May 11 2020, @06:24AM
I haven't been able to put into words my main problem with systemd or modern declarative DSLs. It was nice to finally put my finger on both in the same time.
(Score: 2) by Thexalon on Monday May 11 2020, @12:45PM (2 children)
The pattern of systemd adoption has always had me wondering whether its supporters had non-technical motivation for using it, i.e. bribes or threats.
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 0) by Anonymous Coward on Tuesday May 12 2020, @02:25AM
Based on what I've seen first hand, heard about second hand, and read reliable accounts of, in regards to microsoft products and specifications, I'd say that it is a matter of a few well-placed, credible threats now and again. systemd is not from microsoft but it is about high stakes control over global communication. If groups play hardball like that over the desktops, they're going to be even more wicked for a chance at control over the larger market consisting of everything else.
(Score: 0) by Anonymous Coward on Tuesday May 26 2020, @12:04PM
Threats basically, and ever so oftne getting their guy to take over maintainership before running the project into the ground (Poettering basically did that with consolekit before rolling out systemd-logind).
There is a story floating around about docker devs and RH devs having a spat, where the Docker devs told RH to pound sand regarding some changes they wanted. Soon after RH started talking up systemd-nspawn as a Docker replacement.
It is the same kind of behavior one has seen from both Apple and Microsoft over the years, where they would either buy out or roll out a competitor for a successful piece of software on their respective platforms.
(Score: 4, Informative) by RamiK on Sunday May 10 2020, @08:11PM (3 children)
If you're capable of doing cost-analysis for what it takes to develop and maintain a C++ project with the size and complexity of systemd, you're not going to bother developing it as-is or fork it in the first place. And by not going to bother, I mean you'll soon realize there's better and cheaper ways to get the same features without limiting yourself to a handful of highly skilled and limited-in-availability developers.
I mean, Embedded doesn't need it... Gentoo and kin get most of it with fairly simple scripts... Guix's Shepherd is getting all of it and more with LISP scripts... So, why bother?
compiling...
(Score: 2) by NCommander on Sunday May 10 2020, @08:29PM (2 children)
I've seen systemd on "embedded", although that's more people trying to stretch the definition.
Still always moving
(Score: 2) by RamiK on Monday May 11 2020, @01:06AM
Rather fitting for the age of network-connected kitchen appliances and toothbrushes. :/
compiling...
(Score: 2) by PocketSizeSUn on Monday May 11 2020, @08:47PM
Ex: AFAIK all the nividia embedded development base is Ubuntu. For example Tesla uses NVIDIA chips and therefore Ubuntu because that's the kernel with the blobs and swimming upstream in embedded is a real PITA that rarely rewards quickly enough for management to agree to it.
(Score: 0) by Anonymous Coward on Monday May 11 2020, @03:54AM
It's all fun and games until you have to use it in Live
(Score: 4, Insightful) by Subsentient on Sunday May 10 2020, @07:31PM (16 children)
The longtimers here know I'm no fan of systemd. Hell, I hate it so much I wrote my own init system [universe2.us].
But 347 NFS mounts... Jesus Christ. What the hell are you doing that you need that many filesystems? The output of "mount" must be a real eyerape.
Systemd needs a bit of logic to try to unmount sequentially over a threshold, but, I don't blame systemd too much for this, more like a bizarre use case. What the hell?
"It is no measure of health to be well adjusted to a profoundly sick society." -Jiddu Krishnamurti
(Score: 4, Insightful) by Anonymous Coward on Sunday May 10 2020, @08:00PM (3 children)
You are right that the story submitter had a crazy amount of NFS mounts.
However, I think you are wrong to excuse systemd from not handling this correctly. As an init system, its job is to handle daunting startup and shutdown scenarios. Systemd is supposed to be more robust than the average software we all bumble along with. It seems this attitude towards extreme reliability just doesn't exist with Poettering who has no one else to answer to. Linux isn't more reliable than Windows these days -- at least Windows Server. I remember when Linux got the basics right, but that was when Linux was still reimplementing a stable standard (Unix, POSIX) created by more competent people. Most Linux "innovations" are frankly not that good.
(Score: 0) by Anonymous Coward on Monday May 11 2020, @03:10PM (2 children)
Systemd can't handle situations that they never had a test case for. Why would they bother prioritizing this if a user never reported any problems with unmounting 347 NFS drives before now?
(Score: 2) by rleigh on Monday May 11 2020, @06:08PM (1 child)
Competent software developers ensure that such situations can't arise in the first place. As well as competent developers, you also need a good specification and a good design, which also take these situations into account, as well as unit and integration testing that tests the extremes. It should all be documented, up front. Do you think the systemd developers did this, or did they just bash out some code and hope for the best? The most critical part of the system should not be written by dangerously incompetent arseholes.
Testing parallelism is hard, but several of the projects I work on have threading tests which are designed to provoke edge cases with locking and resource limitations, it can be done.
This isn't just the systemd developers. I could previously cause a kernel panic with LVM just by creating and deleting a few snapshots in parallel. Some kernel locking defect I assume. Back when I was testing Debian by rebuilding the whole archive on a 24 CPU system, I thought I'd test out ~30 parallel builds, and it repeatedly panicked the system within 5 minutes every time. No idea if it's fixed, I gave up on it at this stage, but that usage had clearly not been tested at all.
(Score: 0) by Anonymous Coward on Wednesday May 13 2020, @02:42PM
>Competent software developers
This should be corrected to say "Software developers with an excessive amount of time and fungible resources on their hands". It's easy to ask for these things, a lot harder to swallow when you get stuck with the bill.
(Score: 1, Interesting) by Anonymous Coward on Sunday May 10 2020, @08:30PM (3 children)
As an ex-code-plodder-along who dealt more with databases than kernels, but now increasingly just an ageing, helpless passenger on the Linux train... what would it take to have a spin of Mint or Manjaro (forget Debian & Arch) that have your alternative init system? Or how would one go about replacing systemd with epoch on an existing system? Mint, years back used upstart, but then upstream debian and ubuntu switched to systemd and it was a choiceless done deal.
(Score: 2) by Reziac on Monday May 11 2020, @03:14AM
PCLinuxOS does not use systemd.
From here in userland I don't care one way or the other so long as the OS is stable, but this is my preferred distro, so... there it is. Or rather, isn't.
And there is no Alkibiades to come back and save us from ourselves.
(Score: 2) by janrinok on Monday May 11 2020, @08:10AM (1 child)
Ubuntu still let you install SysV-init on versions up to and including 19.10 [ubuntu.com], but I'm not going to re-install one to check which packages it used. I've just checked 20.04 and sysv-init-utils is still there, but there is also systemd-sysv there too which suggests it isn't the sysv-init that we all know. As I've never tried switching to it I do not know how well (or otherwise) it works.
A quick online search suggests that some have reverted to sysv-init successfully but not in huge numbers.
(Score: 2) by janrinok on Monday May 11 2020, @08:14AM
(Score: 5, Interesting) by bloodnok on Sunday May 10 2020, @08:47PM
This may be an unusual use-case and well outside of the comfort-zone of most of us, but it is not unreasonable. And, in any case, it is not the job of the OS to determine what is unreasonable. Its job is to deal with whatever is thrown at it with as much grace as possible. Spawning tasks without apparent limit is entirely graceless.
This identifies a pretty serious deficiency in the systemd implementation/design/philosophy. Any system that does things in parallel *must* provide a way to limit the number of tasks that are executed at once. That is a fundamental requirement. The limit to parallelism should not be the imagination, or cussedness, of the system administrator but something that systemd itself provides. I would hope that this is taken seriously by the systemd team, but as I am trying to rid myself of systemd I find myself caring only i na very abstract sense.
Oh and If you're not convinced by the need for limits, trying building the kernel with -j 200.
(Score: 4, Insightful) by Bot on Monday May 11 2020, @12:45AM (4 children)
indeed this is a corner case.
but Unix is the OS for corner cases, and this incident proves right those talking about the Unix philosophy and those (see my comment history) judging a DSL for booting as a mistake.
yes the solution is to complicate either the DSL or its umount routines. inconveniencing everybody for the need of the guy mounting 45739 nfs volumes.
so basically, what would be a user in Unix is an outlier in systemd land
and I bet switching to a systemdless Deb distro is faster than addressing the bug.
Account abandoned.
(Score: 2) by Thexalon on Monday May 11 2020, @12:49PM (3 children)
Pardon my ignorance, but WTF does "DSL" mean in this context?
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 2) by Bot on Monday May 11 2020, @03:51PM (1 child)
Sorry, I was referring to systemd units configuration needing to replace turing complete and infinitely interfaceable bash scripts, so dsl means domain specific language, good summary in https://yakking.branchable.com/posts/domain-specific-languages/ [branchable.com]
Account abandoned.
(Score: 2) by Thexalon on Monday May 11 2020, @05:03PM
Ah yes, makes perfect sense now.
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 0) by Anonymous Coward on Monday May 11 2020, @07:00PM
Diesel
(Score: 3, Interesting) by boltronics on Monday May 11 2020, @02:33AM
I run Debian Stretch to host a bunch of Xen VMs, and on each system I've done this (that's running systemd), the system always crashes on shutdown - unless I manually shutdown all Dom-U (guest) hosts first. I haven't spent a significant amount of time investigating, but it seems like systemd is doing something to the logical volumes (or the associated volume groups) prior to the guests finishing shutdown - maybe deactivating the volume group because it doesn't detect mounted volumes on the Dom-0 or something silly. I'm not sure. I just know it is not an issue under SysVinit.
Anyway, imagine an environment with a bunch of NAS boxes that host VM OSs, and somebody wanting to fire up a bunch of VMs on a single host, maybe to simulate a production environment for troubleshooting something, and the number of NFS mounts would quickly add up. It's certainly a valid use case, common or not.
It's GNU/Linux dammit!
(Score: 0) by Anonymous Coward on Monday May 11 2020, @04:04AM
An unusual use case to be sure. The question is if it is significant enough to cater for.
(Score: 2, Insightful) by exa on Sunday May 10 2020, @08:23PM (5 children)
Systemd OSes are for shiny toys, not servers. (Not talking about ubuntu...)
Technically, this is more likely a problem in the underlying kernel NFS implementation, which you can still dodge by making a service that mounts and unmounts your stuff in a slower manner.
(Score: 4, Funny) by qzm on Monday May 11 2020, @01:15AM (4 children)
So could you please point at the supported and widespread syerver read Linux distros that are available from cloud services?
(Seriously, in his discussions of the new homeD, Pottering actually handwaved away the inability to ssh into a newly booted machine - 'you shouldnt need to do that'.)
(Score: 2) by janrinok on Monday May 11 2020, @08:28AM (3 children)
You don't have to use homed on systemd distros - it is optional. If you want to leave home directories exactly as they are now you can do so - and you can still ssh into them. If, however, you would rather have encrypted home directories then you cannot use ssh. But that is nothing new - it is not possible to ssh into any encrypted storage until it has been unlocked. Encryption that can be defeated by ssh isn't encryption anyway.
This doesn't excuse Poetterings hand waving away of using ssh - but I would have hoped (against all odds) that he could have explained the situation better.
(Score: 2) by DannyB on Monday May 11 2020, @05:26PM (2 children)
It's optional -- for now. And it's called systemd-homed.
The lower I set my standards the more accomplishments I have.
(Score: 2) by janrinok on Tuesday May 12 2020, @06:41AM (1 child)
I know its systemd.homed - you know that from 'systemd' in the title. Homed is perfectly understandable. People use journald, systemctl , and loginctl without having to prefix them with 'systemd.' Why should homed be any different?
The day it is no longer optional is the day I will object to it. I don't object to black cars on the road, people wearing funny hats, or those that want to wear odd coloured socks. I will complain about them when they are made compulsory.
I have that version of systemd (245) on 4 of my systems but I do not use homed. It is OPTIONAL until as you say, it isn't (maybe, one day, eventually, ...) It works just like it always has done. I can ssh into my home directories without any hassle. There hasn't been an outcry when every copy of Ubuntu, Debian or Fedora users suddenly stopped working - because it hasn't happened. Like all software, you have to know how and when to use it.
(Score: 2) by DannyB on Tuesday May 12 2020, @03:46PM
I'll bet you don't write GNU/Linux either.
(nor do I)
The lower I set my standards the more accomplishments I have.
(Score: 3, Interesting) by PinkyGigglebrain on Sunday May 10 2020, @08:29PM (8 children)
347 NFS mounts.
systemd has a problem with it.
Would a different init package have handled it better?
As much as I love reading about systemd's failings In this example I have to play the Devil's advocate.
"Beware those who would deny you Knowledge, For in their hearts they dream themselves your Master."
(Score: 4, Informative) by Anonymous Coward on Sunday May 10 2020, @11:24PM (1 child)
Even sysVinit would have handled that without blinking. It would have taken a while, but that is a given with that many mounts. Tripping the OOMkiller is not acceptable.
(Score: 2) by PinkyGigglebrain on Monday May 11 2020, @12:56AM
Thanks for the info. I've never had to deal with anywhere even remotely near that many NFS mounts on a sysVinit system so I never thought to really look into how sysVinit handled unmounting remote drives before.
"Beware those who would deny you Knowledge, For in their hearts they dream themselves your Master."
(Score: 2) by tekk on Monday May 11 2020, @12:39AM (3 children)
Other init systems are less obsessively parallel. The problem is that it tried to spawn a few hundred unmount instances at once on a system with (for these days) relatively little ram.
(Score: 2) by PinkyGigglebrain on Monday May 11 2020, @12:58AM
Thank you for the reply and info.
One more item on my list of "why systemd will NEVER be on a server I set up" reasons.
"Beware those who would deny you Knowledge, For in their hearts they dream themselves your Master."
(Score: 5, Informative) by rleigh on Monday May 11 2020, @12:07PM (1 child)
Parallelising unmounting is also (as a general rule) completely pointless. You've got to consider the I/O resulting from each umount, and also that you have to process the graph of mounts from leaf nodes up to the root. It's 100% reliable to unmount in reverse mount order in sequence (like 'tac /proc/mounts | while read fs; do umount "$fs"; done', IIRC the old Debian initscripts did this when I last worked on them). And it's usually not going to be much slower than parallelising it since it's I/O bound. And you're not introducing any obscure failure cases.
You also don't strictly need to unmount. Switch each mountpoint to readonly, which will flush any pending writes, and then power off with them all mounted readonly once everything is synced.
Unix is all about having simple and composable pieces which can be assembled into larger more complex functionality. From the system calls to the userspace tools. Anyone who wants to learn should be able to understand how any part of the system functions. The old way may have been imperfect, but it was unique compared with proprietary systems in that you could see every command used to boot the system, from the initramfs all the way to bringing up the GUI. That was powerful, and empowering: anyone could tweak it to meet their needs. Declarative syntax might be superficially "simpler" but we've lost a great deal of what made Linux, Linux.
(Score: 4, Insightful) by TheRaven on Monday May 11 2020, @12:49PM
It's worse than that. You're typically going to be hitting kernel locks when you mount or unmount a filesystem, so even if you run 300 umount processes, you'll almost certainly then have them all wait on the same in-kernel lock. You could probably improve performance by having a umount process that does a load of umount system calls in sequence, rather than creating an entire new process for each process though.
sudo mod me up
(Score: 2) by Bot on Monday May 11 2020, @12:51AM
I bet the GNU umount command has already been confronted with the problem of a shitload of volumes to umount. decades of use tend to make software resilient. it's a matter of spending half an hour and test, so, you might want to surprise us.
Account abandoned.
(Score: 2) by HiThere on Monday May 11 2020, @12:55AM
My guess:
Many shutdown managers would have shut them down in sequence rather than in parallel. Other managers would limit the number of active threads based on available resources. The second is probably faster, but (slightly) more complicated.
Javascript is what you use to allow unknown third parties to run software you have no idea about on your computer.
(Score: 5, Informative) by Anonymous Coward on Sunday May 10 2020, @09:02PM (1 child)
Once again, stallman was right.
"Open" is weasel marketing term. "Free" is what we want.
Be Free.
(Score: 2) by Thexalon on Monday May 11 2020, @12:52PM
While I agree RMS has an annoying tendency to be right about technical stuff, this isn't a question of "Free" versus "Open Source", but rather that the BDFL model of project organization doesn't work if the Dictator For Life isn't at all Benevolent but somehow still gets his stuff adopted.
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 0) by Anonymous Coward on Monday May 11 2020, @03:41AM (1 child)
So, how long before systemd either sprouts another sub-daemon to replace NFS or incorporate such a replacement into homed?
(Score: 0) by Anonymous Coward on Monday May 11 2020, @06:31AM
In a way, they sort of already have one in mount or automount units. Or you can use the automount option in the fstab
(Score: 0) by Anonymous Coward on Monday May 11 2020, @04:08AM
It's not an init system! It's a joke!
Gotcha!
(Score: 2) by bradley13 on Monday May 11 2020, @08:53AM (2 children)
Ok, shutting down your computer shouldn't cause it to run out of memory. Creating huge numbers of parallel processes is...not smart. It seems pretty obvious that you want to limit this. So: systemd is bad. This also shows a lack of testing: a piece of software like systemd ought to be stress-tested in every conceiveable way.
But, really? Having a VM with 347 mounted devices? That's not even an edge case, that's an edge of an edge of an edge. This guy shouldn't be even remotely surprised when he runs into problems with a configuration like that.
Everyone is somebody else's weirdo.
(Score: 3, Informative) by DannyB on Monday May 11 2020, @05:30PM (1 child)
As a Java developer, I could point out the Java solution. Simply require systems to have more memory. Much more memory. Much MUCH more memory.
The lower I set my standards the more accomplishments I have.
(Score: 1, Funny) by Anonymous Coward on Monday May 11 2020, @07:13PM
Nah, just allocate huge amounts of swap and put swap in a ram disk to keep it snappy!