from the stormy-weather dept.
System adminsitrator Chris Siebenmann has found Modern versions of systemd can cause an unmount storm during shutdowns:
One of my discoveries about Ubuntu 20.04 is that my test machine can trigger the kernel's out of memory killing during shutdown. My test virtual machine has 4 GB of RAM and 1 GB of swap, but it also has 347 NFS[*] mounts, and after some investigation, what appears to be happening is that in the 20.04 version of systemd (systemd 245 plus whatever changes Ubuntu has made), systemd now seems to try to run umount for all of those filesystems all at once (which also starts a umount.nfs process for each one). On 20.04, this is apparently enough to OOM[**] my test machine.
[...] Unfortunately, so far I haven't found a way to control this in systemd. There appears to be no way to set limits on how many unmounts systemd will try to do at once (or in general how many units it will try to stop at once, even if that requires running programs). Nor can we readily modify the mount units, because all of our NFS mounts are done through shell scripts by directly calling
mount; they don't exist in
/etc/fstabor as actual
We've been here before and there is certainly more where that came from.
(2020) Linux Home Directory Management is About to Undergo Major Change
(2019) System Down: A systemd-journald Exploit
(2017) Savaged by Systemd
(2017) Linux systemd Gives Root Privileges to Invalid Usernames
(2016) Systemd Crashing Bug
(2015) tmux Coders Asked to Add Special Code for systemd
(2016) SystemD Mounts EFI pseudo-fs RW, Facilitates Permanently Bricking Laptops, Closes Bug Invalid
(2015) A Technical Critique of Systemd
(2014) Devuan Developers Can Be Reached Via firstname.lastname@example.org
(2014) Systemd-resolved Subject to Cache Poisoning
Whether you're running systemd happily or begrudgingly, it's best if you disable systemd-resolved as your DNS resolver for the time being. Reported today at seclists is a new DNS cache poisoning bug in systemd-resolved.
At its simplest, an attacker triggers a query to a domain he controls via SMTP or SSH-login. Upon receipt of the question, he can just add any answer he wants to have cached to the legit answer he provides for the query, e.g. providing two answer RR's: One for the question asked and one for a question that has never been asked - even if the DNS server is not authoritative for this domain.
Systemd-resolved accepts both answers and caches them. There are no reports as to the affected versions or how widespread the problem may be. Comments over at Hacker News suggests that it might not be widespread, most users would still be running the backported 208-stable while the DNS resolver was committed in 213 and considered fairly complete in 216, but that is if they enabled systemd-resolved in /etc/nsswitch.config.
Devuan is spelled in Italian and it is pronounced just like "DevOne" in English.
[...]is it really a fork?
This is just the start of a process, as bold as it sounds to call it a fork of Debian.
[...]Devuan aims to be a base distribution whose mission is protect the freedom of its community of users and developers. Its priority is to enable diversity, interoperability and backward compatibility for existing Debian users and downstream distributions willing to preserve Init freedom.
Devuan will derive its own installer and package repositories from Debian, modifying them where necessary, with the first goal of removing systemd, still inheriting the Debian development workflow while continuing it on a different path: free from bloat as a minimalist base distro should be. Our objective for the spring of 2015 is that users will be able to switch from Debian 7 to Devuan 1 smoothly, as if they would dist-upgrade to Jessie, and start using our package repositories.
Devuan will make an effort to rebuild an infrastructure similar to Debian, but will also take the opportunity to innovate some of its practices. Devuan developers look at this project as a fresh new start for a community of interested people and do not intend to enforce the vexation hierarchy and bureaucracy beyond real cases of emergency. We are well conscious this is possible for us mostly because of starting small again; we will do our best to not repeat the same mistakes and we welcome all Debian Developers willing to join us on this route.
This is a in-depth architectural critique of systemd. It claims to be the first purely technical review of systemd internals, and provides a detailed analysis of several components. It criticizes on the basis of ordering related failures, a difficult to predict execution model, non-determinism in boot-order, as well as several other points.
Though many users would perceive the long processing pipeline to increase reliability and be more "correct" than the simpler case, there is little to acknowledge this. For one thing, none of jobs, transactions, unit semantics or systemd-style dependencies map to the Unix process model, but rather are necessary complications to address issues in systemd being structured as an encapsulating object system for resources and processes (as opposed to a more well-defined process supervisor) and one accommodating for massive parallelism. Reliability gains would be difficult to measure, and that more primal toolkits like those of the daemontools family have been used in large-scale deployments for years would serve as a counterexample needing overview.
A number of users have reported that running "rm --no-preserve-root -rf /" not only deletes all their files (as expected), but also permanently bricks their computers (which is not). Tracing the issue revealed that the ultimate cause was that SystemD mounted the EFI pseudo-fs as read-write even when this FS was not listed in fstab, and deleting certain files in this pseudo-fs causes certain buggy, but very common, firmware not to POST anymore. A user reported this bug on SystemD's GitHub issue tracker, asking that the FS be mounted read-only instead of read-write, and said bug was immediately closed as invalid. The comment thread for the bug was locked shortly after. Discuss.
The spreading of systemd continues, now actively pushed by themselves unto other projects, like tmux:
"With systemd 230 we switched to a default in which user processes started as part of a login session are terminated when the session exists (KillUserProcesses=yes).
[...] Unfortunately this means starting tmux in the usual way is not effective, because it will be killed upon logout."
It seems methods already in use (daemon, nohup) are not good for them, so handling of processes after logout has to change at their request and as how they say. They don't even engange into a discussion about the general issue, but just pop up with the "solution". And what's the "reason" all this started rolling? dbus & GNOME coders can't do a clean logout so it must be handled for them.
Just a "concidence" systemd came to the rescue and every other project like screen or wget will require changes too, or new shims like a nohup will need to be coded just in case you want to use with a non changed program. Users can probably burn all the now obsolete UNIX books. The systemd configuration becomes more like a fake option, as if you don't use it you run into the poorly programmed apps for the time being, and if they ever get fixed, the new policy has been forced into more targets.
Seen at lobsters 1 & 2 where some BSD people look pissed at best. Red Hat, please, just fork and do you own thing, leaving the rest of us in peace. Debian et al, wake up before RH signed RPMs become a hard dependency.
Security researcher and MateSSL founder, Andrew Ayer has uncovered a bug which will either crash or make systemd unstable (depending on who you talk to) on pretty much every linux distro. David Strauss posted a highly critical response to Ayer. In true pedantic nerd-fight fashion there is a bit of back and forth between them over the "true" severity of the issue and what not.
Nerd fights aside, how you feel about this bug, will probably largely depend on how you feel about systemd in general.
The following command, when run as any user, will crash systemd:
NOTIFY_SOCKET=/run/systemd/notify systemd-notify ""
After running this command, PID 1 is hung in the
pausesystem call. You can no longer start and stop daemons. inetd-style services no longer accept connections. You cannot cleanly reboot the system. The system feels generally unstable (e.g. ssh and su hang for 30 seconds since systemd is now integrated with the login system). All of this can be caused by a command that's short enough to fit in a Tweet.
Edit (2016-09-28 21:34): Some people can only reproduce if they wrap the command in a
while trueloop. Yay non-determinism!
Submitted via IRC for TheMightyBuzzard
A bug in Linux's systemd init system causes root permissions to be given to services associated with invalid usernames, and while this could pose a security risk, exploitation is not an easy task.
A developer who uses the online moniker "mapleray" last week discovered a problem related to systemd unit files, the configuration files used to describe resources and their behavior. Mapleray noticed that a systemd unit file containing an invalid username – one that starts with a digit (e.g. "0day") – will initiate the targeted process with root privileges instead of regular user privileges.
Systemd is designed not to allow usernames that start with a numeric character, but Red Hat, CentOS and other Linux distributions do allow such usernames.
"It's systemd's parsing of the User= parameter that determines the naming doesn't follow a set of conventions, and decides to fall back to its default value, root," explained developer Mattias Geniar.
While this sounds like it could be leveraged to obtain root privileges on any Linux installation using systemd, exploiting the bug in an attack is not an easy task. Geniar pointed out that the attacker needs root privileges in the first place to edit the systemd unit file and use it.
[...] Systemd developers have classified this issue as "not-a-bug" and they apparently don't plan on fixing it. Linux users are divided on the matter – some believe this is a vulnerability that could pose a serious security risk, while others agree that a fix is not necessary.
See, this is why we can't have nice init systems.
The other day, Michael W Lucas, who is normally known for good technical literature, put up a wild experiment of a short story: Savaged by Systemd. It's erotica, sort of. It's computer erotica, to be specific. It's Linux sysadmin erotica, to be more specific. OK, fine, it's systemd erotica. Really. Anyway, despite the subject and the genre, and in spite of the combination of the two, the e-book is trending and rising in quite a few lists.
Hopefully he can still remain focused on Absolute FreeBSD and be able to get that finished by the next BSDCan.
[Ed note: Has anybody actually bought and read this short story? I wasn't going to spend $2.99 to see what the hubub was about. - cmn32480]
From TFA (the friendly article) at https://www.openwall.com/lists/oss-security/2019/01/09/3:
We discovered three vulnerabilities in systemd-journald (https://en.wikipedia.org/wiki/Systemd):
- CVE-2018-16864 and CVE-2018-16865, two memory corruptions (attacker-controlled alloca()s);
- CVE-2018-16866, an information leak (an out-of-bounds read).
CVE-2018-16864 was introduced in April 2013 (systemd v203) and became exploitable in February 2016 (systemd v230). We developed a proof of concept for CVE-2018-16864 that gains eip control on i386.
CVE-2018-16865 was introduced in December 2011 (systemd v38) and became exploitable in April 2013 (systemd v201). CVE-2018-16866 was introduced in June 2015 (systemd v221) and was inadvertently fixed in August 2018.
We developed an exploit for CVE-2018-16865 and CVE-2018-16866 that obtains a local root shell in 10 minutes on i386 and 70 minutes on amd64, on average. We will publish our exploit in the near future.
To the best of our knowledge, all systemd-based Linux distributions are vulnerable, but SUSE Linux Enterprise 15, openSUSE Leap 15.0, and Fedora 28 and 29 are not exploitable because their user space is compiled with GCC's -fstack-clash-protection.
This confirms https://grsecurity.net/an_ancient_kernel_hole_is_not_closed.php: "It should be clear that kernel-only attempts to solve [the Stack Clash] will necessarily always be incomplete, as the real issue lies in the lack of stack probing."
The article goes on with more detailed information on exploits.
<sarcasm>It's a good thing that systemd does not affect very many systems and no systems running anything important.</sarcasm>
With systemd 245 comes systemd-homed. Along with that, Linux admins will have to change the way they manage users and users' home directories.
[...] Prior to systemd every system and resource was managed by its own tool, which was clumsy and inefficient. Now? Controlling and managing systems on Linux is incredibly easy.
But one of the creators, Leannart Poettering, has always considered systemd to be incomplete. With the upcoming release of systemd 245, Poettering will take his system one step closer to completion. That step is by way of homed.
[...] let's take a look at the /home directory. This is a crucial directory in the Linux filesystem hierarchy, as it contains all user data and configurations. For some admins, this directory is so important, it is often placed on a separate partition or drive than the operating system. By doing this, user data is safe, even if the operating system were to implode.
However, the way /home is handled within the operating system makes migrating the /home directory not nearly as easy as it should be. Why? With the current iteration of systemd, user information (such as ID, full name, home directory, and shell) is stored in /etc/passwd and the password associated with that user is stored in /etc/shadow. The /etc/passwd file can be viewed by anyone, whereas /etc/shadow can only be viewed by those with admin or sudo privileges.