Hugh Pickens writes:
Systemd has turned into the Godzilla of Linux controversies. "Everywhere you look it's stomping through blogs, rampaging through online discussion threads, and causing white-hot flames that resemble Godzilla's own breath of death," writes Jim Lynch. Now Sam Varghese reports at iTWire that although Linus Torvalds is well-known for his strong opinions, when it comes to systemd, Torvalds is neutral. "When it comes to systemd, you may expect me to have lots of colorful opinions, and I just don't," says Torvalds. "I don't personally mind systemd, and in fact my main desktop and laptop both run it."
Oh, there's been bitter fights before. Just think about the emacs vs vi wars. Or, closer to systemd, the whole "SysV init" vs "BSD init" differences certainly ended up being things that people had "heated discussions" about. Or think about the desktop comparisons.
I'm not really sure how different the systemd brawls are from those. It's technical, but admittedly the systemd developers have also been really good at alienating people on a purely personal level too. Not that that is anything particularly new under the sun _either_: the (very) bitter wars between the GPL and the BSD license camps during late-80s and early-90s were almost certainly more about the persons involved and how they pissed off people than necessarily deeply about other differences (which existed, obviously, but still).
Torvalds was asked if systemd didn't create a single point of failure which makes a system unbootable if it fails. "I think people are digging for excuses. I mean, if that is a reason to not use a piece of software, then you shouldn't use the kernel either."
at least not to the extent of a proper industry-grade product....<snip>...There are complex problems with it that you're unaware of as an armchair developer.
Just to clarify my previous post, for many years I've been the lead developer for a server based data center product currently running all over the world, which also has been in heavy use for years by data center assessment teams for IBM and HP, among others. It's that very sort of experience with the server side of things that makes me find systemd and everything about it totally revolting. It also makes it a little hard to accept condescending crap from some arrogant AC frankly.
Obviously I'm not saying there isn't a place for binary formats, with things like databases being the obvious example. Remember however this all started with my comment about binary log files. There's no possible way you're convincing me, or most anyone, that that's a good idea. There's no excuse for critical log files to be dependent on anything aside from a readable file system.
...and how about not assuming you know anything about me. I've been an "armchair" developer for a living since 1985.
1. Yeah, because corruption can't happen on plaintext files.. the thing is you'd probably not even notice
2. Yes, I just described things that can't be realistically done with plaintext formats. It's been tried, it always results in headaches. Many of those headaches have been on my head. Yes, you're right, HYPOTHETICALLY it can (and there are very compelling arguments why it should) be done with plaintext. HYPOTHETICALLY the same is true of microkernels vs. monolithic kernels, you tell me how that one worked out. Things that work hypothetically don't always work out in practice.
3. No, Windows hasn't proved the failure of the approach. To say that is like to say that Windows has proved that terminals suck because Windows's terminal sucks, or that microprocessors are unreliable because Windows can crash. (And I am as much of a Microsoft hater as _gweg, I go out of my way to avoid anything MS and, where possible, try to make my stuff explicitly incompatible with it)
It's just as easy to write user friendly frontends for text formats as binary.
I wish. What this tells me about you is that you've never actually done such a thing, at least not to the extent of a proper industry-grade product. There are complex problems with it that you're unaware of as an armchair developer. I've seen them first hand, they're one of the things that made me fed up with KDE (which I otherwise mostly loved) and made me flee to other options.
Wow...I don't even know where to start. Nothing in anything you wrote makes a single solid argument for binary formats, especially ones that can get corrupt like the systemd logs, which apparently is NOTABUG:
You've described nothing that can't be done, and shouldn't be done with text formats. Windows has proved that failure of the approach for decades. It's just as easy to write user friendly frontends for text formats as binary. Insane. There's a reason the 'nix has avoided this for 40 years.
binary logs is definitely a point against it
Agreed : ) I run openRC and have little interest in systemd.
Yes we need something like systemd. Don't believe me? Try making a system which is suitable for end-users with just a bare kernel.
The only way to end up with something that meets the expectations (the user's, not your's) is to rewrite systemd. Maybe you'll do it better, maybe worse, but you'll still eventually rewrite systemd's functionality.
For me, I just can't imagine how anyone isn't sent screaming into the night by the very idea of that binary logging alone.
That's me. I think binary logging is fine. Hell, I even prefer it. I am NOT a stranger to either binary formats or to fixing systems that use them and which have become unusable. I use Enlightenment as my daily desktop and it uses a binary format for all of it's config files. Turns out that editing configuration with a purpose-built program is, in practice, much handier than using something generic like vim (setting up Enlightenment is more comfortable than, say, PekWM or Fluxbox would be if they had the same levels of functionality, which they only have a small fraction of), sticking to simple plaintext formats is impractical because it comes with A LOT of subtle failure modes (see: KDE with it's magically disappearing and easily corruptible configuration), and making a "plaintext" format complex enough to support less-error-prone development results in formats that are a PITA to view/edit in their plaintext form (see: Openbox with it's xml config file -- or anything with xml configs, really) and are still error-prone, with the errors just having been shifted elsewhere (ask anyone who has worked with xml).
Just because a format is binary doesn't mean you can't use your standard tools on it. The one time I locked myself out of Enlightenment because I enabled an option that made my graphics driver go nuts? Extract the plaintext with the included tool, edit with vim, and encode it again. No big deal.
While that was for config files, and not log files, the way I see it the binary format brings similar benefits to logging. I'm all for that.
Thanks, I'll use that.
I actually always spell it wrong on purpose, because I can't quite remember the right spelling (but strangely I can remember this one).
Some would argue that compressed logs are as unreadable as systemd logs because an external tool is required.
An external tool? I'm pretty sure you can find gzip and tar (log files aren't typically tar'd) on the system the compressed log file is on. And you can sure as hell find them on more systems than you can find systemd, should you need to view the log file somewhere else. You can even use pipes to send these log files to whichever viewer you like. By the way, last time I used 'less', it would automatically uncompress files to view them.
In no way is saying 'compressed text logs are as unreadable as binary logs' a fair statement. Basically any platform on earth can understand a gzip'd text file. What format is the systemd binary log written in? If someone writes a parser for it, how can they be sure it won't change in two years?
I really don't know much about systemd, and thus obviously haven't made up my mind about it, but binary logs is definitely a point against it.
I have never, not once, understood why people whined (yes, whined) about SystemD killing Linux (Paul Venezia comes to mind). Did Poettering go shoot all of the SysVinit devs and personally come to your house, uninstall SysVinit on all your machines, and delete all copies of SysVinit in existence? Let's not even talk about which one is better or worse. If you want to use SysVinit, then keep using it.
Now, you might say, all of the major distros are moving to SystemD. So what? It is the distro devs' prerogative to make choices about their distribution. If you don't like it, fork it. You don't even need majority share or anything, you just need to attract enough like-minded people to keep the distro afloat. If you can't, well, maybe that says something about the reasons you want to use SysVinit over SystemD, namely, that no one else agrees with you enough to contribute to your cause (or that the people who think SysVinit is better are people who are not skilled or knowledgeable enough to maintain a distribution, which says something about the validity of their opinions in the first place). Remember, open source is a (brutal) meritocratic anarchy. The best one wins, whether best means technically best, best community, or the de facto Unix philosophy, "worse is better". So instead of whining about SystemD, go make an alternative init or a new SysVinit distro or something.
Which differs depending on version that needs a special source code with special recursive dependencies that requires download and chat room help. Which depends on the network you were trying to repair.. ? :D
In another world one would just read the text log and change the text based configuration file?
SystemD seems like a development with good intentions but without experience and foresight. But the path to debug hell is often paved with a nice vision that doesn't match the full reality.
This is a problem with many long-term projects. When the project starts, everyone is inexperienced in the developing technology. However, after a decade or so, a core or regular contributors develop a style of collaboration which can be impenetrable to newbies. Linux kernel development is an extreme example but I understand that it occurs to a lesser extent with Perl development. In this case, a skills gap has occurred where people are either doing elementary stuff, like deploying blog software on cheap virtual hosting, or they're doing bleeding-edge stuff on their own infrastructure. This situation is worsened because the virtual hosting providers don't update their packages unless there is a pressing security issue. So, the lowest common denominator is a five year old interpreter and two year old set of modules. Steps have been taken to improve advancement, portability and uptake but differing use cases will perpetuate a divide.
Returning to Linux kernel development, relatively few people have to venture into kernel-space and those who do may not be aware of typical use cases which are understood by regular contributors.
Of course there is. You just have to go through those binary logs...
At first he compared systemd to the kernel, then walked that back and compared it to glibc instead.
I forgot to mention in my reply above: I'm sure he backed off on that because he realized the bad analogy. When it comes to the kernel, there a countless modules you'll never load, for example, on your companies web server obviously. You use whatever you need. Systemd is decidedly not modular in that way, and it seems as though that's by design.
I don't want to showcase my naivete any more than I have to, but I can't agree with a lot of the things he said from a technical perspective, especially the dismissal of introducing complexity to a single point of failure.
Absolutely, especially if you're talking about servers. Even kernel devs have commented on the 1 MiB size of the systemd executable as being troubling and being a potentially big attack surface. Combine that with the fact that almost nothing in there is desirable on a server in the first place, which means you're just asking for more bugs and vulnerabilities with no upside whatsoever.
I think Linus knows his opinion would carry far too much weight for him to get involved, and he's arguably correct.
Code to de-integrate systemd dependancies from common libraries. Example from recent debian bug report (762116):
I suspect the culprit here is packages which perform a broad array of functions, rather than doing one thing and doing it well. So brasero needs X functionality, which can be found in package W. Package W also provides Y functionality, which depends on systemd-sysv. So therefore brasero depends on systemd-sysv, even though it doesn't *need* it.
... meaning, library package W is in desperate need of refactoring.
The response from an apologist was, well, apologetic and didn't make a lot of sense at first:
brasero needs to know when a CD is inserted, and it needs access to the raw device to actually burn the CD. This access is granted to the person physically logged on the machine. The only way to know who is logged on is through systemd-logind.
I was surprised to read this, since last I looked under the hood, debian used consolekit. Unfortunately a bit of research today revealed that consolekit (which was reasonably platform neutral, having ports to BSDs) was deliberately orphaned with Poettering on the team. I'm starting to see a pattern with this particular developer.
Systemd is probably a bad design, and the debian project's decision to have alternate init systems available is coming up against the hard reality that systemd does not play well with others even when in a different room.
The purpose of that package is to allow linux programs with systemd dependency be easily ported and ran on xBSD.
# zfs set compress=lz4 rpool
"I mean, if that is a reason to not use a piece of software, then you shouldn't use the kernel either."
It IS a good reason not to use the kernel. It's just that as of yet, we don't have any sufficiently functional microkernel options available for general purpose computing. GNU Hurd seems like it might get there sometime in the next decade or so, but others that exist seem mostly suited only to lab use. Given a reasonably secure Hurd or other microkernel distro with enough hardware support to make it worth developing for, I'd be pretty thrilled to ditch the kernel, personally.
This stance from Linus isn't a strange one; he's always been pretty cozy with monoliths.
Assuming the cancer hasn't spread beyond Linux.
It tried but was "benignified" at the gate.
See for example here [undeadly.org].
Current text logs are "rotated" after a period of time. During rotation they are tarred and gziped which makes them very small. Some would argue that compressed logs are as unreadable as systemd logs because an external tool is required.
Welp, looks like I'll be installing PC-BSD when Xubuntu 12.04 support lapses. Assuming the cancer hasn't spread beyond Linux.
See if I talk to you anymore.
"and he'd literally be the cause of shutting down multiple markets, likely "
Oh and forgot to mention that old piece of computing history mythology the "vaxen don't belong here" story was not so ancient at this time, maybe only 10 years old, and everyone knew that story. You know you're really in the shit when you're living out something like mythology in real life. Thats the time to panic, or at least a good excuse. Other than our scenario was more like Sun gear with "sunos" (ancestor of Solaris aka slow-aris) but VAXen. But whatever.
You also know its a good time to panic when you get to work and the CEO of the corporation has stolen your chair to sit in the machine room looking very unhappy. My first reaction was OMFG I hope my networking gear is up and it isn't my fault. Which it wasn't. Somehow nobody got fired that day because it was no ones fault and IBM never really admitted what killed that machine backplane/chassis. Cosmic rays or who knows.
Small world, i was reading up on Minux last night. Was thinking about attempting a retro-computer and exploring OS options.
"motherboard? I cant say I've seen any servers with redundant motherboards."
Talk to IBM about "real computing" aka mainframes. Thats old stuff. Lost a backplane on a clustered machine in 1994 in the middle of a trading day. No big deal. Cluster didn't even hiccup although the IBM CE was tearing his hair out in panic. (OMFG 5% of all stock trades in the entire conus in all markets went thru this system and we were redundant normally, but we lose one more, or the CE makes one false step, and he'd literally be the cause of shutting down multiple markets, likely ...) This was an outsourced financial services provider for brokerages back when outsourcing meant onshore just not your own W2 employees, not by default sending to India.
You can get something conceptually similar in some scenarios with BGP anycast but thats cheating (this ip address exists in every routing table... it just seems to exist on numerous hardware). To do this right your BGP speaker has to be the service providing machine such that if it crashes it takes the BGP advertisement with it. That was related to my second "real job" in the 90s-00s. The first time a router jockey runs into the idea that "a" IP address doesn't really mean "a" machine it confuses the F out of them, guaranteed. This works better with things that provide a reasonably stateless service like NTP, not so good with bank balance databases perhaps.
Let the particular well understood subsystems do their job and hook into the 'new' way.
Let the particular well understood subsystems do their job and hook into the 'new' way.
Isn't that basically what the BSDs are doing with OpenRC? Replace the quirky part (rc.d), and hook the replacement up with the working parts of the system, instead of replacing everything.
From what I heard (IAKALKD) the Linux kernel development community is really good at alienating people too.
Takes longer to fill up the file system.
That is a pretty good assessment.
Much like the old xfree86 stuff. We used it because it worked and was about all there was. Once the real X11 guys paid any attention to linux and didnt do crap things everyone ran for the hills. The same will happen with systemd.
The *idea* of systemd is a decent one. Get the system started up in a consistent non cryptic way. Put in watchdogs for known services. Give external applications one API to get the status. The init.d way served us well for a long time. But it has shown it is brittle and easy to break. But by necessity it logged itself very well and very simply. People are understandably upset that it is now 'binary' for the logging. So instead of using less/cat/more/tail/grep to find something out you need to use some other specialized tool that needs all the features of those apps.
The systemd guys have a brilliant idea. They need to stop dragging things into it though. Let the particular well understood subsystems do their job and hook into the 'new' way. Reinventing the wheel is pretty much how linux got started. But they have decided everyone should use their new wheel. Which is not how linux started... It just serves to piss people off. Eventually someone *WILL* fork it and leave RH as irrelevant.
For me as a normal 'end user'. It does not affect me much. Other than it is a pain to look at the logs now. As there is 'yet another way' to look things up. As I still have to use all my old systems and old busybox distro installs.
Bitkeeperd is greatWe are going to use bitkeeperdUntil the developers pull one asshole stunt too manyThen linus has to author a replacement
This should be enough to reject it right there.
No debugging ability?
I got into a war withe systemd last weekend. System was doing strange things, requesting authorization (user password) on each plugging and unplugging of the USB stick or drive. Trying to access the network, again the authorization panels. I rebooted and rebooted and rebooted, it fainlly went away. It was like a function was not starting correctly and there was no word what that function was. I came close to reloading back to 12.04.
Do I like systemd? -- do not care one way or another. But the above was not a good user experience.
Unfortunately people really like to do some bashing more that contributing.
Contributing to what exactly?
Haven't tried it myself, but supposedly Minix is (much) more than just an/for academic exercise.
It runs on x86 and ARM CPUs, is compatible with NetBSD, and runs thousands of NetBSD packages.
It was only with the third version, MINIX 3, and the third edition of the book, published in 2006, that the emphasis changed from teaching to a serious research and production system, especially for embedded systems. [http://www.minix3.org/other/read-more.html]
For me, the systemd "war" appears to be divided between two camps; on one side, young reckless developers who think that experience isn't all and that if users don't like their radical new ideas they should just fork the code and stop complaining, on the other, experienced power users and system administrators who like things simple and modular, have seen complex crazy ideas blow up spectacularly several times, but can't develop - be it by a lack of time, or lack of skill in that area.Developers just dismissing warnings and complaints with a "if you don't like fork it" only serves to get the other camp mad and dismiss developers as "arrogant idiots". Neither of the groups understands the other...
Yeah, I never got around to trying that one out. Thanks.
Does it work?
so the linux kernel is like the car engine and the rest is... the rest.so obviously the future is shaped by people that DO not by people that TALK.so just like google took the engine and made a android, I guess these systemDee guys are taking the kernel and build8ng their own kind of car.speculationonly but at this point it seems systemdee people are using as mich programming resources as possible to get this car on the road and once it runs okay it will probably be called ... gnome OS.this is all just a top down impression im getting, tbh I dont understand enough of this linux stuff anyways, andd there is some interesst in keeping it difficult and hidden because it can provide a livelihood for a "sys admin". I mention this because some migjt say, if you dont like it fork it or roll your own.in the end I think in the end systemdee proves that interpreted programs (scripts) are slower then compiled programs but that scripts are easier to read and fix then binary blobs?I guess this is just the direction the "young uns" are taking ... it.which leaves just one important closing thought: maintainability. it was important innoriginal unix philosophy and im not sure if systemdee will be in 30 years when the torch will pass naturally to the next generation?also maybe think about "unrealistic" situations like a momster meteor strike and lota of dead people world wide or plague or a cascading nuclear meltdowns that leave lots of brain damaged savages ... how maintainable would it be in such a situation?
We could use a microkernel too.
Well, then you'll be happy to know that Minix 3.3 was recently released [minix3.org].
We could use a microkernel too. And working one. So we could have more choice, and I would definately choose that.
To go with your point that Linus isn't afraid of complaining, he went off on Kay Sievers [theregister.co.uk] back in April. He's probably going to stay out of the fight unless one of the systemd devs does something else that causes him pain.
I don't want to showcase my naivete any more than I have to, but I can't agree with a lot of the things he said from a technical perspective, especially the dismissal of introducing complexity to a single point of failure. At first he compared systemd to the kernel, then walked that back and compared it to glibc instead. But there are other libcs, and glibc doesn't require applications to call glibc-specific functions to work properly [ewontfix.com].
I suspect most people remember the udev fiasco a while back, but it's worth noting that the eudev team kicked off their fork by citing Linus' complaints ( http://article.gmane.org/gmane.linux.gentoo.project/2262 [gmane.org] ). So Linus is certainly capable of complaining, and his complaints get things done. Perhaps he's also remembering that, because he was careful to avoid saying anything that could be taken as hard advocacy for anything.
HP superdome series.Even the backplane has redundancy in the latest models.
[..] I'm rather suspicious of fancy new models that change everything and claim it solves things (maybe the newness and complexity and fancy details just make it hard to say that it _doesn't_ have the problems existing systems have, so then that is seen as an argument that the problems no longer exist - not because they truly went away, but simply because they got too hard to argue about because so much changed)
He uses this very good argument against Poettering's new idea, but this is actually the argument I usually bring up about systemd (while he doesn't).
We need a Kernel. We don't need systemd. Options are good. Systemd takes those away. Simple as that.
remember when Dell's website crashed because the HP non-stops they were using stopped?
They just only provide a benefit if proper isolation is enacted between mainboards and baseboard, and if both hardware and software are functioning correctly under all probable failure circumstances (hint: some don't, and some aren't found until they're in the field, where it may or may not be possible to repair, retrofit, or rework them.)
motherboard? I cant say I've seen any servers with redundant motherboards.
Unless you're running a mainframe with redundant processors...
Computer hardware is a single point of failure, think I'll start an argument about that.