Noted Linux expert Chris Siebenmann has described two catastrophic failures involving systemd.
One of the problems he encountered with systemd became apparent during a disastrous upgrade of a system from Fedora 20 to Fedora 21. It involved PID 1 segfaulting during the upgrade process. He isn't the only victim to suffer from this type of bad experience, either. The bug report for this problem is still showing a status of NEW, nearly a month after it was opened.
The second problem with systemd that he describes involves the journalctl utility. It displays log messages with long lines in a way that requires sideways scrolling, as well as displaying all messages since the beginning of time, in forward chronological order. Both of these behaviors contribute to making the tool much less usable, especially in critical situations where time and efficiency are of the essence.
Problems like these raise some serious questions about systemd, and its suitability for use by major Linux distros like Fedora and Debian. How can systemd be used if it can segfault in such a way, or if the tools that are provided to assist with the recovery exhibit such counter-intuitive, if not outright useless, behavior?
Editor's Comment: I am not a supporter of systemd, but if there are only 2 such reported occurrences of this fault, as noted in one of the links, then perhaps it is not a widespread fault but actually a very rare one. This would certainly explain - although not justify - why there has been so little apparent interest being shown by the maintainers. Nevertheless, the fault should still be fixed.
(Score: 4, Insightful) by Anonymous Coward on Sunday December 21 2014, @05:03PM
There's a maxim in software testing that goes like, "For every user who reports a bug, there are thousands of affected users who didn't report it."
I think that's exactly what we're seeing with systemd. Thousands and thousands and thousands of people are getting fucked by it and its numerous flaws. That's why it's such a controversial subject. Yet only a very small fraction of these people ever both to report these systemd bugs.
The rest of us don't bother, because we're moving on to Slackware, or FreeBSD, or OS X, or even Windows. We're done with Linux. Why should we waste our time reporting bugs with software we didn't want to ever use in the first place, and will likely never use ever again?
So when I see two bug reports, I know that realistically, we're talking about probably 20,000 to 50,000, if not more, users who experienced the same problem. That's a serious issue, if you ask me.
(Score: 3, Insightful) by choose another one on Sunday December 21 2014, @05:36PM
Yes, but... thing about a fatal bug in init is that you have a dead system, which is one reason people are wary of new init systems and systemd in particular.
Given that, and that these reports are issues on upgrades (not new installs where users may just give up), and are repeatable crashes (not a one-off retry and never see it again), I think you will end up with a much much higher fraction of the affected userbase reporting the issue.
> The rest of us don't bother, because we're moving on to Slackware, or FreeBSD, or OS X, or even Windows. We're done with Linux.
Um, Slackware is not Linux these days ???
(Score: -1, Flamebait) by Anonymous Coward on Sunday December 21 2014, @05:44PM
Slackware may use the Linux kernel and include open source software, but it's clearly very different from most other contemporary Linux distributions.
I tried it out recently. It's still stuck in the 90s. The user experience was more like what I remember from when I used Xenix and Digital UNIX way back. It doesn't offer any of the good stuff that Linux distros have offered us since the mid 2000s.
Slackware uses the Linux kernel, but it's not a Linux distribution in the modern sense of the term.
(Score: 2) by novak on Sunday December 21 2014, @05:54PM
I'm not really sure what you're talking about. Are you complaining about the lack of a package manager or is there something else?
I keep hearing this "slackware is stuck in the 90s," and I can't say I agree. Especially if you're installing a server which you don't need a whole lot of software on, slackware is a really good system.
If you have a list of things you think are wrong/old, I'd be genuinely interested to see them. I bet I think most of them are features.
novak
(Score: -1, Flamebait) by Anonymous Coward on Sunday December 21 2014, @06:17PM
Yes, the lack of a modern package manager is a good example. This doesn't just affect installing software with many dependencies, but also easily keeping software updated.
But there's also the manual configuration of just about everything. I can install Ubuntu, and have a working desktop system within a few minutes, with only limited, unavoidable manual configuration (like specifying passwords and such). It's the complete opposite when it comes to Slackware.
Slackware has avoided the really bad new stuff, like systemd, only because it has avoided pretty much everything that has been supported by most distros since 2000!
(Score: 2) by VLM on Sunday December 21 2014, @06:30PM
the manual configuration of just about everything.
Doing it by hand = doing it wrong. If not puppet or chef, something homegrown. Computers are supposed to eliminate manual labor, not move it to keyboard/mouse.
Less than 10 minutes from cold dead iron to working and that includes the freebsd install. Images are of course much faster and seem limited solely by NAS bandwidth (which means pretty fast, except when it isn't...) I guess there's ways to automate OS installers but I'm not doing that enough to be motivated to set it up. I would guess slackware can't be much worse than freebsd.
(Score: 2, Informative) by Anonymous Coward on Sunday December 21 2014, @06:32PM
Why would I want to waste my time with Puppet or Chef when I'm just installing Linux on a single workstation for my own personal use?
I just use Ubuntu, which sets up everything I need for me, so I don't have to do it myself. Slackware does none of this.
(Score: 2) by VLM on Sunday December 21 2014, @06:45PM
on a single workstation
Thats a world far away from me... I suppose when you reinstall due to hardware failure or whatever, its nice to be automated and do it in minutes/seconds instead of hours of rework.
Then again you probably have a lot less work to do... No need to screw around with single signon and kerberos and all that
(Score: 3, Informative) by novak on Sunday December 21 2014, @07:08PM
I can also have a working desktop within minutes on slackware. Seconds, if you're talking about post OS install. It probably takes longer in ubuntu unless you consider unity a working desktop, which is sort of a stretch (could be wrong about time to install new GUI in ubuntu, as I never run it. I tried 8.04 once and hated it.). I don't know what magical things there are in other distros that slackware lacks, but I'm guessing GUI menus.
That being said, the reason that slackware doesn't suck is that it requires you to manually configure things. Now... I do prefer having a package manager, but most things are just better when not poorly automated as by ubuntu and friends. Slackware doesn't come with nearly as much crap pre-installed and pre-configured.
If you don't want to spend five minutes setting up your machine the way you want it then enjoy systemd. That's exactly the kind of attitude that birthed it- features first, ease of use before configurability, users are lazy/idiots and must not have any work to do, and a shiny exterior.
novak
(Score: 1) by fido_dogstoyevsky on Sunday December 21 2014, @10:31PM
can also have a working desktop within minutes on slackware. Seconds, if you're talking about post OS install. It probably takes longer in ubuntu unless you consider unity a working desktop, which is sort of a stretch (could be wrong about time to install new GUI in ubuntu, as I never run it. I tried 8.04 once and hated it.). I don't know what magical things there are in other distros that slackware lacks, but I'm guessing GUI menus.
I'm converting from Opensuse to Slackware. The only things I have problems with (1) are things I haven't found out how to do yet (and these are much smaller than the problems I had in going from, say, Windows to OSX (2) or Win98 to Vista (3)).
I haven't had any trouble at all with any of the package managers (choice for them, too) in installing/removing software. If you don't like CLI there's always Gslapt (which is good for when I'm feeling lazy).
It's simple, it works, it's reliable and, when you change your mindset less than you thought you might have to it's also easy. Sounds pretty 21st century stuff to me.
------------------------------------------------------------------
(1) Instead of continuing "doing stuff" with Opensuse and playing with Slackware by dual booting I'm "doing stuff" with Slackware and kept Opensuse on another partition for handholding - haven't needed it yet.
(2) Had to at work, not on my PC.
(3) Vista was the big impetus to get me to try Linux (Mandrake)
It's NOT a conspiracy... it's a plot.
(Score: 2) by novak on Sunday December 21 2014, @05:50PM
Um, Slackware is not Linux these days ???
Obviously it is, but I would say that Slackware is GNU/Linux, and most of the rest are transforming into something called systemd/Linux. Sure, that's a bit of an overblown gripe now, but with systemd busy replacing every userspace service, it probably won't be for long.
Right now if you run systemd it might only change your init system, but it's going to replace a lot more, and odds are, given its history, that a lot of that won't be optional.
novak
(Score: 0) by Anonymous Coward on Monday December 22 2014, @02:18AM
Since when was "GNU" a synonym for "SysV UNIX"?
(Score: 2) by novak on Monday December 22 2014, @02:36AM
Never. The same amount of time that systemd was only an init system.
novak
(Score: 2) by jackb_guppy on Sunday December 21 2014, @06:05PM
At this point Slackware is not "Linux". Linux is kernel. "Linux" now is Debian and Redhat with systemd and the trash it brings.
"Linux" is not longer an inclusive set of functions that make system. There was a day, that "Linux", I could use any old piece of equipment and it would work and run. Windows keep dropping support and abandoning old equipment. Now "Linux" with g++/gcc can no longer be ran on a 128MB machine. It does not use the swap space - IT MUST BE FULLY IN MEMORY to work. How stupid!
And about PID 1, another poor stupid design. I can PID 1 as an old design from year days, but today, PID should be any random value, instead of lazy programming trick. Future with any PID allows and multiple entres on multiple PIDs, so systemd is not another parallel "OS/Kernel" with multiple modules directly linked to itself.
Where is the elegance in program design? Plans for security? The benefits of "Linux" of old?
Damn old MS is everywhere! Now we call it Redhat and Debian.
(Score: 2) by Arik on Sunday December 21 2014, @06:43PM
If laughter is the best medicine, who are the best doctors?
(Score: 4, Funny) by FatPhil on Sunday December 21 2014, @11:35PM
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 0) by Anonymous Coward on Monday December 22 2014, @11:12AM
And Linus, the fuck, is totally onboard.
FUCK him. He betrayed us.
(Score: 5, Insightful) by zocalo on Sunday December 21 2014, @06:14PM
I think that's the difference and crux of the problem right there. The systemd team have demonstrated more than once that they regard fixing bugs in their code as either someone else's problem or not as important as adding the next great feature. That's not a problem unique to systemd (far from it), but with systemd it is in one of the few places where you simply can't afford to have that kind of attitude - another being the kernel itself. It's an approach leads to code that has more than a passing resemblance to a house of cards, and when that particular house of cards is also the foundation upon which other projects sit things you are risking the whole lot coming crashing down.
UNIX? They're not even circumcised! Savages!
(Score: 0) by Anonymous Coward on Sunday December 21 2014, @06:30PM
I don't get your signature. What does UNIX have to do with circumcision?
(Score: 2) by DECbot on Sunday December 21 2014, @08:04PM
Unix sounds like eunuchs.
cats~$ sudo chown -R us /home/base
(Score: 0) by Anonymous Coward on Sunday December 21 2014, @11:56PM
OK... But eunuchs are usually circumcised, not just at the foreskin, but the scrotum and testes, too. So it still doesn't make any sense.
(Score: 2) by khedoros on Monday December 22 2014, @12:42AM
(Score: -1, Troll) by Anonymous Coward on Monday December 22 2014, @01:00AM
Most puns that I've ever seen have not been as dumb and contradictory as that particular one.
(Score: 0) by Anonymous Coward on Monday December 22 2014, @02:02AM
"UNIX?"
"Eunuchs? No, they are not even circumsised. They are savages!"
(Score: 0) by Anonymous Coward on Monday December 22 2014, @03:04AM
Thank you! That one finally makes sense. He should replace his signature with what you just wrote.
(Score: 0) by Anonymous Coward on Tuesday December 23 2014, @07:04AM
Circumcision is a biblical metaphor. Colossians 2:11 "...in putting off the body of the sins of the flesh by the circumcision of Christ..." The GP speaks of UNIX as a religion.
(Score: 2) by lajos on Monday December 22 2014, @02:14PM
I think that's the difference and crux of the problem right there. The systemd team have demonstrated more than once that they regard fixing bugs in their code as either someone else's problem or not as important as adding the next great feature.
No, that's not the crux of the problem.
Nobody is stooping you from checking out the repo, firing up your fav text editor, fixing bugs and submitting patches.
The real problem is people bitching about open source bugs without putting their actions where their mouth is.
(Score: 2, Interesting) by Anonymous Coward on Sunday December 21 2014, @06:53PM
Guilty as charged. I have no interest in systemd, so when I encountered numerous bugs, I just purged systemd from my systems, and figured the folks pushing this crap down everyone's throats will eventually discover them on their own.
The worst of the bugs was that an fsck that required manual intervention was impossible without either assigning the disk to another system, or exploiting bugs in systemd's consoled/logind.
I run some systems ext4 without a journal. Ext4 without a journal is very fast, and probably helps ssd longevity a bit too. One of these systems crashed (hardware), and when bringing it back up, needed to do an fsck. fsck -a failed, so it dropped to a single user shell-- all good so far. But, this system had migrated to systemd when Debian decided to make it default (this is a dev box running Jessie). I gave the root pword. Tried to type a command, but no echoing. and after hitting enter, got "bad command", then a password prompt again. I type reset. I got very lucky it turns out. nearly everything I typed had the first character eaten. So, I typed lls, and got an ls. I typed llls at the pword prompt, and also got an ls listing. Hmmmm... systemd executes stuff typed at the password prompt as root! Playing some more, since I had got some garbage echoed back, and not just the characters I had typed. I tried lls a bunch of times, and even got the cleartext root password echoed back to an lls with some garbage pre-pended. Wow! So, eventually I ran ffsck /dev/vg_xxx/root and fixed the system. Then purged systemd, and installed sysvinit-core again.
Another of the unintentionally migrated to systemd jessie boxes also went down for the same reason. It ran a journal on its rootfs, but I wanted to make sure things were ok, so ran "shutdown -rF now" and got back an error. Turns out systemd breaks the ability to schedule an fsck on reboot!
Systemd is not ready for prime time-- I found several, what I would call release critical bugs, just interacting with it for a few minutes-- I wonder if the devs do any QA at all. I purged systemd and pinned to -1 on all (hundreds) of systems I manage, and never looked back.
(Score: 2) by HiThere on Sunday December 21 2014, @08:03PM
Maybe, but Fedora21 was NOT ready for release. I tried to install it, got as far as disk partitioning, and it never found a single one of my disks. So I installed Debian testing on that drive, and it didn't hiccup, not only finding the disk I intended to install it on, but three others as well. (Two were usb disks, and one was my main system disk.)
I really expected better of RedHat than this. This isn't even up to beta quality.
Javascript is what you use to allow unknown third parties to run software you have no idea about on your computer.
(Score: 0) by Anonymous Coward on Monday December 22 2014, @11:10AM
Linux is full of feminists and faggots these days. They have taken over. Linus, with his fat ugly wife (was fat and ugly when he married her) approves (and loves systemd). Fuck him. Fuck his system.
(Score: 0) by Anonymous Coward on Monday December 22 2014, @05:19PM
He IS a Finn. I assumed he was looking for the rest of the shark (or whale, as it may be.)
(Score: 0) by Anonymous Coward on Sunday December 21 2014, @05:07PM
I just can't understand why Debian is still going forward with systemd. How many more of these idiotic, unacceptable problems will there need to be before it's removed?
At this point, I think it'd be perfectly appropriate to postpone Jessie until systemd can be removed. Get rid of it, and start the Jessie stabilization process from scratch. If that means the release is delayed for months or years, so be it. It's just the right thing to do.
(Score: 4, Insightful) by Justin Case on Sunday December 21 2014, @05:58PM
> I just can't understand why Debian is still going forward with systemd. How many more of these idiotic, unacceptable problems will there need to be before it's removed?
Perhaps you are misunderstanding their motives. You think "If I were doing this, I would want to produce a quality product." and so you assume they think the same. In your world, once they finally understand the error of their ways, they would repent and sin no more, even if it meant throwing out years of wasted (if shoddy) work.
Imagine someone whose internal thoughts instead go like this: "Sure there are problems here and there, but I'm going to plow ahead anyway." What is that person's motivation?
Now you start to understand what we're up against.
(Score: 2) by jimshatt on Monday December 22 2014, @12:22AM
(Score: 3, Interesting) by novak on Sunday December 21 2014, @07:16PM
While you're correct and they do need to put the brakes on hard, and release "when it is ready," I think it's pretty obvious that's not going to happen. Gnome3 and Pulseaudio both skated happily into the last release as defaults, and those are pretty broken, hated pieces of software. Systemd is just the formalization of their commitment to mainstream over quality.
novak
(Score: 3, Funny) by digitalaudiorock on Sunday December 21 2014, @07:55PM
...not more than releasing "when it is needed" which would be "never"...
(Score: 2) by novak on Sunday December 21 2014, @08:11PM
Yeah, by "when it is ready" I meant Debian 8, not systemd. I can agree that systemd will never be needed and probably never high quality enough to deserve to be in a real linux distro.
My point is that Debian has acting questionably for years, and now they are clarifying that they do indeed intend to adopt whatever comes down the pike instead of slashing anything that fails to meet their standards.
Hopefully enough people who remember the old debian will support devuan and they'll get some traction.
novak
(Score: 0) by Anonymous Coward on Monday December 22 2014, @11:15AM
The debian faggots should be beaten. They are SJW fucks who ruined something good. They infiltrated and took over.
(Score: 2) by sjames on Monday December 22 2014, @04:16AM
I would at least like to see them add an option to the installer to choose init systems. Currently, I can install Jessie with sysV but only by using expert mode and opening a shell to switch init packages in the target.
(Score: 0) by Anonymous Coward on Sunday December 21 2014, @07:43PM
I've been recommending Debian to people for years, for many reasons.
I've gone into shops and replaced dozens of RH/Cent boxes with Debian machines, which work so much better.
I've spent a career making fun of the *BSD people.
Now that Debian has turned into Windows+, I'm forced to go back to these *BSD people and admit they were right.
Thanks a lot, Debian.
(Score: 0) by Anonymous Coward on Monday December 22 2014, @12:02AM
You're not wrong. Debian is still the best there is. By Debian, though, I don't mean Jessie. Jessie isn't Debian in any sense. It's just not true Debian, because it doesn't follow the philosophy of Debian. So I consider it to be a fork of Debian.
(Score: 0) by Anonymous Coward on Monday December 22 2014, @12:54AM
I bet Jessie ain't Scottish either!
(Score: 0) by Anonymous Coward on Monday December 22 2014, @11:07AM
Debian has been taken over by SJWs.
All conservatives have been kicked out.
SJWs like change.
(Score: 0) by Anonymous Coward on Sunday December 21 2014, @05:10PM
I'm grateful that SN keeps posting these systemd stories. They've helped me avoid a lot of problems so far. I was going to upgrade my Fedora 20 system later today oddly enough, but I won't risk it now that I know about this problem. I don't read about these systemd problems at other news sites, so I'm relying on SN to keep me informed. Please keep putting these stories on the front page! They keep saving my ass!
(Score: 0, Troll) by Ethanol-fueled on Sunday December 21 2014, @06:08PM
From the summary:
Well boo-hoo-goddamn-fucking-doo! Is that really a SERIOUS PROBLEM?!
(Score: 2) by Tork on Sunday December 21 2014, @06:12PM
🏳️🌈 Proud Ally 🏳️🌈
(Score: 0, Funny) by Anonymous Coward on Sunday December 21 2014, @06:20PM
Are you saying that log files are for faggots?
A big, manly system administrator like yourself just feels the problem deep in his scrotum, right?
Your fat, throbbing ballsack just knows what the problem is, huh?
What is this, penile intuition or something?
(Score: 3, Insightful) by digitalaudiorock on Sunday December 21 2014, @07:35PM
I'd say it certainly is when you're the one trying to diagnose the problem and aren't able to in the manner you've been able to for decades.
More importantly, the whole issue is in the name of this insane binary log design that's never been more than a cure for which there is no known disease. Every time I bring this up of course, someone chimes in with arguments to the contrary, and they always amount to just more and more and more of these imaginary shortcomings of text logs. Pure fucking insanity...all of it.
(Score: 0) by Anonymous Coward on Monday December 22 2014, @01:41PM
It is depressing me. I will have to make a choice soon. Debian or FreeBSD. This is my pc. I really prefer Debian on my desktop. Perhaps it is time to see how far FB had come since 4.5
(Score: 0) by Anonymous Coward on Sunday December 21 2014, @05:13PM
A bug that affects even just one user is as important as a bug that affects thousands of users, or even millions of users. It's easy to say, "It's one user!" when you aren't the one affected, but your attitude will quickly change when you are the person suffering from the bug's effects!
You won't be saying it's a "rare fault" when it's your Fedora system that no longer works because systemd crashed during the upgrade, ruining the system!
Rare bugs aren't less important. They're just as important as any other bug. And they should be fixed quickly, not years later, or even never!
(Score: 4, Insightful) by Arik on Sunday December 21 2014, @05:21PM
Now hold on, hear me out.
The thing is, the two problems that he mentions are not just bugs. They are bugs, but they are not JUST bugs. If they were just bugs, it would be a minor annoyance.
What makes them critical show-stoppers, not just minor annoyances, is the flawed systemd *design*. Most programs can throw a page fault now and then, and we wont mind. Just as long as it's not PID1. Better to put something with a simple, robust design in PID 1 than to bother fixing the bug at issue.
And the second - the poor log interface. Again, on a sane system that would not be any more than a minor annoyance. So the log viewer sucks, what did you expect? Close it and open the logs with your choice of text editor, doh. It's the bad design choice of using a binary log file that makes this minor annoyance into a show-stopper bug.
So, fix the *problems* yes, but don't fix these particular *bugs*. Because as long as you stick with this fundamentally wrong-headed design, minor bugs will continue to turn into show-stoppers on a regular basis.
If laughter is the best medicine, who are the best doctors?
(Score: 0) by Anonymous Coward on Sunday December 21 2014, @05:29PM
So these bugs are just symptoms of the greater bug, which is that systemd is fundamentally broken and needs to be discarded?
I can agree with that.
(Score: 4, Interesting) by VLM on Sunday December 21 2014, @05:40PM
So these bugs are just symptoms of the greater bug, which is that systemd is fundamentally broken
Yes AC you are basically correct although a little vague. The fundamental brokenness is in the design. If they scrap this batch of code and start over, they'll end up with another steaming pile equivalent to this one, because (follow the money) their core philosophy is wrong. Nobody wants to turn linux into a gnome desktop bootloader except people paid to do exactly that. Nobody wants to bring windows software development architecture (weakly interacting massive programs, inner platform effect taken to an extreme, who cares about security or debugging all that matters is feature-lists, etc) into linux except people paid to do it. The recent success is due to product tying, so the only hope of linux survival is simply to drop gnome, or whatever else gets product-tied in.
I escaped and get to laugh from the freebsd equipped sidelines so I don't have a dog in the fight anymore, although the savage destruction is sad to behold.
(Score: 1) by Nuke on Monday December 22 2014, @12:10AM
LoL, you've ruined my keyboard! That puts it in a nutshell.
(Score: 0) by Anonymous Coward on Sunday December 21 2014, @06:21PM
> Rare bugs aren't less important. They're just as important as any other bug.
That is only true in a world in which the developers have infinite resources.
Since they don't, it ain't.
(Score: 0) by Anonymous Coward on Sunday December 21 2014, @06:28PM
Right, so it makes sense that the systemd bugs mentioned recently haven't been fixed immediately. But why are some of them taking years to be fixed?
If Red Hat and whoever else is pushing this systemd crap doesn't have the resources to do the job properly, then they just couldn't do it at all.
It's better for us not to have systemd at all, then it is for us to have our systems trashed repeatedly by systemd bugs like these.
(Score: 0) by Anonymous Coward on Sunday December 21 2014, @06:44PM
Are you flat-out stupid? Developer time (like any time) is limited, so you need to prioritize what bugs you spend time fixing, and the bugs that affect more people are the biggest bang for your back (or maximum utility, if you prefer).
(Score: 2, Insightful) by Anonymous Coward on Monday December 22 2014, @12:05AM
You're the one who's fucking stupid, obviously. THIS IS A GODDAMN PID 1 BUG! Outside of a kernel bug, IT'S AS SERIOUS OF A FLAW AS YOU CAN GET! Don't you know a goddamn fucking thing about how Linux works? Jesus Fucking Christ! JESUS FUCKING CHRIST! This bug is not excusable, you sick sack of donkey shit. THE LINUX SYSTEM WON'T EVEN BOOT BECAUSE OF THIS BUG, BECAUSE SYSTEMD IS CRASHING YOU FILTHY SON OF A BITCH! What the fuck is wrong with you? Seriously!
(Score: 3, Insightful) by digitalaudiorock on Sunday December 21 2014, @07:43PM
Of course. What if it's just a very rare race condition or the like? They might be "rare" only because they're in fact a timebomb waiting to take down your mission-critical server.
(Score: 2) by DrMag on Sunday December 21 2014, @06:30PM
Both of these behaviors contribute to making the tool much less usable, especially in critical situations where time and efficiency are of the essence.
I'm sorry, but I'm having a difficult time seeing how poorly formatted logs become serious, even in a supposed "critical situation". It's an inconvenience, and you may lose a little more time scrolling through everything to find what you're looking for, but it's hardly the disaster being connoted.
I'm personally undecided about the whole systemd issue. There are things about it that I actually kind of like, and the disadvantages I've heard about so far don't really affect me. I've certainly not had the trouble that people are supposed to be having. I think part of the problem, in addition to the editor's comment about bug reports being a rarity, is that so often the people who complain loudest about systemd have the attitude of, "OMG! Dis iz da worst ting ever!! It must be fixed naaaooo!!!!!1!!1!" At least it seems that way to me sometimes. It'd be refreshing to sit down and discuss seriously the pros and cons, and figure out how to improve, or if necessary, replace, systemd. And it would be done in a way where someone's thoughts aren't completely written off simply because they think there is a benefit (or disadvantage) to the software as it stands now. There does seem to be some serious resistance from some of the key decision makers in addressing its issues, though, and that is a problem too.
(Score: 0) by Anonymous Coward on Sunday December 21 2014, @06:37PM
Of course you wouldn't understand these problems. You're obviously not a system administrator.
No, the home user dicking around with Linux in a VM on his MacBook Pro won't notice these problems. But that person isn't really using Linux seriously, either.
But when you're managing systems that do serious work, often affecting a business' ability to operate and profit, you need log files that work so you can diagnose problems quickly and easily.
Every sideways scroll could waste time, resulting in thousands of dollars being lost each second.
Welcome to the real world, bud. We don't have time for your shitty binary log files fucking up the businesses we're trying to run.
(Score: 2) by DrMag on Sunday December 21 2014, @06:59PM
No need to be condescending and presumptuous.
No, I'm not a sysadmin by trade, but I've learned a good bit about it for managing my own linux server that I maintain. I do use linux seriously, and suggesting that the only people who do are full-time sysadmins is ridiculous.
And I would venture to guess that if your business is losing thousands of dollars because you had to scroll to the side for a second, your business is poorly managed to begin with. But perhaps that's me being condescending and presumptuous. =)
(Score: 0) by Anonymous Coward on Sunday December 21 2014, @07:23PM
A sysadmin contracted to solve a problem can be on contract to be payed hundreds of dollars per hour to fix that issue. Time spent scrolling to find a problem using this rather than the tried and tested tools like 'tail' and 'grep' can result in larger support bills.
(Score: 4, Insightful) by maxwell demon on Sunday December 21 2014, @07:28PM
So the tools don't allow to pipe the output to tail or grep? Then that is the real problem, not overly long log lines or too long logs.
The Tao of math: The numbers you can count are not the real numbers.
(Score: 5, Insightful) by Arik on Sunday December 21 2014, @07:31PM
If laughter is the best medicine, who are the best doctors?
(Score: 2) by choose another one on Monday December 22 2014, @11:30AM
Is there something about "ForwardToSyslog=yes" that doesn't work for you - does that not produce log files in text ?
Seems to me that going from binary to text is the right way - it's trivial once you've decided on a format - going the other way (if you need the binary logs with full metadata, which I presume somebody does...) requires ensuring the text formattinging is reversible (quoting etc.) and writing a parser. Seems to me that binary-first is the right design if someone needs binary logs.
(Score: 1) by Arik on Monday December 22 2014, @11:49AM
Logs should be in text and converted to binary later if someone really needs binary, not the other way around, because this allows you to read the logs after a failed boot, and binary does not. You can convert from one to another to your hearts content, after you fix the problem and get the machine back on its feet.
If laughter is the best medicine, who are the best doctors?
(Score: 2) by fnj on Sunday December 21 2014, @07:42PM
Who told you that, or did you just make it up? Tools exist, but if THE DATA ITSELF IS UNWIELDY (e.g., the long lines mentioned), then dealing with the data using any tools, including tail and grep, is going to be unwieldy.
(Score: 2) by maxwell demon on Sunday December 21 2014, @08:03PM
Did you actually read the post I replied to?
The Tao of math: The numbers you can count are not the real numbers.
(Score: 2) by fnj on Sunday December 21 2014, @08:22PM
Yes, actually, I did. What caused you to leap to "So the tools don't allow to pipe the output to tail or grep?" Because journalctl fully allows such piping.
(Score: 2) by maxwell demon on Sunday December 21 2014, @08:42PM
This:
The Tao of math: The numbers you can count are not the real numbers.
(Score: 0) by Anonymous Coward on Sunday December 21 2014, @11:45PM
journalctl -b -u ntpd.service | grep waking
Dec 19 00:51:57 wheatley ntpd[1765]: new interface(s) found: waking up resolver
Dec 19 00:51:59 wheatley ntpd[1765]: new interface(s) found: waking up resolver
Dec 19 17:07:45 wheatley ntpd[1765]: new interface(s) found: waking up resolver
Dec 20 00:19:19 wheatley ntpd[1765]: new interface(s) found: waking up resolver
Dec 20 07:53:49 wheatley ntpd[1765]: new interface(s) found: waking up resolver
Dec 20 07:53:51 wheatley ntpd[1765]: new interface(s) found: waking up resolver
Dec 21 17:49:27 wheatley ntpd[1765]: new interface(s) found: waking up resolver
I don't see the problem, here.
(Score: 0) by Anonymous Coward on Monday December 22 2014, @12:07AM
let me correct that for you.
journalctl -b -u ntpd.service | grep waking
journalctl: log corrupted.
because they refuse to fix the automatic log corruption if you turn on log rotation.
(Score: 2) by tempest on Monday December 22 2014, @12:11AM
Wow... Fuck.
(Score: 3, Insightful) by Anonymous Coward on Monday December 22 2014, @12:36AM
I'll highlight the problems for you:
The first problem I see is that you need to use journalctl. It should never be a requirement that a log file be filtered through some program before grep can work with it.
The second problem I see is that some of the lines are out of order. I don't know if the timestamps are wrong, or if they were logged in the wrong order, or if journalctl screwed them up, but clearly some are where they don't belong.
The third problem I see is the random binary data at the end of some of the lines. That probably should not be there as far as I can tell.
(Score: 2) by choose another one on Sunday December 21 2014, @09:58PM
> Of course you wouldn't understand these problems. You're obviously not a system administrator.
If you want to get into a pissing contest about number of systems managed then I seriously doubt you are going to win against the RedHat cloud admins, unless you admin for Google or Amazon, and for some reason they want systemd and binary logs. No, I don't _know_ why, I don't manage thousands of servers either, but i can take some good guesses.
Furthermore, binary logs can also be trivially turned into text (oh look, systemd even provides a configuration to do that) - the converse is not true.
> No, the home user dicking around with Linux in a VM on his MacBook Pro won't notice these problems. But that person isn't really using Linux seriously, either.
I guess the sysadmin managing a few tens of on-premise or colo-ed servers won't notice the kind of problems that the RedHat guys managing multiple data centres either, maybe that's because they aren't using Linux seriously ?
(Score: 4, Insightful) by Arik on Sunday December 21 2014, @11:11PM
Oh, look, that assumes that you booted correctly.
This is a system designed to fail just when it is needed most. When the system is working fine, you can export the logs that show it working fine, but when it fails and you really need to see those logs, you cannot.
If laughter is the best medicine, who are the best doctors?
(Score: 2) by darkfeline on Sunday December 21 2014, @10:09PM
Going from `cat foobar | tail` to `journalctl foobar | fold | tail` (Yes, useless cat, I know)
Wow, that was hard! Damn that systemd making everything harder!
Join the SDF Public Access UNIX System today!
(Score: 0) by Anonymous Coward on Monday December 22 2014, @12:12AM
Now try it when the log file is partially corrupted due to a bad disk. If you aren't a dumbass, you can still make out some of the log entries when things are done the proper way and text files are used. Now try doing that with journalctl. Oh, fuck you, there's no output at all! Fold and tail all you want, it's not going to do a fucking ounce of good. You'll sit there scratching your head, wondering where the hell the log entries are.
(Score: 1) by Anonymous Coward on Monday December 22 2014, @12:42AM
No disk problems needed. Just turn on log rotation and systemD corrupts the log's it's self 1/4 of the time. It's a known bug that they marked as 'won't fix'
(Score: 0) by Anonymous Coward on Monday December 22 2014, @12:57AM
LOL! Do you have the bug number for this?
(Score: 2) by cafebabe on Thursday December 25 2014, @08:29AM
I've been in a situation where downtime cost more than US$1 per second. Specifically, on rotation, I was solo 24 hour support for a 4,000 core renderfarm. I believe that RenderMan licences were US$3,000 per processor pair. So, that's US$6 million per year. I also believe that average salary for 500 people was about US$50,000. So that's about US$25 million per year. There was also nine other applications licenced for the renderfarm, electricity, hardware deprecation (not restricted to 3,000 harddisks) and business taxes.
In these circumstances, being deprived of grep and/or tail -f is going to cost money. In more serious circumstances, it could foreseeably cost lives.
1702845791×2
(Score: 3, Interesting) by digitalaudiorock on Sunday December 21 2014, @08:02PM
There have been several posts here about the relative importance of bugs...some I've already replied to.
It almost seems as though people are forgetting this this is (or is supposed to be) first and foremost, and init system. What bugs aren't important in an init system? Are "rare" bugs (translated: bugs of frighteningly unknown origin) somehow not important?? Maybe the solution is...OH I don't know...a simple init system, doing what an init system is intended to do, and doing it well?
(Score: 2) by VLM on Sunday December 21 2014, @09:16PM
Maybe the solution is...OH I don't know...a simple init system, doing what an init system is intended to do, and doing it well?
Some of the PR about boot times is weird.
So the problem is that sysv init takes 50 million machine cycles to boot and a massive reimplementation of everything will "solve all our boot time problems" at the mere cost of zillions of sysadmin hours cleaning up the bugs in a new system. Unfortunately 50M machine cycles was "kinda slow" on my first 40 MHz 386 in '93 taking a bit over a second, but on a 3 ghz machine its best measured in microseconds, so a complete rewrite to make things faster seems kinda a waste of everyone's time. What with the conversion from spinning hds to ssds, and images connected to giant NAS machines, the boot time aspect just doesn't matter anymore. You could run sysvinit ten times and it would still be faster than sysvinit a decade ago because "hardware" so everything you did a decade ago will work now.
By analogy you could create the worlds fastest "hello_world.c" as an exercise, but once its faster than your eyes can see it run, who cares? Or video game rendering framerates (far) above 60 hz when the physical LCD hardware can't update faster than 60 hz or so anyway, just no point.
(Score: 4, Interesting) by fnj on Sunday December 21 2014, @08:08PM
FTFS:
Not a systemd advocate here, but that is a question of familiarity with the tools, which is to say willingness to learn. The author whose article is being summarized, did not try very hard. In fact, I would say he is not much of a Unix user in spirit. Try "man journalctl" sometime. You will see stuff like:
"-b [ID][±offset], --boot=[ID][±offset] Show messages from a specific boot. This will add a match for "_BOOT_ID="."
How do you do that when not using systemd? Hmmm?
"-- no-full The second problem with systemd that he describes involves the journalctl utility. It displays log messages with long lines in a way that requires sideways scrolling, as well as displaying all messages since the beginning of time, in forward chronological order. Both of these behaviors contribute to making the tool much less usable, especially in critical situations where time and efficiency are of the essence."
That is more control than you get with "less /var/log/messages".
"-n, --lines Show the most recent journal events and limit the number of events shown. If --follow is used, this option is implied. The argument is a positive integer or "all" to disable line limiting. The default value is 10 if no argument is given."
That gives you recent lines at least as easily as using tail, or feeding options to less, does.
"-r, --reverse Reverse output so that the newest entries are displayed first."
I'll leave that as an exercise for you to figure out when not using journalctl - have a ball.
"--since=, --until= Start showing entries on or newer than the specified date[/time], or on or older than the specified date[/time], respectively."
I'd say that is a lot more straightforward than doing similar selections without using journalctl.
There is a lot more; journalctl has an incredibly rich set of options, and you can pipe it to use your favorite traditional tools too. The claim that journalctl "displays all messages since the beginning of time" is just ignorant.
So in summary, out of fairness, I am calling utter bullshit.
(Score: 0) by Anonymous Coward on Sunday December 21 2014, @08:38PM
> So in summary, out of fairness, I am calling utter bullshit.
It really doesn't help the anti-systemd people to be spewing their low-information complaints. I'm sure there are significant problems with systemd, but every PEBKAC and ultra-low impact bug that gets its own damn story here ends up making me doubt the competence of the complainers Crying wolf ain't helping their cause.
(Score: 2) by fnj on Sunday December 21 2014, @08:43PM
Couldn't agree more. I'll just point out, it is only the less important part of the article which I was focusing on. The most important point is highly valid.
(Score: 0) by Anonymous Coward on Sunday December 21 2014, @09:17PM
I agree, and at this point I'm not taking anything said pro- or anti-systemd seriously. This long ago passed through its brief phase as a discussion on technological merit to some sort of socio-political inter-faction conflict. In that kind of environment, any figure or argument may very well be a sort of strawman, or a fake strawman set up to expose strawmanning by the other side, etc. etc.
The anti-systemd spamming AC who is over-fond of the carriage return, for example, doesn't make any sense. Those comments ask me to believe that there exists a person who is so terrified of his own system that he reads SN specifically to hear about problems with systemd, yet that this person is unwilling or unable to 1) switch distros or 2) read through the systemd bug report list directly, where each of these stories ends up linking, since none of these are new bugs! That is completely irrational. However, it provides a nice way to build up examples of those who are against systemd as deranged. It would be quite effective for systemd proponents to simply link to each of these comments in order to show their own comparative rationality. ...unless, of course, certain evidence will at some point appear indicating that these comments are actually made by someone who has some vested interest in the success of systemd, which would be conveniently effective as for the anti- folks.
In short: none of this makes sense on the surface, and to attempt to look beneath the surface is a Sisyphean effort.
(Score: 0) by Anonymous Coward on Monday December 22 2014, @12:24AM
I don't think that that AC is lying.
I, too, do find that I'm constantly surprised by the bugs in systemd. Not being able to cancel a boot-time fsck run? I never would have expected that to be broken under systemd. Pid 1 crashing? I could see it happening, but I would have thought that systemd would be much more reliable than that. Binary log files? I thought everybody with a brain figured out that was a bad idea just by thinking about it for a few seconds. Poor formatting of the text version of the binary log files? This is something that even an amateur programmer can do properly.
I'm also surprised that these bugs haven't been fixed, even after years in some cases. These are major bugs. They're not tolerable at all. The mere fact that they exist means that something is very wrong with systemd's architecture and development practices.
I don't really have to worry about all of this, though, having moved to FreeBSD some time ago. But not everyone has that luxury, so I can see why they need this site to keep them updated with the latest dumbness that has been discovered with systemd.
(Score: 3, Interesting) by novak on Sunday December 21 2014, @09:25PM
Hm, yeah, that's because there's (in my mind, anyhow) two categories of systemd opposers.
First, you've got the people actually interested in minimalism. They have, over time, probably been moving to more minimal systems, been building or testing software which replaces bloated parts of userspace, and generally simplifying their computing. They do not plan to use systemd, because it's the opposite of what they have been working towards. So they don't. They either use another distro or remove systemd from their favorite, because it comes with the territory of minimalism that you have to know some system internals. Or they jump to BSD because they realize that a lot of BSD code is cleaner and more minimal anyways.
Then, you've got complainers. People that can barely pipe two commands together but that worship the mythical "unix philosophy," which, to be fair, linux does not adhere to particularly well anywhere. They have no particular allegiance to any software, and no particular goals for their software either, but because someone told them it was bad, they start screaming bloody murder. So they want their ubuntu or fedora, just without systemd, like that was the only problem with modern linuxes. They think that systemd is going to crash all their computer multiple times daily and delete the logs. (to be fair, it probably will tank reliability... But there's too much momentum behind it to leave it that badly off)
These complainers drown out all the serious commenters with knowledge of actual problems of systemd, who are just sort of sitting back chuckling that anyone would even consider using systemd at all. I... don't really care about this bug, honestly. I'll never use systemd, but I know there are going to be a ton of bugs, as systemd is all about feature creep. But instead, most of what you hear are wild rants from the uninformed. "A bug! A bug! I told you it was broken!"
This bug doesn't "prove" that systemd is broken/unreliable. Basic knowledge about its design proves that. This bug is just a bug, which happens to all software, and especially badly-designed bloatware like systemd.
novak
(Score: 2, Insightful) by Nuke on Monday December 22 2014, @12:25AM
You have given me a flashback to the Monty Python Spanish Inquisition sketch. That's three categories.
How about a fourth category - those worried abot the unwillingness of the sytsemd team even to acknowledge bugs, like here [linuxveda.com] and here [soylentnews.org].
(Score: 2) by novak on Monday December 22 2014, @12:39AM
Bet you didn't expect the Spanish inquisition!
More seriously, the people interested in minimalism could understand how systemd works and dismiss it due to design. Still just two categories.
novak
(Score: 0) by Anonymous Coward on Monday December 22 2014, @12:14AM
I can't wait until systemd shits out on you, and you find yourself dealing with several of these alleged "low-information complaints" at the same time, and unable to do a damn thing to fix them.
I hope it costs you a lot of money, or maybe even your reputation.
(Score: 0) by Anonymous Coward on Monday December 22 2014, @01:59AM
Wishing ill upon somebody just because they don't disagree with your viewpoint is spiteful and narrow-minded. Regardless of how you feel on the subject, you are way out of line here.
I hope you are just trolling, but if you really are that emotionally invested in a debate over a peace of software, then you need some help, because this is not a healthy mindset to have.
(Score: 3, Insightful) by Foobar Bazbot on Monday December 22 2014, @06:52AM
Y'know, there's actually a pretty serious problem here, but I completely agree that it's not the problem the complainer thinks it is. Even if you can't be bothered to RTFM, when confronted with a utility outputting to a tty through a weirdly-behaving pager, the most obvious thing to try is not giving it a tty, e.g. journalctl | less, because any sane program checks if it's connected to a tty before doing weird tty stuff. Sure enough, that works -- journalctl dumps plain text to stdout, and less folds or chops long lines exactly as you told it to in your LESS environment variable.
No, the real problem is that journalctl (and other parts of the systemd ecosystem) takes it upon itself to provide "sane" defaults, and to provide a way to override those defaults -- all completely ignoring and overriding the existing configuration mechanism. In this case, it's journalctl feeding less a fistful of options to override the settings in your LESS environment variable (but you can override the override by setting SYSTEMD_LESS), but much the same thing is the cause of the Magic SysRq story.
And every time this happens, and someone complains, the systemd developers/enablers just can't see what their problem is -- after all, you can configure what setting systemd overrides the existing configuration with, and that should be enough for anyone!
(Score: 3, Informative) by hemocyanin on Monday December 22 2014, @08:37PM
Are you saying that showing a text file with line order reversed is hard?
A handy reference is the sed one liners document. Note, it's ancient, probably first reproduced in cuneiform on clay tablets, but even with sed, some of the regular expression examples are useful:
http://sed.sourceforge.net/sed1line.txt [sourceforge.net]
Or there's "tac" as referenced in the example: http://linux.die.net/man/1/tac [die.net]
(Score: 2) by jelizondo on Monday December 22 2014, @03:53AM
I been wondering, why all major (almost all, anyway) Linux distros changed to systemd?
I'll give you my conspiracy theory: some key people in there are being funded my Microsoft!
When your upgrade is as iffy as a Windows upgrade, you have to concede that someone from *that* camp has crossed over and messed things up.
Maybe it is non-sense, but I really wonder why stable distros have choosen uncertainty and chaos when the bedrock foundation of Linux was robustness; not ease of use or click-next to install, but rock solid performance year after year. Why?
If someone knows why, please share with the unwashed...
(Score: 0) by Anonymous Coward on Monday December 22 2014, @11:18AM
They should be found and beaten. BEATEN.