Stories
Slash Boxes
Comments

SoylentNews is people

posted by martyb on Sunday April 19 2015, @06:02PM   Printer-friendly
from the with-all-this-logging-we-need-a-lumberjack dept.

World-renowned Unix master Chris Siebenmann has written an article entitled 'I wish systemd would get over its thing about syslog'. It addresses the strained relationship between the systemd init system and the traditional syslog approach to logging used on many Linux systems.

Chris writes:

Anyone who works with systemd soon comes to realize that systemd just doesn't like syslog very much. In fact systemd is so unhappy with syslog that it invented its own logging mechanism (in the form of journald). This is not news. What people who don't have to look deeply into the situation often don't realize is that systemd's dislike is sufficiently deep that systemd just doesn't interact very well with syslog.

This is a must-read article for anyone who needs to use systemd and syslog together.

This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 5, Insightful) by Anonymous Coward on Sunday April 19 2015, @06:11PM

    by Anonymous Coward on Sunday April 19 2015, @06:11PM (#172891)

    This author is far too kind to systemd, and he's very naïve if he believes this is not on purpose. It's consistent enough to be policy. Anything that works properly must be replaced by broken Poettering-ware.

    Forget systemd. Use slackware, or gentoo, or *bsd.

    And short redhat if you can.

    • (Score: 5, Insightful) by Jeremiah Cornelius on Sunday April 19 2015, @07:41PM

      by Jeremiah Cornelius (2785) on Sunday April 19 2015, @07:41PM (#172907) Journal

      “Once is happenstance. Twice is coincidence. Three times is enemy action.”
      -- Auric Goldfinger

      --
      You're betting on the pantomime horse...
      • (Score: 5, Funny) by FatPhil on Sunday April 19 2015, @10:06PM

        by FatPhil (863) <pc-soylentNO@SPAMasdf.fi> on Sunday April 19 2015, @10:06PM (#172943) Homepage
        3?

        http://narf-archive.com/pix/bd0fb252416206158627fb0b1bff9b4779dca13f.gif
        --
        Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
      • (Score: 0, Funny) by Anonymous Coward on Sunday April 19 2015, @11:47PM

        by Anonymous Coward on Sunday April 19 2015, @11:47PM (#172967)

        No wonder he became a third-rate villain, the guy's understanding of statistics rivals that of a liberal arts major.

    • (Score: 0) by Anonymous Coward on Monday April 20 2015, @01:07PM

      by Anonymous Coward on Monday April 20 2015, @01:07PM (#173111)

      He seems to be accepting of what Systemd is trying to do in general, but is pointing out some rough edges.

      Note that he has shown himself accepting enough that not only the normal white knights of Systemd shows up to tell he is wrong, but Poettering himself makes an appearance. This indicates that he is not expecting to get any real resistance to his claims.

    • (Score: 2, Informative) by mvdwege on Monday April 20 2015, @07:31PM

      by mvdwege (3388) on Monday April 20 2015, @07:31PM (#173249)

      The author is in fact blaming systemd for rsyslog fuckups. Read Lennart's response on the original article.

    • (Score: 0) by Anonymous Coward on Tuesday April 21 2015, @04:14PM

      by Anonymous Coward on Tuesday April 21 2015, @04:14PM (#173572)

      Or you may want to try some other OS, *BSD or an OpenSolaris derivative (OpenIndiana/OmniOS/SmartOS)..
       

  • (Score: 5, Interesting) by RamiK on Sunday April 19 2015, @06:53PM

    by RamiK (1813) on Sunday April 19 2015, @06:53PM (#172897)

    http://utcc.utoronto.ca/~cks/space/blog/linux/SystemdAndSyslog?showcomments#comments [utoronto.ca]

    I personally had very little interaction with syslog but in each case (windows machines, a few routers... Stuff I can't or won't ssh to basically) something broke. Line breaks, carriage returns, tabs, use of punctuation, double spacing all over (not char set... I saw the code. It was introduced manually into the string deliberately) and so on... Essentially each device OEM treated the protocol like it's own in-house dev tool and would keep it that way on release.

    --
    compiling...
    • (Score: 2, Redundant) by frojack on Sunday April 19 2015, @07:32PM

      by frojack (1554) on Sunday April 19 2015, @07:32PM (#172906) Journal

      Agreed. Logging is almost universally abused.

      Logging (regardless of platforms) tend contain massive amounts of stuff that doesn't need to be logged at all, is obtuse, and only meaningful to the coder, using terminology that makes them deliberately unintelligible.

      And any complaining users are instructed to change their system logging level to avoid these messages. Never considering that you may need logging at a specific level for other equally non-compliant software.

      --
      No, you are mistaken. I've always had this sig.
    • (Score: 5, Informative) by Marand on Monday April 20 2015, @02:02AM

      by Marand (1081) on Monday April 20 2015, @02:02AM (#172994) Journal

      TL;DR for anyone that doesn't want to read Poettering's entire comment:

      Poettering's reply, as usual, is "not a bug, someone else's problem" and passes blame to another party. Now, to be fair, it's possible that it's true in this case; I'm not able to verify it either way, so I make no assumption about it. It's just hard to take the comment seriously when that's his and Sievers' response to any sort of criticism.

      Journal corruption issue? RESOLVED NOTABUG. systemd causes machine to not boot if kernel is run with 'debug' option? Fix your kernel not our problem. Etc. I'm sure they're occasionally correct and a problem really isn't their fault, but it seems to be their default method of deflecting criticism. (Sometimes they get creative and go for something like "you just hate handicapped people", though)

      • (Score: 3, Informative) by mvdwege on Monday April 20 2015, @07:34PM

        by mvdwege (3388) on Monday April 20 2015, @07:34PM (#173250)

        Actually, the kernel 'debug' option was hashed out after Linus' first tirade at Kay Sievers. It was in fact admitted by Linux that systemd had correctly interpreted the 'debug' option, but was generating too much output for the kernel log buffer, which was admitted by Lennart to be a bug in systemd and fixed.

        • (Score: 2) by Marand on Monday April 20 2015, @11:28PM

          by Marand (1081) on Monday April 20 2015, @11:28PM (#173322) Journal

          You deserve an Informative mod, but my points haven't refreshed yet, unfortunately. I lost track of the kernel debug discussion somewhere between Sievers claiming no fault and Poettering admitting they actually did do something wrong, so I didn't know one of them admitted error about something. Last thing I saw about it, the kernel devs were discussing ways to suppress notification spam because Sievers was being difficult.

          Sievers' part of that is what was relevant to the point I was making: they always seem to default to "it's not our fault, we did nothing wrong", not just with systemd, but with everything. It's been the same thing for years, all the way back to the creation of, and problems with, pulseaudio. When you cry "wolf" often, nobody will believe you when it's legitimate.

    • (Score: 3, Funny) by tibman on Monday April 20 2015, @01:44PM

      by tibman (134) Subscriber Badge on Monday April 20 2015, @01:44PM (#173120)

      aaaand i just learned that journald (optionally) has a small webserver in it : /

      --
      SN won't survive on lurkers alone. Write comments.
  • (Score: 5, Insightful) by frojack on Sunday April 19 2015, @07:23PM

    by frojack (1554) on Sunday April 19 2015, @07:23PM (#172905) Journal

    Journald is more of a database rather than just a growing text file. Its Structure [freedesktop.org] is less susceptible to log tampering than is syslog. Message numbers and metadata are encoded into the journal, and there's no way you can fake them like some malware in the passed was known to do.

    You can have both. You can use journalctl to have journald echo all or some of the messages out to oldschool log(s).

    Once you get use to some of the command structure, searching and finding things in he journal is actually easier (and fasterO than syslog.

    Journald is simply an implementation of structured logging [gregoryszorc.com], and structured logging was being developed long before systemd came around. Many people who were working on Structured Logging got all butt-hurt over it because Lennart Poettering and Kay Sievers did not work with them. Structured Logging people had been dicking around so slowly that there stuff was never really implemented anywhere and then two arrogant asses came in and cut them off at the knees. Of course they are pissed.

    But as a end user, journald is different, imposes a slight learning curve, but otherwise I find it quite usable.
    Yes, I have run into the barfed up "old messages" upon a reboot. It took me all of 15 seconds to read the time stamp and figure out it was old detritus. Yes I've seem messages be available to users that should not be. (Fixed by my distro in short order).

    But really, the linked article reads like a hissy fit rather than a significant new complaint. He found exactly two things to complain about.

    For the record, I've never had systemd or journald cause me any grief yet.

    I still don't like them because of the relearning they imposed without offering ANY improvement to the average user. Its not faster, its not smaller, it doesn't scale meaningfully, it doesn't offer anything that Joe User needed. It wasn't even meant for Joe User. And besides that the two principals need a bitchslap on a daily basis.

    --
    No, you are mistaken. I've always had this sig.
    • (Score: 5, Interesting) by darkfeline on Sunday April 19 2015, @08:05PM

      by darkfeline (1030) on Sunday April 19 2015, @08:05PM (#172914) Homepage

      You forgot one important fact.

      Just install rsyslog and you can have your plain text logs and eat your systemd cake too. My computer running systemd has both metadata-tagged journald logs AND plain text logs. People who bemoan journald are just looking for a reason to hate systemd when there are other VALID reasons for doing so.

      --
      Join the SDF Public Access UNIX System today!
      • (Score: 2, Informative) by frojack on Sunday April 19 2015, @08:38PM

        by frojack (1554) on Sunday April 19 2015, @08:38PM (#172925) Journal

        Believe I covered that in my second paragraph.

        --
        No, you are mistaken. I've always had this sig.
      • (Score: 2) by zocalo on Sunday April 19 2015, @08:43PM

        by zocalo (302) on Sunday April 19 2015, @08:43PM (#172926)
        Or, if you prefer, you can install your syslog daemon of choice, configure systemd to copy all it's message to that and then simply turn off systemd's journal so you don't get the binary files at all. There's probably a little extra overhead as messages get passed between the Kernel, systemd and syslog, but unless you are generating an insane level of messages at log level debug I doubt it would be noticeable. To do this, in /etc/systemd/journald.conf set:

        Storage=none
        ForwardToSyslog=yes


        Then create a file named /etc/rsyslog.d/sd-socket.conf that contains:

        $AddUnixListenSocket /run/systemd/journal/syslog
        --
        UNIX? They're not even circumcised! Savages!
        • (Score: 5, Insightful) by Anonymous Coward on Sunday April 19 2015, @09:19PM

          by Anonymous Coward on Sunday April 19 2015, @09:19PM (#172937)

          Great! So now, after only a few months, I know how to do a quick-n-dirty little hack that gets me back to only slightly less functionality than before systemd landed.

    • (Score: 2) by aristarchus on Sunday April 19 2015, @08:16PM

      by aristarchus (2645) on Sunday April 19 2015, @08:16PM (#172917) Journal

      For a second there I thought "Oh No!! They've gotten to frojack!" But then I read on and was so relieved.

    • (Score: 5, Insightful) by Whoever on Sunday April 19 2015, @08:45PM

      by Whoever (4524) on Sunday April 19 2015, @08:45PM (#172929) Journal

      Its Structure [freedesktop.org] is less susceptible to log tampering than is syslog.

      If you are worried about log tampering, you should not be relying on local logs at all. Instead, a hardened, dedicated server should collect logs for you.

      • (Score: 0) by Anonymous Coward on Sunday April 19 2015, @09:23PM

        by Anonymous Coward on Sunday April 19 2015, @09:23PM (#172938)

        Log tampering? Why? When you have systemd to own. No need to tamper with the logs.

      • (Score: 2) by tempest on Monday April 20 2015, @02:31PM

        by tempest (3050) on Monday April 20 2015, @02:31PM (#173135)

        Or if you want to be super hardcore, send it to a line printer. I always liked that as an option.

    • (Score: 4, Insightful) by FatPhil on Sunday April 19 2015, @10:17PM

      by FatPhil (863) <pc-soylentNO@SPAMasdf.fi> on Sunday April 19 2015, @10:17PM (#172948) Homepage
      > and there's no way you can fake them like some malware in the passed was known to do

      If you have software on your system that's trying to do that, it's probably no longer your system anyway.
      --
      Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
    • (Score: 4, Informative) by Thexalon on Monday April 20 2015, @01:23AM

      by Thexalon (636) on Monday April 20 2015, @01:23AM (#172987)

      For the record, I've never had systemd or journald cause me any grief yet.

      I have:
      - I have daemons that start up in the wrong order, which means that ordinary users cannot shut down the system. There is not a clear fix, since there's absolutely no debugging information provided.
      - Both startup and shutdown taking significantly longer. Since this was supposed to be the primary benefit of systemd, that's a pretty significant knock.

      Meanwhile, my openrc-based Gentoo system is humming along without difficulty.

      --
      The only thing that stops a bad guy with a compiler is a good guy with a compiler.
      • (Score: 2) by frojack on Monday April 20 2015, @01:39AM

        by frojack (1554) on Monday April 20 2015, @01:39AM (#172990) Journal

        You do know that you can control the order of the process start up, right?

        It probably would take 15 minutes of RTFM to take care of that.

        Like I said, there is a learning curve. And there is little advantage of systemd to joe user. But the problems are solvable, even if annoying.

        --
        No, you are mistaken. I've always had this sig.
        • (Score: 1, Touché) by Anonymous Coward on Monday April 20 2015, @11:35PM

          by Anonymous Coward on Monday April 20 2015, @11:35PM (#173324)

          The point seems to be that while he could change the order, he does not know exactly what to change because journald fails to log the errors.

    • (Score: 5, Touché) by Anonymous Coward on Monday April 20 2015, @02:55AM

      by Anonymous Coward on Monday April 20 2015, @02:55AM (#173001)
      So it fixes problems that were not bothering me, and screws up functionality I rely on.

      Still doesnt sound like a good idea.
    • (Score: 2) by danomac on Monday April 20 2015, @05:10PM

      by danomac (979) on Monday April 20 2015, @05:10PM (#173193)

      For the record, I've never had systemd or journald cause me any grief yet.

      I have, but I only discovered it recently as I rarely reboot my computer.

      I use an Intel fakeraid (via mdadm) due to my dual-boot with Windows. For some reason, `systemctl reboot` and `systemctl shutdown` hang, and on restart cause the IMSM raid to break and rebuild, and there's no indication as to why. During my testing, `systemctl poweroff` works normally.

      I haven't had time to even figure out how to compare the three targets to see if there's any difference between them, nevermind coming up with some sort of solution.

      • (Score: 2) by frojack on Monday April 20 2015, @10:05PM

        by frojack (1554) on Monday April 20 2015, @10:05PM (#173301) Journal

        It seems that all three of those commands go to the same place [freedesktop.org] and systemd replaces itself with /usr/lib/systemd/systemd-shutdown with an argument. I suspect the problem is with that shutdown tool.

        Come to think of it, I've had BTRFS barf up its lunch twice, and moved away from it.
        I don't know what caused this, it was running on a systemd machine, so I guess I can't totally rule it out I suppose.

        But it wasn't obviously systemd as far as I can tell.
        Twice in 6 months was too much for me.

        I tend to suspend my machine rather than shutting down, and this all seems to work, but even shutting down has not caused any problems since I moved away from BTRFS. I might revisit BTRFS in a couple releases.

        I've yet see no advantage to systemd.

        --
        No, you are mistaken. I've always had this sig.
        • (Score: 2) by danomac on Monday April 20 2015, @10:40PM

          by danomac (979) on Monday April 20 2015, @10:40PM (#173309)
          Well, I think I'll change kdm's reboot & shutdown commands to `systemctl poweroff` for now.

          I also noticed that shutdowns take a lot longer - about 3 minutes compared to openrc's 20 seconds. systemd seems to get hung up shutting down apcupsd (even though apcupsd indicates it shut down successfully in journalctl) systemd doesn't seem to notice.

          When I was using openrc the IMSM raid array would corrupt at every reboot.

          My laptop has no issues with systemd. Just my main desktop.
  • (Score: 1) by ghost on Monday April 20 2015, @01:58PM

    by ghost (4467) on Monday April 20 2015, @01:58PM (#173124) Journal
    systemd is the emacs vs vi of soylent.

    Though personally, systemd is the new 7 Up -- never had it, never will.

    • (Score: 0) by Anonymous Coward on Thursday April 23 2015, @09:26AM

      by Anonymous Coward on Thursday April 23 2015, @09:26AM (#174235)

      odds say you already have if you use linux for a reasonable amount of tasks

  • (Score: 1, Informative) by Anonymous Coward on Monday April 20 2015, @07:55PM

    by Anonymous Coward on Monday April 20 2015, @07:55PM (#173255)

    Get the bit in Lennarts comment he rules that:

    1) syslog/udp didn't age well (What the fuck does that even mean in this context?)
    2) reccomends http/json (YES, you cannot make this stuff up)

    It's clear Lennart is as complete and utter fucking cunt who has no fucking idea about unix or linux in general.

    Suggesting syslog/udp is dead and should be replaced with json and http?

    The fuck?

    You fuckwit distro maintainers need to get his systemd out of your OSes NOW!

    • (Score: 0) by Anonymous Coward on Monday April 20 2015, @11:42PM

      by Anonymous Coward on Monday April 20 2015, @11:42PM (#173327)

      In essence Poettering is a developer for devops by devops.

      One may wonder if Apple didn't want to hire him so instead he set his sights on remaking Linux into OSX.

      Sadly he seems to have found allies in the Gnome/Fedora camp, and backing by Red Hat. Likely because of IaaS or some other "cloud" marketing spiel.

      • (Score: 0) by Anonymous Coward on Wednesday April 22 2015, @08:42PM

        by Anonymous Coward on Wednesday April 22 2015, @08:42PM (#174103)

        They can take systemd and shove it up their cloud, but that shouldn't mean everyone has to shove it up their respective clouds too!

        I want choice in what I shove up my cloud!

        • (Score: 0) by Anonymous Coward on Thursday April 23 2015, @12:08AM

          by Anonymous Coward on Thursday April 23 2015, @12:08AM (#174167)

          When money talk, principles walk...

          Containers and "web apps" are the hip new thing in the cloud, and so everyone is gunning for it.

          It seems Canonical had some kind of grasp on it, but now RH is grabbing it by slipping systemd in below everything Canoncial has to offer.

          And so Canonical is adopting systemd because they do not have the developer resources to maintain a parallel "stack" to systemd.