Stories
Slash Boxes
Comments

SoylentNews is people

posted by NCommander on Monday May 18 2015, @10:00AM   Printer-friendly
from the standing-by-our-principles dept.

Normally, when I make a post on SoylentNews, it's to talk about some exciting new feature, our future, or something similar.

Unfortunately though, on rare occasions, I have to make announcements like this one. Sometime between May 12-13th, one of our email accounts was breached. The account ("test1") was left over from go live, over a year and half ago, and had a very weak password protecting it. We believe that an automated password guesser was able to find and access the account. Once breached, the account was used to send a significant amount of spam until we deleted the affected account on the 14th May 2015.

As a result of the compromise, several spam services have blacklisted our mail server; we're currently working to try and get ourselves cleared whenever we become aware of one of these blocks. We do not believe any user information or sensitive data was compromised; the account in question was simply a virtual dovecot account with no corresponding UNIX account attached to it.

mechanicjay was primarily responsible for handling this and cleaning up the mess, and I wish to personally thank him and the rest of the sysops team for their handling of this issue. We are looking at taking steps to prevent a reoccurence such as using fail2ban and the like. Unfortunately, most IDS systems like fail2ban are incompatible with IPv6 which we use extensively internally within our network.

A sysops meeting is being scheduled to discuss this and other changes we're making to the infrastructure.

I will update this article (or post a new one) with additional information should it become available,
NCommander

Related Stories

We've Killed IPv4! 71 comments
As part of wanting to be part of a brighter and sunny future, we've decided to disconnect IPv4 on our backend, and go single-stack IPv6. Right now, reading to this post, you're connected to our database through shiny 128-bit IP addressing that is working hard to process your posts. For those of you still in the past, we'll continue to publish A records which will allow a fleeting glimpse of a future without NAT.
Moderation Roundtable 184 comments
After my last SN post the topic of moderation was brought up. Since its been quite awhile since we last openly discussed the state of moderation, I want to give the community a venue to discuss their feelings on it, and if the system needs further refinement. As a reminder, here's a review for how the system is currently setup:
  • 5 mod points are handed out to at 00:10 UTC to users with positive karma
  • ACs start at +0, users with karma less than 40 post at +1, users above that can post at +2
  • You need 10 karma to mark some spam or troll
  • Under normal circumstances, the staff do *not* have unlimited mod points, but can (and have) banned abusers of the moderation system

Please also review our SoylentNews Moderation Guidelines.

As always, we are willing to make changes to the system, but please post examples *with* links to any cases of suspected mod abuse. It's a lot easier to justify changing the system when evidence is in black and white. I also recommend that users make serious proposals on changes we can make. I'm not going to color the discussion with my own opinions, but as always, I will respond inline with comments when this goes live, and post a follow up article a few days after this one

This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 3, Funny) by Anonymous Coward on Monday May 18 2015, @10:27AM

    by Anonymous Coward on Monday May 18 2015, @10:27AM (#184447)

    -1 Breach

    • (Score: 2, Funny) by Anonymous Coward on Monday May 18 2015, @01:19PM

      by Anonymous Coward on Monday May 18 2015, @01:19PM (#184512)

      Is there any indication that Cypherpunks where involved?

      • (Score: 2) by meisterister on Monday May 18 2015, @06:21PM

        by meisterister (949) on Monday May 18 2015, @06:21PM (#184682) Journal

        And I think we may have a new meme on our hands...

        --
        (May or may not have been) Posted from my K6-2, Athlon XP, or Pentium I/II/III.
        • (Score: 2) by aristarchus on Tuesday May 19 2015, @07:39AM

          by aristarchus (2645) on Tuesday May 19 2015, @07:39AM (#184963) Journal

          I think we may have a new meme on our hands...

          No, no we don't! Only positive and uplifting memes here on Soylent News what got an email account hacked. But seriously, Soylent News is People, after all, and people, and I am referring to "people" here, (do I get extra points for a Serenity quote?), make mistakes like picking weak passwords because it is only a test account and should deleted in the next update of the whole system. Or, not. People. I can live with it, I always have.

    • (Score: 0) by Anonymous Coward on Monday May 18 2015, @08:08PM

      by Anonymous Coward on Monday May 18 2015, @08:08PM (#184767)

      If this email account was breached, how can we be sure that the database or databases containing the user data were not breached, as well?

      • (Score: 2) by NCommander on Tuesday May 19 2015, @05:49AM

        by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Tuesday May 19 2015, @05:49AM (#184939) Homepage Journal

        There's no indication that anything managed to get in over SSH, and as I said before, the account was a virtual dovecot account. Even if berylliun was breached (which again we have no reason to believe), the only information on that box is the wiki. There are no kerberos keytabs on that box beside the standard host keytab, so it would have been impossible to SSH to another machine with store credentials.

        Once authenticated, you can relay freely through postfix since the staff frequently have to send emails from beyond their normal address; i.e., replying as qa@soylentnews.org or something like that.

        --
        Still always moving
  • (Score: 5, Insightful) by Kell on Monday May 18 2015, @10:29AM

    by Kell (292) on Monday May 18 2015, @10:29AM (#184448)

    Thanks for being open and up-front about this. That kind of transparency is important to keep the faith with the users. Being proactive and honest does a lot to protect the reputation of an organisation, compared to burying it and hoping nobody notices.

    --
    Scientists ask questions. Engineers solve problems.
    • (Score: 0) by Anonymous Coward on Monday May 18 2015, @12:23PM

      by Anonymous Coward on Monday May 18 2015, @12:23PM (#184474)

      This is another great reason not to have an account here, and just to post as AC all of the time. An account that doesn't exist cannot be compromised.

      • (Score: 5, Funny) by Anonymous Coward on Monday May 18 2015, @01:08PM

        by Anonymous Coward on Monday May 18 2015, @01:08PM (#184504)

        Actually I just hacked your Anonymous Coward account, and now I can post as you! ;-)

        • (Score: 0) by Anonymous Coward on Monday May 18 2015, @01:10PM

          by Anonymous Coward on Monday May 18 2015, @01:10PM (#184506)

          Are you a Cypherpunk?

          • (Score: 0) by Anonymous Coward on Monday May 18 2015, @02:05PM

            by Anonymous Coward on Monday May 18 2015, @02:05PM (#184536)

            I am the Supreme Ultimate Cypherpunk EleventyOne of all time!

            I have hacked ALL the AC accounts, and NOW all your AC are belong to ME!!

            Now bow down and pay tribute to me. I want all of your 'one' bits sacrificed to my Honor and Glory. Since I am feeling generous, I will let you keep your puny 'zero' bits.

        • (Score: 0) by Anonymous Coward on Monday May 18 2015, @05:16PM

          by Anonymous Coward on Monday May 18 2015, @05:16PM (#184640)
          I'm a completely different AC and I agree with this post!
          • (Score: 1, Funny) by Anonymous Coward on Monday May 18 2015, @09:10PM

            by Anonymous Coward on Monday May 18 2015, @09:10PM (#184818)

            HAHAHA, DISREGARD THAT, I SUCK COCKS

  • (Score: 5, Funny) by GreatAuntAnesthesia on Monday May 18 2015, @10:29AM

    by GreatAuntAnesthesia (3275) on Monday May 18 2015, @10:29AM (#184449) Journal

    I mean, you left a small vulnerability in one of your services, which is fair enough. Plenty of reputable organisations have done the same.

    But then you have the audacity to secure everything else so that the attacker can't escalate access, steal user information and payment data and then download all your incriminating emails for subsequent wikileakage. I notice you didn't let the attacker piss unintelligible l33tspeak graffiti all over your site's front page! What is this, are we back in the eighties or something?

    Next thing you'll be telling us you didn't even try to cover the whole mess up before being forced to admit what happened. Furthermore I bet you completely failed to threaten some altruistic security researcher with legal action for giving you warning about the security flaw months before it was exploited.

    Honestly, I don't know what the web is coming to.

    • (Score: 5, Interesting) by NCommander on Monday May 18 2015, @10:58AM

      by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Monday May 18 2015, @10:58AM (#184456) Homepage Journal

      We've gone through a *lot* of pain to keep things as isolated as possible from each other. For example, we don't use SSH private keys for server-to-server communication, instead using kerberos keytabs which can easily be nuked from a central location in case of breach. If it wasn't for the fact that kerberos is a pain to use over the open internet, I'd love to remove the use of SSH key all together; *way* too easy to leave an authorized_keys somewhere after a breach and get pwned for it.

      We give minimum privileges possible preventing non-sysops from having access to boxes they don't need to reduce our attach service; critical infrastructure runs in AppArmor, with the hope that if someone finds a remote execution exploit in slashcode/rehash, it would prevent breach; access logs in the database are regularly purged. Our firewall rules could probably be tighter though in retrospect though.

      No security can be perfect, but we want at least to be a tough enough nut that only a dedicated attack aimed at us would stand a chance at succeeding. If I ever get to the post of rewriting the DBI layer, rehash would only talk to the SQL server via prepared statement and not execute any raw SQL directly. Unfortunately, that's pretty deep down on the TODO list.

      --
      Still always moving
      • (Score: 2) by Gaaark on Monday May 18 2015, @01:30PM

        by Gaaark (41) on Monday May 18 2015, @01:30PM (#184515) Journal

        If I ever get to the post of rewriting the DBI layer, rehash would only talk to the SQL server via prepared statement and not execute any raw SQL directly. Unfortunately, that's pretty deep down on the TODO list.

        Is this itself not a security leak of info? Or honeypot? Hmmm....onions within onions, swimming pools within TARDIS'S.....
        ...my Snowden senses are tingling!

        --
        --- Please remind me if I haven't been civil to you: I'm channeling MDC. ---Gaaark 2.0 ---
        • (Score: 2) by Yog-Yogguth on Monday May 18 2015, @02:42PM

          by Yog-Yogguth (1862) Subscriber Badge on Monday May 18 2015, @02:42PM (#184562) Journal

          Nope. SQL stands for Structured Query Language, it's a given that it uses itself i.e. structured queries as defined by SQL. SQL gotcha's are well known and only allowing a small selection of previously prepared statements is going the extra mile, not hard to see why it isn't a top priority (note that it doesn't stop the SQL database from doing anything but it makes it harder for Rehash to send a bad or malicious query to the SQL database).

          --
          Bite harder Ouroboros, bite! tails.boum.org/ linux USB CD secure desktop IRC *crypt tor (not endorsements (XKeyScore))
          • (Score: 3, Informative) by NCommander on Monday May 18 2015, @03:42PM

            by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Monday May 18 2015, @03:42PM (#184595) Homepage Journal

            MySQL prepared statements are something of a joke. You can't do much with them and they don't offer much improvement over raw SQL. To fully replace what Slash does would require T-SQL or similar. The entire database layer is wrapped with insert/select/etc. functions which take various arguments and autoescape on passing through to the MySQL level (see Slash/DB for implementation details). Slash is coded to be resistent to injection attacks, but the sheer amount of SQL is bloody staggering. I have a project on my TODO list to port the mess to PostgreSQL so I can take the entire database wrapper, and move it *into* the database where it becomes.

            --
            Still always moving
            • (Score: 2) by tibman on Monday May 18 2015, @08:45PM

              by tibman (134) Subscriber Badge on Monday May 18 2015, @08:45PM (#184795)

              Prepared statements do take the burden of sanitation away from the developers (as far as the DB goes anyways). Since slash has business logic in the DB that could make moving to anything else a bit of a pain : /

              --
              SN won't survive on lurkers alone. Write comments.
              • (Score: 2) by NCommander on Tuesday May 19 2015, @05:43AM

                by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Tuesday May 19 2015, @05:43AM (#184936) Homepage Journal

                Relatively few FOSS projects seem to use the database layer for business logic. The odds if/when I get to it, I can remove 30k LOC or so from the DB layer into smaller stored procedures ....

                --
                Still always moving
      • (Score: 0) by Anonymous Coward on Monday May 18 2015, @05:39PM

        by Anonymous Coward on Monday May 18 2015, @05:39PM (#184653)

        if it suits what you're doing, openssh 6 and authorized key command feature are enough to centralize key credential management. ldap, sql, your grandma's kitchen cupboard, as long as you have or write an executable taking a login id and returning public keys you can store and manage them in a single place, wherever you like.

  • (Score: 5, Informative) by Anonymous Coward on Monday May 18 2015, @10:32AM

    by Anonymous Coward on Monday May 18 2015, @10:32AM (#184450)

    fail2ban and the like. Unfortunately, most IDS systems like fail2ban are incompatible with IPv6

    sshguard [sshguard.net]

    • (Score: 5, Interesting) by NCommander on Monday May 18 2015, @10:49AM

      by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Monday May 18 2015, @10:49AM (#184455) Homepage Journal

      Hello Solution. I dunno who you are, but I 3 you. I'm going to bring this up at the sysops meeting as a solution.

      --
      Still always moving
      • (Score: 2) by Techwolf on Monday May 18 2015, @03:51PM

        by Techwolf (87) on Monday May 18 2015, @03:51PM (#184601)

        I hope you did at least move off the default port. Did you?

        • (Score: 0) by Anonymous Coward on Monday May 18 2015, @05:33PM

          by Anonymous Coward on Monday May 18 2015, @05:33PM (#184649)

          I hope you did at least move off the default port. Did you?

          They did one better. They put everything on port 22. That should confuse any script kiddie, black hat, white hat, sys admin, casual observer, or even uninterested party. Most people say "42" is the answer. I'm here to tell you that "22" is the answer!

        • (Score: 2) by NCommander on Tuesday May 19 2015, @05:44AM

          by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Tuesday May 19 2015, @05:44AM (#184937) Homepage Journal

          Honestly, I've never understood the point of doing this; nmap will find it regardless. There's security through obscurity, but this seems like it would barely make any difference.

          --
          Still always moving
          • (Score: 2) by Techwolf on Tuesday May 19 2015, @06:41PM

            by Techwolf (87) on Tuesday May 19 2015, @06:41PM (#185170)

            Somewhat true. But in this case, it was a spammer. They don't bother using nmap. The porpus is too stop logfiles from filling up with script kiddies running scripts due to not knowing anything else but point and click. And spammers too sence they don't know how to crack a system via hacking on it.

  • (Score: -1, Offtopic) by Anonymous Coward on Monday May 18 2015, @11:41AM

    by Anonymous Coward on Monday May 18 2015, @11:41AM (#184462)

    I wish you editors yank Appalbarry's cliched (and deluded) rant on the pending queue. Obvious flamebait is obvious.

    • (Score: -1, Troll) by Anonymous Coward on Monday May 18 2015, @12:38PM

      by Anonymous Coward on Monday May 18 2015, @12:38PM (#184481)

      It can't be any worse than gewg_'s "USAian Researchers Impressed With Cuba's Drugs" submission, which not only is full of his usual pro-communist bullshit, but it also contains a slur ("USAian") within its title.

      • (Score: 2) by aristarchus on Tuesday May 19 2015, @07:56AM

        by aristarchus (2645) on Tuesday May 19 2015, @07:56AM (#184966) Journal

        And pull all these tiresome and redundant complaints to the "admins" to censor stuff! Are these ACs so clueless as to not realize that Soylent News is people, and that means there are no "admins"? So all this "unfair" "incorrect', and my favorite, "abusive" moderation needs to continue unabated! But one question, where has the Mighty Buzz been of late? Has he submerged? That would explain a lot. Ever since the Jade Helm accusation that it was Okies that would be invading Tejas . . . .

  • (Score: -1, Offtopic) by Anonymous Coward on Monday May 18 2015, @12:33PM

    by Anonymous Coward on Monday May 18 2015, @12:33PM (#184479)

    That email account is not the only thing being abused to harm this site.

    The other thing being abused is the moderation system.

    In pretty much every story I read, I see good comments modded down.

    Now I have to browse at -1 all of the time.

    That defeats the purpose of having a moderation system.

    Instead of flagging good comments, the abused moderation system just suppresses good comments.

    Major changes are needed.

    Major fixes are needed.

    First each modded comment should list who modded it.

    It should be possible to flag a downmod moderation as abusive.

    If a moderator gets more than three moderations flagged as being abusive, then that moderator never moderates again.

    That will be simple to build, and it will restore sanity to this site.

    It will make people think twice about downmodding.

    It will also properly save our community from those who abuse moderation to prevent discussion, instead of encouraging it.

    We need this built right away, before the moderation abuse gets any worse.

  • (Score: 3, Insightful) by Phoenix666 on Monday May 18 2015, @12:40PM

    by Phoenix666 (552) on Monday May 18 2015, @12:40PM (#184484) Journal

    Same thing happened to my server April 28th. Much sympathy and commiseration.

    --
    Washington DC delenda est.
  • (Score: 2) by zafiro17 on Monday May 18 2015, @01:49PM

    by zafiro17 (234) on Monday May 18 2015, @01:49PM (#184527) Homepage

    Those early days were fast and furious, with a huge amount of drama, dissent, and disdain. Glad they're behind us.

    That said, 'fast and furious' is a recipe for making mistakes. No surprise that some mistakes were made. Clean it up, patch the breach, make sure it doesn't happen again, and move on with life. These days, spam is a nuisance but still more tolerable than some of the other things happening when servers get breached.

    Surprised nobody started a paranoid thread here about the breach being actually the work of the CIA/NSA/FBI, hoping to infiltrate our little band of nitwits ... oops, I've said too much already.

    --
    Dad always thought laughter was the best medicine, which I guess is why several of us died of tuberculosis - Jack Handey
    • (Score: 0) by Anonymous Coward on Monday May 18 2015, @01:58PM

      by Anonymous Coward on Monday May 18 2015, @01:58PM (#184531)

      Dude, you commented on the wrong story. You want this one [soylentnews.org].

  • (Score: 0) by Anonymous Coward on Monday May 18 2015, @01:53PM

    by Anonymous Coward on Monday May 18 2015, @01:53PM (#184529)

    Why choose to be pioneers on IPv6 when so many
    security principles and tools that experienced admins
    use to secure things are not applicable on IPv6?
    It reminds me of the days when much software wouldn't
    run on 64-bit hardware, due to developers writing
    bugs that were masked on 32-bit platforms. That took
    a good 10 years to sort out, 64-bit being common now.
    But I don't think IPv6 will be common before I retire.

    • (Score: 3, Interesting) by NCommander on Monday May 18 2015, @02:02PM

      by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Monday May 18 2015, @02:02PM (#184535) Homepage Journal

      Because when it got to the point I would have to setup NAT to properly interface off-site servers with ours, I said "fuck it", and put IPv6 entries and saved everyone a lot of headache, vs having to deal with a spilt DNS setup, or other madness. End-to-end routability is a good thing, and has drastically reduced the amount of pain we have to do to make everything talk to everything.

      --
      Still always moving
      • (Score: 0) by Anonymous Coward on Monday May 18 2015, @02:21PM

        by Anonymous Coward on Monday May 18 2015, @02:21PM (#184548)

        Why not use a VPN?

        • (Score: 2) by NCommander on Monday May 18 2015, @03:40PM

          by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Monday May 18 2015, @03:40PM (#184591) Homepage Journal

          Same problem. Bridging two 192.168.x.x networks proved to be too much hassle. NAT is a hack on the best of days, and for DNS to work, I would have had to populate it with internal addresses which in turn would complicate other issues. My general opinion is if NAT can be avoided, it should be avoided.

          --
          Still always moving
          • (Score: 0) by Anonymous Coward on Monday May 18 2015, @06:30PM

            by Anonymous Coward on Monday May 18 2015, @06:30PM (#184687)

            Why is the infrastructure for this site so convoluted? It's not a particularly complex site, even when including the wiki, IRC, and other offerings.

            • (Score: 1) by Frost on Monday May 18 2015, @08:06PM

              by Frost (3313) on Monday May 18 2015, @08:06PM (#184766)

              IPv4 address pressure is causing lots of problems for servers everywhere. Unless you can run your entire site on one host you're going to have to deal with such craziness at some point.

  • (Score: 1) by Frost on Monday May 18 2015, @07:44PM

    by Frost (3313) on Monday May 18 2015, @07:44PM (#184754)

    What is the address of the SMTP server that got blacklisted? I'd like to add it to my whitelist.