Stories
Slash Boxes
Comments

SoylentNews is people

posted by n1 on Saturday June 10 2017, @11:07AM   Printer-friendly
from the i-am-spartacus dept.

Software engineers go crazy for the most ridiculous things. We like to think that we're hyper-rational, but when we have to choose a technology, we end up in a kind of frenzy — bouncing from one person's Hacker News comment to another's blog post until, in a stupor, we float helplessly toward the brightest light and lay prone in front of it, oblivious to what we were looking for in the first place.

This is not how rational people make decisions, but it is how software engineers decide to use MapReduce.

As Joe Hellerstein sideranted to his undergrad databases class (54 min in):

The thing is there's like 5 companies in the world that run jobs that big. For everybody else... you're doing all this I/O for fault tolerance that you didn't really need. People got kinda Google mania in the 2000s: "we'll do everything the way Google does because we also run the world's largest internet data service" [tilts head sideways and waits for laughter]

Having more fault tolerance than you need might sound fine, but consider the cost: not only would you be doing much more I/O, you might be switching from a mature system—with stuff like transactions, indexes, and query optimizers—to something relatively threadbare. What a major step backwards. How many Hadoop users make these tradeoffs consciously? How many of those users make these tradeoffs wisely?

Source: https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 0) by Anonymous Coward on Saturday June 10 2017, @06:23PM (2 children)

    by Anonymous Coward on Saturday June 10 2017, @06:23PM (#523541)

    I have found basing things on a fundamentally scalable architecture then KISS-Inc the hell out of it usually works well.

    1) That's fine if you're basing on OSS or free stuff. Then your scaling considerations don't need to include stuff like software licensing ;). Whereas if you base on non-free stuff, you may find it starts costing a lot more to scale.

    2) It's also fine if you're not basing it on really crappy stuff. e.g. MySQL and/or PHP. Yes I know Facebook managed despite that but I'm sure it was quite painful.

  • (Score: 0) by Anonymous Coward on Saturday June 10 2017, @07:14PM

    by Anonymous Coward on Saturday June 10 2017, @07:14PM (#523559)

    2) It's also fine if you're not basing it on really crappy stuff. e.g. MySQL and/or PHP. Yes I know Facebook managed despite that but I'm sure it was quite painful.

    Just like you are not google; you are also not facebook.

  • (Score: 2) by Nerdfest on Saturday June 10 2017, @09:08PM

    by Nerdfest (80) on Saturday June 10 2017, @09:08PM (#523587)

    That's pretty funny, I was going to pick FaceBook as a bad example as well. They overcame it, but it took quite a lot of effort. As someone else mentioned, imagine if they used proprietary tech. I think StackOverflow did for software (and I'm not sure how much they get boned on licences), but they went cheap on hardware, which was good. I've worked for places where they "buy" lameframe processing from IBM and what would run on a few PCs costs millions.