SoylentNews is people
SoylentNews
Software engineers go crazy for the most ridiculous things. We like to think that we're hyper-rational, but when we have to choose a technology, we end up in a kind of frenzy — bouncing from one person's Hacker News comment to another's blog post until, in a stupor, we float helplessly toward the brightest light and lay prone in front of it, oblivious to what we were looking for in the first place.
This is not how rational people make decisions, but it is how software engineers decide to use MapReduce.
As Joe Hellerstein sideranted to his undergrad databases class (54 min in):
The thing is there's like 5 companies in the world that run jobs that big. For everybody else... you're doing all this I/O for fault tolerance that you didn't really need. People got kinda Google mania in the 2000s: "we'll do everything the way Google does because we also run the world's largest internet data service" [tilts head sideways and waits for laughter]
Having more fault tolerance than you need might sound fine, but consider the cost: not only would you be doing much more I/O, you might be switching from a mature system—with stuff like transactions, indexes, and query optimizers—to something relatively threadbare. What a major step backwards. How many Hadoop users make these tradeoffs consciously? How many of those users make these tradeoffs wisely?
Source: https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb
(Score: 2) by Runaway1956 on Saturday June 10 2017, @02:56PM (1 child)
"It's always a call that needs to be done based on experience,"
There's a term for that - "judgement". Some people are qualified to make a judgement call, others are not. Personally, I've always "over engineered", but my work is in the more concrete real world. Yes, you can overdo it, but even when watching costs, I've always engineered a little above expectations. Even if I'm only building a doghouse, I want to make it sturdy enough that my dog doesn't end up in the Land of Oz. (Yeah, my dog is an Australian sheep dog, but I like her, and the Ozzies can't have her back!)
That doesn't mean I build $10,000 dog houses. But, I take the time to drive some posts into the ground, build a frame on those posts, then close it and roof it. I've never bricked a doghouse, but that would be pretty cool . . .
Point is, you can build to higher standards than the people around you, without breaking the bank. And, that's where judgement comes in.
(Score: 0) by Anonymous Coward on Saturday June 10 2017, @04:47PM
You should have built your doghouse like Google does. Less stability, more big data. I almost modded this "Offtopic", but then I saw who it was, and asked myself, "What would Google do?"