SoylentNews is people
SoylentNews
Software engineers go crazy for the most ridiculous things. We like to think that we're hyper-rational, but when we have to choose a technology, we end up in a kind of frenzy — bouncing from one person's Hacker News comment to another's blog post until, in a stupor, we float helplessly toward the brightest light and lay prone in front of it, oblivious to what we were looking for in the first place.
This is not how rational people make decisions, but it is how software engineers decide to use MapReduce.
As Joe Hellerstein sideranted to his undergrad databases class (54 min in):
The thing is there's like 5 companies in the world that run jobs that big. For everybody else... you're doing all this I/O for fault tolerance that you didn't really need. People got kinda Google mania in the 2000s: "we'll do everything the way Google does because we also run the world's largest internet data service" [tilts head sideways and waits for laughter]
Having more fault tolerance than you need might sound fine, but consider the cost: not only would you be doing much more I/O, you might be switching from a mature system—with stuff like transactions, indexes, and query optimizers—to something relatively threadbare. What a major step backwards. How many Hadoop users make these tradeoffs consciously? How many of those users make these tradeoffs wisely?
Source: https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb
(Score: 1, Informative) by Anonymous Coward on Saturday June 10 2017, @05:56PM (1 child)
I had a PHB that had Hadoop on the brain. Insisted that we use Hadoop because he thought it would help the marketing.
I tried to explain that having 1 TB of data split across thousands of files wasn't nearly enough to justify the Hadoop overhead and got shouted down. Fortunately we never started implementing before our team shifted to another org.
Sometimes it isn't the engineers that make these silly decisions...it is the marketing/sales types who think it will drive sales.
(Score: 2) by YeaWhatevs on Saturday June 10 2017, @09:39PM
PHB: We have to use this latest technology, our competitors are using it.
Dilbert: Let's just say we do and then don't.
PHB: Does that work?
Dilbert: It almost did on us.