SoylentNews is people
SoylentNews
Software engineers go crazy for the most ridiculous things. We like to think that we're hyper-rational, but when we have to choose a technology, we end up in a kind of frenzy — bouncing from one person's Hacker News comment to another's blog post until, in a stupor, we float helplessly toward the brightest light and lay prone in front of it, oblivious to what we were looking for in the first place.
This is not how rational people make decisions, but it is how software engineers decide to use MapReduce.
As Joe Hellerstein sideranted to his undergrad databases class (54 min in):
The thing is there's like 5 companies in the world that run jobs that big. For everybody else... you're doing all this I/O for fault tolerance that you didn't really need. People got kinda Google mania in the 2000s: "we'll do everything the way Google does because we also run the world's largest internet data service" [tilts head sideways and waits for laughter]
Having more fault tolerance than you need might sound fine, but consider the cost: not only would you be doing much more I/O, you might be switching from a mature system—with stuff like transactions, indexes, and query optimizers—to something relatively threadbare. What a major step backwards. How many Hadoop users make these tradeoffs consciously? How many of those users make these tradeoffs wisely?
Source: https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb
(Score: 4, Interesting) by The Mighty Buzzard on Saturday June 10 2017, @12:56PM (4 children)
Technically, we're greatly over-engineered from both a code and a network perspective. It's been working out pretty well for both us and the community though.
From a network perspective, we could probably get by on like three servers. I, however, quite enjoy the lack of downtime, separation of critical services from non-critical services, a dedicated development box, redundancy, and an off-site backup box.
From a code perspective we're slightly slower than we could be at the moment but it's saved us I don't know how many hours of having to optimize/rewrite/etc... already and I expect it to continue doing so as growth continues. I don't know if we'll ever outgrow the codebase to the extent that it requires us moving to something like fcgi or compiled executables instead of mod_perl and I enjoy not having "scrap everything and rewrite it" pressure.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Saturday June 10 2017, @06:45PM (3 children)
Servers are the network? Do you have a microsoft background prior to programming? this is a serious question -- only server people generally think the server is the network.
it's like truck drivers claiming trucks are the highway system.
(Score: 1) by khallow on Saturday June 10 2017, @09:15PM
(Score: 2) by The Mighty Buzzard on Saturday June 10 2017, @10:24PM (1 child)
Only cable jockeys think the network does not include the hardware attached to it.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Sunday June 11 2017, @12:53AM
As much as TMB is a boob at times, he makes a valid point that a network is worthless without servers on it. In a way, the servers are the purpose for the network and can -by extension- be equated with it for all intents and purposes. After all, that'd be a nice network you got going there, without any servers on it.