SoylentNews is people
SoylentNews
Software engineers go crazy for the most ridiculous things. We like to think that we're hyper-rational, but when we have to choose a technology, we end up in a kind of frenzy — bouncing from one person's Hacker News comment to another's blog post until, in a stupor, we float helplessly toward the brightest light and lay prone in front of it, oblivious to what we were looking for in the first place.
This is not how rational people make decisions, but it is how software engineers decide to use MapReduce.
As Joe Hellerstein sideranted to his undergrad databases class (54 min in):
The thing is there's like 5 companies in the world that run jobs that big. For everybody else... you're doing all this I/O for fault tolerance that you didn't really need. People got kinda Google mania in the 2000s: "we'll do everything the way Google does because we also run the world's largest internet data service" [tilts head sideways and waits for laughter]
Having more fault tolerance than you need might sound fine, but consider the cost: not only would you be doing much more I/O, you might be switching from a mature system—with stuff like transactions, indexes, and query optimizers—to something relatively threadbare. What a major step backwards. How many Hadoop users make these tradeoffs consciously? How many of those users make these tradeoffs wisely?
Source: https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb
(Score: 3, Insightful) by Nerdfest on Saturday June 10 2017, @11:13AM (8 children)
Well, over-engineering for expected growth is against the "XP" sort of idea that you should "do the simplest thing that could possibly work", but usually not designing for potential growth or future changes bites you in the ass in one way or another. It's always a call that needs to be done based on experience, but I think every time I've over-engineered something based on what I expect to happen in the future, it's paid off in a huge way. As with everything, it's cost/ benefit, or at least potential cost and potential benefit.
(Score: 0) by Anonymous Coward on Saturday June 10 2017, @11:25AM
Maybe it's about predicting where the project will be in n years and deciding to make certain subsystems expandable while making other KISS-adherent.
(Score: 3, Insightful) by Bobs on Saturday June 10 2017, @12:08PM (3 children)
Yeah. I have found basing things on a fundamentally scalable architecture then KISS-Inc the hell out of it usually works well.
You can usually buy time to fix and upgrade individual subsystems as long as you don't have to rip out and rebuild the whole system.
Balanced against
(Score: 0) by Anonymous Coward on Saturday June 10 2017, @06:23PM (2 children)
1) That's fine if you're basing on OSS or free stuff. Then your scaling considerations don't need to include stuff like software licensing ;). Whereas if you base on non-free stuff, you may find it starts costing a lot more to scale.
2) It's also fine if you're not basing it on really crappy stuff. e.g. MySQL and/or PHP. Yes I know Facebook managed despite that but I'm sure it was quite painful.
(Score: 0) by Anonymous Coward on Saturday June 10 2017, @07:14PM
Just like you are not google; you are also not facebook.
(Score: 2) by Nerdfest on Saturday June 10 2017, @09:08PM
That's pretty funny, I was going to pick FaceBook as a bad example as well. They overcame it, but it took quite a lot of effort. As someone else mentioned, imagine if they used proprietary tech. I think StackOverflow did for software (and I'm not sure how much they get boned on licences), but they went cheap on hardware, which was good. I've worked for places where they "buy" lameframe processing from IBM and what would run on a few PCs costs millions.
(Score: 2) by Runaway1956 on Saturday June 10 2017, @02:56PM (1 child)
"It's always a call that needs to be done based on experience,"
There's a term for that - "judgement". Some people are qualified to make a judgement call, others are not. Personally, I've always "over engineered", but my work is in the more concrete real world. Yes, you can overdo it, but even when watching costs, I've always engineered a little above expectations. Even if I'm only building a doghouse, I want to make it sturdy enough that my dog doesn't end up in the Land of Oz. (Yeah, my dog is an Australian sheep dog, but I like her, and the Ozzies can't have her back!)
That doesn't mean I build $10,000 dog houses. But, I take the time to drive some posts into the ground, build a frame on those posts, then close it and roof it. I've never bricked a doghouse, but that would be pretty cool . . .
Point is, you can build to higher standards than the people around you, without breaking the bank. And, that's where judgement comes in.
(Score: 0) by Anonymous Coward on Saturday June 10 2017, @04:47PM
You should have built your doghouse like Google does. Less stability, more big data. I almost modded this "Offtopic", but then I saw who it was, and asked myself, "What would Google do?"
(Score: 2) by sjames on Saturday June 10 2017, @07:04PM
There is design for expansion, there's prudent over-engineering, and then there's shooting a gnat with an elephant gun.