Science is a messy, error fraught business, which is why reproducibility is so essential. Unfortunately, that doesn't appear to be one of psychology's strong suits, according to a massive analysis published yesterday in Science.
A years-long effort to reproduce more than 100 psychology studies across three leading journals paints a pretty dismal picture. When re-tested by independent research psychologists, the conclusions of more than 60 studies on personality, relationships, learning, and memory, turned out to be far less whelming. Strongly significant findings often became weaker, while weakly significant findings became non-existent.
http://gizmodo.com/a-lot-of-published-psychology-results-are-bullshit-1727228060
[Source]: The New York Times
Related Stories
Psychologists Call Out the Study That Called Out the Field of Psychology
Remember that study that found that most psychology studies were wrong? Yeah, that study was wrong. That's the conclusion of four researchers who recently interrogated the methods of that study, which itself interrogated the methods of 100 psychology studies to find that very few could be replicated. (Whoa.) Their damning commentary will be published Friday in the journal Science. (The scientific body that publishes the journal sent Slate an early copy.)
In case you missed the hullabaloo: A key feature of the scientific method is that scientific results should be reproducible—that is, if you run an experiment again, you should get the same results. If you don't, you've got a problem. And a problem is exactly what 270 scientists found last August, when they decided to try to reproduce 100 peer-reviewed journal studies in the field of social psychology. Only around 39 percent of the reproduced studies, they found, came up with similar results to the originals.
That meta-analysis, published in Science by a group called the Open Science Collaboration, led to mass hand-wringing over the "replicability crisis" in psychology. (It wasn't the first time that the field has faced such criticism, as Michelle N. Meyer and Christopher Chabris have reported in Slate, but this particular study was a doozy.)
Now this new commentary, from Harvard's Gary King and Daniel Gilbert and the University of Virginia's Timothy Wilson, finds that the OSC study was bogus—for a dazzling array of reasons. I know you're busy, so let's examine just two.
The first—which is what tipped researchers off to the study being not-quite-right in the first place—was statistical. The whole scandal, after all, was over the fact that such a low number of the original 100 studies turned out to be reproducible. But when King, a social scientist and statistician, saw the study, he didn't think the number looked that low. Yeah, I know, 39 percent sounds really low—but it's about what social scientists should expect, given the fact that errors could occur either in the original studies or the replicas, says King.
takyon: Recycle or repudiate comments made back in August.
(Score: 0, Touché) by Anonymous Coward on Sunday August 30 2015, @05:00AM
Many Published Psychology Results are Hogwash
That is what a delusional person would say. Take these pills which will dull your senses and make you docile. Now repeat after me: Psychology is a perfectly legitimate science.
(Score: 2) by Runaway1956 on Sunday August 30 2015, @05:11AM
80% of all the crap I've ever heard from psychologists is hogwash. I put as much faith in witch doctors as I do psychologists, psychiatrists, psychoanalysts.
Remember Freud and his "penis envy" theories about women? Retarded shit - and psycho "doctors" gobbled that shit up.
(Score: 0) by Anonymous Coward on Sunday August 30 2015, @05:14AM
But it's useful for people who want to control others. A study says violent video games make people violent? Ban violent video games! Don't even bother reporting that the study you cited that concluded that violent video games make people violent was later debunked and isn't accepted by the majority of scientists.
The media takes garbage studies and reports them in such a way that they appear even worse. They're very good at that.
(Score: 5, Insightful) by frojack on Sunday August 30 2015, @05:20AM
You forgot to toss in Chiropractors, and homeopaths.
The problem is that insurance pays for quackery, and the amount of quackery expands to consume the available funds.
No, you are mistaken. I've always had this sig.
(Score: 0) by Anonymous Coward on Sunday August 30 2015, @05:22AM
You say that now, but wait till you get a case of subluxation - you'd be calling dr. bob.
(Score: 2) by Runaway1956 on Sunday August 30 2015, @05:24AM
I suppose that I should look up subluxation, and dr. bob, and offer some witty response.
I'll just stay out of the polluted gene pools to avoid catching it.
(Score: 0) by Anonymous Coward on Sunday August 30 2015, @12:42PM
It's a reference to one of the few great trolls of pre-beta /.'s later years.
(Score: 4, Interesting) by caffeine on Sunday August 30 2015, @05:43AM
I read something from one of the managers of a health insurance fund in Australia a while ago that discussed this issue. They know full well that chiropractors and homeopaths are BS but offer coverage for this as it tends to attract younger members to join their fund. Their real big ticket items are knee & hip replacements and they have to keep signing up healthy young people to keep their fund viable as their member base ages. Last time I checked, in Australia the only fund that did not pay for BS was the doctors health fund.
(Score: 2) by mrchew1982 on Monday August 31 2015, @01:21AM
Chiropractors are OK for back and neck pain, or at least the good ones are. Most of the crap that they do can be accomplished with a good stretch though. The biggest problem that they all have is that they would like to think that they can cure cancer and AIDS just by waving their hands, and they preach that shite any chance that they get...
Fully agreed on homeopathy, another thing that quack chiropractors like to peddle...
(Score: 1, Funny) by Anonymous Coward on Sunday August 30 2015, @06:26AM
80% of all the crap I've ever heard from psychologists is hogwash. I put as much faith in witch doctors as I do psychologists
Houston, we have a problem!
This study of the reproducibility of psychology studies was, you guessed it, conducted by psychologists!
Runaway's mind is now stuck in a recursive loop.
(Score: 0) by Anonymous Coward on Sunday August 30 2015, @06:34AM
Thank Bob it wasn't an infinite loop. Runaway will back to normal as soon as recursion smashes his stack.
(Score: 0) by Anonymous Coward on Sunday August 30 2015, @07:19AM
They say you have to hit rock bottom before you can get better.
(Score: 2) by Runaway1956 on Sunday August 30 2015, @07:58AM
Poor child - you should read the post again. If 80% of shrink talk is bullshit, that leaves 20% which is something other than bullshit. No recursive loop here.
(Score: 0) by Anonymous Coward on Sunday August 30 2015, @08:37AM
Poor child - you should read the post again. If 80% of shrink talk is bullshit, that leaves 20% which is something other than bullshit. No recursive loop here.
Lol, I saw it. And yet these psychologists found substantially less than 80% was bullshit. So in the loop of cognitive dissonance you remain.
(Score: 0) by Anonymous Coward on Thursday September 03 2015, @01:10PM
They should check out the social 'sciences' specifically.
(Score: 0) by Anonymous Coward on Friday September 04 2015, @09:30PM
These psychologists showed no such thing. This was about reproducibility. Even if a study can be reproduced, that doesn't mean it is understand why researchers are getting the result that they are. So it is still possible that their theories about why they are getting those results are nonsense.
(Score: 0) by Anonymous Coward on Sunday August 30 2015, @08:03AM
Houston, we have a problem!
Notice how he said "80%" and not all. Not sure where that number came from, but still.
Furthermore, generally, the more complex whatever you're studying is, the more difficult it is to study it properly. Reproducing the studies to see if you get the same result is more simple than doing original and good science in the first place, as there is less that can go wrong. That is why this is more believable than the types of studies that reach arbitrary and subjective conclusions about how violent video games make people more violent, how X makes people "callous" towards Y, etc.
(Score: 4, Informative) by Phoenix666 on Sunday August 30 2015, @11:38AM
As someone trained in the social sciences, I have on occasion thought that physical scientists rather take the easy way out in studying natural phenomena that can be pinned down, controlled, and whose behaviors can be reproduced. People are devilishly difficult to study, because they are complex and moving all the time. They're conscious, and consciously or unconsciously mess with your study. As such it's much more complex, much more multi-variate a system than what, say, chemists tackle.
Washington DC delenda est.
(Score: 2) by c0lo on Sunday August 30 2015, @12:58PM
If the "social scientists" would deal with populations on the order of 1023, I reckon the social sciences would be as accurate as chemistry (which is not that accurate as many people think, those equilibrium points for the reversible reactions are correct just statistically and, anyway, empirically determined).
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 2) by maxwell demon on Sunday August 30 2015, @02:59PM
Actually I've once heard that some chemical reactions won't work if the chemist cleans his equipment too thoroughly.
The Tao of math: The numbers you can count are not the real numbers.
(Score: 0) by Anonymous Coward on Sunday August 30 2015, @01:39PM
I think Dr. Fancyfree [wikia.com] says it best regarding this entire field: I'm going to do something that goes against all my training as a therapist... I'm going to CURE you!!!
(Score: 0) by Anonymous Coward on Sunday August 30 2015, @04:06PM
It's complex, but that is no reason to lower our standards. Bad science is bad science. I'm not saying you were necessarily saying that, though.
(Score: 2) by Nerdfest on Sunday August 30 2015, @05:17PM
Hah. Try quantum physics.
(Score: 2) by Phoenix666 on Sunday August 30 2015, @05:25PM
Touche, nerdfest.
Washington DC delenda est.
(Score: 2) by hendrikboom on Tuesday September 01 2015, @01:38PM
The theoretical calculations of various properties of fundamental particles have often been orders of magnitude more accurate than the observations the theories were originally based on, as evidenced by later, more accurate observations.
No one knows why.
(Score: 0) by Anonymous Coward on Sunday August 30 2015, @06:29PM
It is not specific to social sciences. [washingtonpost.com] There is tons of pressure to bias results in the hard sciences too.
IIRC there was a recent story about how requiring scientists to formally specify their testing regimen up front, before starting the experiment, drastically cut down the 'success rate.' The theory being that the urge to comb through the data after the fact, looking for novel results was just too strong.
(Score: 2) by c0lo on Sunday August 30 2015, @12:47PM
He's trying to align more with the classics than with Sturgeon [wikipedia.org].
From the same page and as a word of caution for those disgusted by 80% crap of everything:
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 0) by Anonymous Coward on Sunday August 30 2015, @06:50AM
Gobbled up penises did they, hmm tell me about your mother.
(Score: -1, Troll) by Anonymous Coward on Sunday August 30 2015, @06:54AM
Your mother wishes her clit were a dick so she could rock you up the ass.
(Score: 3, Insightful) by aristarchus on Sunday August 30 2015, @10:34AM
Sigmund Freud said that sometimes a good homoerotic fantasy was just a good homoerotic fantasy, and not a cigar, or a shoe, or a truck, or a gun, or a large comb-over, or anything else one might mistake a good homoerotic fantasy for. Oh, and the more you deny it, the more it is true. Of course, if you don't deny it, it is true any way. So basically your choice, when you attempt to take on Dragons or Shrinks.
(Score: 2) by c0lo on Sunday August 30 2015, @12:38PM
And the rest of 20% crap is what? Pure crap?
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 2) by Nerdfest on Sunday August 30 2015, @05:18PM
Unproven.
(Score: 0) by Anonymous Coward on Sunday August 30 2015, @05:35AM
Attention! Scientologist on Deck! All's we need is few good men just crazy enough to deny psychology, but not so crazy they can't be controlled. Battlefield Earth! L Ron! It was all just a bet, seriously! How could things get this out of control.
(Score: 2, Interesting) by Anonymous Coward on Sunday August 30 2015, @07:25AM
What shocks me is someone actually managed to get funded to repeat someone else's work. What's that, you say? You want money to copy the same thing as someone else? But where is the shiny in that? Funding DENIED, back on the treadmill monkey.
(Score: 2, Informative) by aristarchus on Sunday August 30 2015, @10:45AM
It is also true the much of what is published as hogwash is actually psychology. When I worked on a hog farm, we had many cases where the pigs did not come clean, only to discover that the wash we had used on the porkers was actually psychology! Boy, did we feel embarrassed! But not as much as the psychologists when we caught up with them! Two stood out, James Mitchell and Bruce Jessen, their stuff was supposed to prevent outbreaks of spontaneous swine explosion, but instead they kept having relations with the subjects in violation of the hogwashers code of ethics. We decided to give them their own medicine, and boy did they squeal all the way home, to nice retirements funded by non-psychologists. So I guess what I am saying is, when you are trying to wash hogs, get the real hogwash. And when you are doing psychology, avoid people who like to support torture. And try not to confuse the two.
(Score: 1, Interesting) by Anonymous Coward on Sunday August 30 2015, @04:11PM
I am a psychology scientist. Fuck Freud. Freud was not a scientist...did not even care about science.
I often call myself a cognitive neuroscientist. People don't think Freud when they hear that.
Reproducibility can be difficult...Contrary to the typical ideas about more data equals more statistical power, it is also true that small samples get statistically significant results more frequently...false positives, mind you. Funding should encourage and pay for larger samples to increase reproducibility and reduce false positives.
Also, when research enters cognitive domains proportionally more influenced by culture then ensuring results are qualified by culture of sample is important...I really can see this as important and one that would impede reproducibility in research, for example, examining attitudes toward sexual partners..
(Score: 2) by cafebabe on Monday August 31 2015, @04:07PM
I'd be one of the first to say the psychologists are charlatans. However, I think I've come to an explanation of why this is the case. They're doing the statistical analysis completely wrong. If your objective is to perform black-box testing on an aggregate sample of people to determine typical behaviour then you don't have N samples of data. You have N=1 with a margin of error. This is not science. This is anecdote. Furthermore, if your sample of people uses affluent, Anglo undergraduates then there is no basis to extrapolate your anecdote beyond this skewed sample.
1702845791×2
(Score: 2) by hendrikboom on Tuesday September 01 2015, @01:48PM
Given how difficult it is to do psychological experiments, I am amazed that there are some psychological maladies that repeatable experiment has shown to be actually treatable much of the time. I'm just astounded.
(Score: 0) by Anonymous Coward on Thursday September 03 2015, @01:14PM
The results might be repeatable, but whether the reasons for the results are what they say they are is another matter.