SoylentNews is people
SoylentNews
The field of psychology has recently been embarrassed by failed attempts to repeat the results of classic textbook experiments, and a mounting realization that many papers are the result of commonly accepted statistical shenanigans rather than careful attempts to test hypotheses.
Now Ed Yong writes at The Atlantic that Anna Dreber at the Stockholm School of Economics has created a stock market for scientific publications, where psychologists bet on published studies based on how reproducible they deemed the findings. Based on Robin Hanson's classic paper "Could Gambling Save Science," that proposed a market-based alternative to peer review called "idea futures," the market would allow scientists to formally "stake their reputation", and offer clear incentives to be careful and honest while contributing to a visible, self-consistent consensus on controversial (or routine) scientific questions.
Here's how it works. Each of 92 participants received $100 for buying or selling stocks on 41 studies that were in the process of being replicated. At the start of the trading window, each stock cost $0.50. If the study replicated successfully, they would get $1. If it didn't, they'd get nothing. As time went by, the market prices for the studies rose and fell depending on how much the traders bought or sold. The participants tried to maximize their profits by betting on studies they thought would pan out, and they could see the collective decisions of their peers in real time. The final price of the stocks, at the end of two-week experiment, reflected the probability that each study would be successfully replicated, as determined by the collective actions of the traders. In the end, the markets correctly predicted the outcomes of 71 percent of the replications—a statistically significant, if not mind-blowing score.
"It blew us all away," says Dreber. "There is some wisdom of crowds; people have some intuition about which results are true and which are not," adds Dreber. "Which makes me wonder: What's going on with peer review? If people know which results are really not likely to be real, why are they allowing them to be published?"
(Score: 1, Funny) by Anonymous Coward on Saturday November 14 2015, @06:14PM
I'll take my $100 and bet that gambling isn't going to save science.
(Score: 2) by captain normal on Sunday November 15 2015, @03:58AM
What?! Science is playing the odds. What else did you expect.
Everyone is entitled to his own opinion, but not to his own facts"- --Daniel Patrick Moynihan--
(Score: 0) by Anonymous Coward on Sunday November 15 2015, @07:35AM
Hookers and blow. For science!
(Score: 0) by Anonymous Coward on Saturday November 14 2015, @06:18PM
Time to end the fake "stock market" with all its cheaters and bring back real investments.
(Score: 0) by Anonymous Coward on Saturday November 14 2015, @07:34PM
https://explainxkcd.com/882/ [explainxkcd.com]
(Score: 3, Interesting) by jmorris on Saturday November 14 2015, @07:45PM
Look at the incentives and understand what is happening.
Scientists must publish or perish. So that is the prime incentive. Most scientists also depend on grants. So the second directive is to publish things grant funding is paying for and that will attract future funds. Until this scandal blew up recently nobody was even trying to replicate studies that the group consensus liked the results of, so there is the third incentive for bad science.
Until publishing a study later proven to be utter rubbish ends a few careers there will be no end of this. People will still publish rubbish and some of it will now be blown down; years later and after many more people have already leveraged it to get their follow on studies fully funded and published. Meaning the incentives to publish whatever the grant making entities (and let us be honest and say that for all intents and purposes that means the large Western governments and their closely allied NGOs) are needing to have 'proven' this year.
(Score: 4, Interesting) by bzipitidoo on Saturday November 14 2015, @09:36PM
I always thought capitalism was a brilliant way to harness a largely negative drive, greed, for the good of humanity. It is that, it does work. The market is one place our competitive natures can be non-destructive. So long as the world was wide, and there was room for everyone who had the will to work a little, maybe take a risk, "Go West young man" was a better bet than any number of more desperate and ugly measures. One such ugly measure was to kill off all your siblings, without being found out or punished, and then you inherit the family farm. Very risky-- what if some of your siblings have the same idea and get you first? Or, resort to corruptive measures such as bribing the officials and rigging the game, without being caught and punished, then your business can prosper easily and very conservatively in the absence of serious competition. So naturally, anything with a good reputation is a target for cheating, and the reputation of science is still unsurpassed.
Now space is getting short. We and our institutions are being tested. If we keep on as we are, the young will have only a choice of evils. The pinch will be very ugly if our room shrinks thanks to Climate Change before we settle into a stable number and end our population growth. If it's bad enough, war or mass starvation could be the only options. On the other hand, a great, long age of peace, prosperity, and happiness is within our grasp. But will we reach for it? Thanks to our sciences, we know of some serious problems ahead. We have the means to avoid them, but do we have the will? We seem more determined to dump science for being too much work (Barbie says "Math is hard!"), and for telling us things we don't want to hear. The entire Republican Party seems to have abandoned fact for propaganda, to paint this fantasy world that one would think is at least all unicorns and rainbows. But it isn't even that, their fantasy vision has some very ugly parts. They seem to really want war, keep maligning the Muslims as terrorists eager to war upon us. Exterminating the entire Islamic faith and all its followers, without destroying the West, if even remotely possible, would open up a lot of room, yes, but at what cost? Also the "drill, baby, drill" mantra has very little to recommend it. Why do we want to drill for more oil? So we can keep driving cars powered by polluting, noisy combustion engines, when we could be switching over to electric motors. Never going to change, the essence of conservatism, despite that the transition from horse to car was a massive and recent change. It is hard to imagine more foolish directions to go. Most exasperating of all is when they admit that things aren't going well, but instead of facing the fact that there will have to be some changes, they start talking about the End Times and preparing for the Rapture! Can't get a fantasy much darker than that.
The 1950s into the 1960s was a Golden Age for science, what with the then recent WWII victory having demonstrated the power of superior knowledge, renewed by the Sputnik scare and the answering push to land a man on the moon. I have heard that scientists who received their PhDs in those days had it made. They hardly had to look for work, work came looking for them, and they were practically handed tenured professorships, labs, and grant money. Now, there is more pressure, and seems likely one manifestation is more cheating. A steady background of doubt in the value of science only makes it worse. Makes it more likely that the sort of people who are total frauds, secretly holding science in contempt, who cynically believe most scientists are cheaters just like them, will manage to bullshit their way all the way to a PhD, and once there, make a big mess publishing bullshit that is just good enough to sucker peer reviewers, but when finally exposed does more than end their fraudulent career, it causes more public doubt of science itself.
(Score: 2) by jmorris on Sunday November 15 2015, @05:51AM
Time doesn't permit taking all of that on, but will shoot a few holes in the key bits of bogosity.
So long as the world was wide, and there was room for everyone...
Doesn't matter. The Industrial Revolution was started in England at a time when a population increase had left them far more people than their social order knew what to do with. The rise of the factories was the solution that worked and changed the world. With the right mindset there is opportunity in just about any situation.
The pinch will be very ugly if our room shrinks thanks to Climate Change before we... [Malthusian blah blah deleted]
Look at the article we are commenting on again. Climate science has made no valid predictions. All of the models ran far enough back to be judged against actual events, i.e. twenty or so years ago, are now outside their error bars. The precise scientific term for that is WRONG. Now we can't even make new predictions because the very datasets are 'adjusted' and in many cases unavailable to the public for outside inspection. The models themselves are closed software and unavailable for inspection. This is not how Science works.
But the big tell is the reaction of Climate Science to the word 'Denier.' Were they scientists they would have denounced the political hacks trying to outlaw dissent. They didn't, telling those with eyes to see everything needed.
At this point the continued belief in AGW is harder to sustain than Christians believing in Transubstantiation or the Mullahs in Iran waiting for the dude to climb out outta that well. Bluntly, it is nuts.
They seem to really want war, keep maligning the Muslims as terrorists eager to war upon us. Exterminating the entire Islamic faith and all its followers, without destroying the West, if even remotely possible, would open up a lot of room, yes, but at what cost?
Yea, when people are crucifying people, hacking off heads and posting the proudly posting the video to social media we certainly should not believe they mean us harm. Makes perfect sense in fact to give them atom bombs!
NOT.
It probably isn't required to exterminate the entire population of Muslims. What is required is two things. One is to kill their dream of a restored Caliphate exerting a world dominating influence. That means destroying ISIS in such a way they can plainly see that ISIS never had a chance once we seriously decided to put an end to them. It means over the top violence followed up by ruthlessly hunting down the survivors like hasn't been seen since the Nazihunters after WWII. Second is to convince them in such a way that there remains zero doubt of our resolve that they have precisely two options. Stop their policy of quietly encouraging their nutters to wage jihad or we will invoke Ann Coulter's solution. That they can, by renouncing their dreams of conquest, live with us in peace as equals or they can be ended as a people, as a religion and as a threat forever. That Mecca can be reduced to radioactive rubble, every mosque on Islamic soil razed, Islam made an outlaw political idea in exactly the way Nazism was. In short, we make them an offer they don't refuse.
Why do we want to drill for more oil?
Because we need the stuff and the odds are that we are going to continue needing the stuff long after thee and me are worm chow. Someday we might run much of our world on fusion power and that will truly be a day worth living to see dawn. But nobody even has a proposal to replace jet fuel for example.
(Score: 2) by bzipitidoo on Tuesday November 17 2015, @03:45AM
Hmph, you are as blind as the worst of the Climate Change deniers. Demanding that climate models be tested by shoveling more and longer periods of data into them until they break down is asking the wrong thing of them, as well as asking too much. No model can perfectly replicate reality. The whole point of a model is to simplify reality the minimum necessary, to work with the always limited and incomplete data available, so that the tools we have can be applied, and results of practical value can be generated in a reasonable amount of time. Asking for more is akin to saying photography is useless and photos are WRONG because they never have enough information to enable a complete reproduction of a scene, right down to the quantum level, allowing the recreation of any dead people who happen to be in it, and so photos and videos should not be admissible evidence in court. You make an argument of that sort, and make it too brief so you can get to the fun part quicker, bashing away at AGW as if it was now beyond debate that scientists have it all wrong and are a bunch of frauds and/or dummies.
You are being blithe when you say "belief in AGW", and say that such is no different than belief in Transubstantiation and other faith based beliefs. Science is not about belief. Science takes NOTHING on faith. It is all about facts, evidence, and logical deductions, inferences, and conclusions. We operate on the assumption that observable reality is reality, pending evidence to the contrary. It may be that no such evidence will ever be found. We will probably want to improve our models of reality as we learn more-- General Relativity is very good, but it isn't the ToE. I don't "believe" in AGW. I accept that there is sufficient evidence and good enough modeling to conclude that AGW is real, and a problem.
(Score: 2) by jmorris on Wednesday November 18 2015, @03:32AM
I guess I failed in communicating. I'm saying the Climate Change Industry is not science. They do not behave as scientists should, they do not follow the scientific method and therefore what comes out of Climatology is not science. It is the core of a new age green religion if one is being generous and knowing fraud if one isn't so inclined.
You confirm my observation. I took the prediction of the warmers in the way they were published, as predictions with error bars and a confidence interval; i.e. as if they were doing science. The predicted results did not appear, therefore the models are useless. Perhaps after a careful analysis of why they failed they can try again.... but again it will take some time for the results to be tested and in the interval we should hold their past failure against them, especially since they are making extraordinary claims and demanding an extraordinary, civilization reorganizing response to their unproven theory. I'm the one behaving according to the teachings of the Scientific Method, Rational thought and the general Enlightenment ideas.
Contrast to your pathetic assertions. I'm being unreasonable, I'm "Demanding that climate models be tested by shoveling more and longer periods of data into them until they break down is asking the wrong thing of them." No, I'm comparing their predictions to reality in exactly the way the authors of the models claimed they should be evaluated in the time frames they claimed confidence predicting results for. When reality differed they changed the goalposts and you now come here and have the brass balls to tell me I'm the one making unreasonable demands. If the models lacked actual predictive power they should have said so. Too bad the models were the only argument the warmers had. Look in a dictionary, what is the word for a firmly held opinion not supported by physical evidence? Your sad devotion to that ancient religion doesn't inspire confidence in your reasoning ability.
I don't "believe" in AGW. I accept that there is sufficient evidence and good enough modeling to conclude that AGW is real, and a problem.
Really. Show me a model that isn't outside the error bars. That is how science works. Inside the error bars is good, outside is bad. What other evidence is there? Average global temps have been much higher than they are currently, they have also been a lot lower. Before man discovered fire, before the Industrial Revolution. Further attempts to even study the issue would now be pointless. The historical record has been destroyed (See ClimateGate) nobody working in the area is trustworthy at this point since they all behave more as politicians and priests compared to scientists. None outside the Climatology (and face it, the definition of Climatology is now the study of AGW in a perfect circular reasoning fallacy) industry without a decade of time to devote to a careful study of the literature and acquiring several advanced degrees can possibly sort out the lies from any science lurking in the morass.
Maybe AGW is correct and we will all suffer greatly as a result of people like me saying "A Pox on your house!." Maybe. That is going to be on your team though for lying. We used to teach tales like the Boy Who Cried Wolf for a reason you know.
(Score: 2) by bzipitidoo on Wednesday November 18 2015, @06:44AM
> Further attempts to even study the issue would now be pointless. The historical record has been destroyed
Ridiculous! The record has not been destroyed. Have you thought about what it would take to destroy all records of past weather? There are many scattered all over the world, with copies. Then there is all the evidence from the environment, things like glacial cores, tree rings, lake bottom sediments, and so on. If records had indeed been all lost or destroyed, we can recreate the data from those sources.
(Score: 3, Insightful) by frojack on Saturday November 14 2015, @08:35PM
Just do the replicate study and stop all the shenanigans with betting. Why add another avenue (and incentive) to game the system?
No, you are mistaken. I've always had this sig.
(Score: 0) by Anonymous Coward on Saturday November 14 2015, @10:27PM
If you read the paper, they write that they do not like replications and are looking to replace (not that these are common in psych anyway...) that part of science. This will just be used as a justification for to continuing to produce BS.
(Score: 0) by Anonymous Coward on Saturday November 14 2015, @09:25PM
Two week gig "correctly predicted" (whatever that means) 71%, as opposed to dart-thrower's 50%.
"It blew us all away," she said.
"What's going on with peer reivew?" she asks.
So psychology is like one of the them surrealist performance arts, eh. Or a campy in-joke?
(Score: 3, Insightful) by BK on Sunday November 15 2015, @12:07AM
Since when was psychology science?
...but you HAVE heard of me.
(Score: 2) by Thexalon on Sunday November 15 2015, @01:05PM
In a nutshell: Generous funding.
Including of things that don't seem immediately useful, because you never know what's actually going to be important until you study stuff. And that includes replication and checking of other scientists' results. Sure, a significant amount of money will likely be wasted. But the value of what we discover is also likely to be greater than cost we pay for it, even given the waste.
The only thing that stops a bad guy with a compiler is a good guy with a compiler.