SoylentNews

Were Those Experiment Results Really So Predictable? These Researchers Aim To Find Out

posted by janrinok on Sunday November 03 2019, @01:43AM   
from the if-everbody-knows,-why-isn't-it-known? dept.

Arthur T Knackerbracket has found the following story:

They say that hindsight is 20-20, and perhaps nowhere is that more true than in academic research.

"We've all had the experience of standing up to present a novel set of findings, often building on years of work, and having someone in the audience blurt out 'But we knew this already!,'" says Prof. Stefano DellaVigna, a behavioral economist with joint appointments in the Department of Economics and Berkeley Haas. "But in most of these cases, someone would have said the same thing had we found the opposite result. We're all 20-20, after the fact."

DellaVigna has a cure for this type of academic Monday morning quarterbacking: a prediction platform to capture the conventional wisdom before studies are run.

Along with colleagues Devin Pope of the University of Chicago's Booth School of Business and Eva Vivalt of the Research School of Economics at Australian National University, he's launched a beta website that will allow researchers, PhD students, and even members of the general public to review proposed research projects and make predictions on the outcome.

[...]Their proposal, laid out in a new article in Science's Policy Forum, is part of a wave of efforts to improve the rigor and credibility of social science research. These reforms were sparked by the replication crisis -- the failure of reproduce the results of many published studies -- and include mass efforts to replicate studies as well as platforms for pre-registering research designs and hypotheses.

"We thought there was something important to be gained by having a record of what people believed before the results were known, and social scientists have never done that in a systematic way," says DellaVigna, who co-directs the Berkeley Initiative for Behavioral Economics and Finance. "This will not only help us better identify results that are truly surprising, but will also help improve experimental design and the accuracy of forecasts."

Journal Reference:

Stefano DellaVigna, Devin Pope, Eva Vivalt. Predict science to improve science. Science, 2019; 366 (6464): 428 DOI: 10.1126/science.aaz1704

---

Original Submission

• Sounds like a for profit scheme Sounds like a for profit scheme (Score: 2, Informative) by khallow on Sunday November 03 2019, @11:32AM (2 children)

  by khallow (3766) on Sunday November 03 2019, @11:32AM (#915288) Journal

  The terms and conditions [socialscienceprediction.org] have this intriguing part:

  12) MODIFICATION & VARIATION
  
  We may, from time to time and at any time without notice to You, modify this Agreement. You agree that We have the right to modify this Agreement or revise anything contained herein. You further agree that all modifications to this Agreement are in full force and effect immediately upon posting on the Website and that modifications or variations will replace any prior version of this Agreement, unless prior versions are specifically referred to or incorporated into the latest modification or variation of this Agreement.
  
  a) To the extent any part or sub-part of this Agreement is held ineffective or invalid by any court of law, You agree that the prior, effective version of this Agreement shall be considered enforceable and valid to the fullest extent.
  
  b) You agree to routinely monitor this Agreement and refer to the Effective Date posted at the top of this Agreement to note modifications or variations. You further agree to clear Your cache when doing so to avoid accessing a prior version of this Agreement. You agree that Your continued use of the Website after any modifications to this Agreement is a manifestation of Your continued assent to this Agreement.
  
  c) In the event that You fail to monitor any modifications to or variations of this Agreement, You agree that such failure shall be considered an affirmative waiver of Your right to review the modified Agreement.

  Here's another:

  b) How We Use Information: We use the information gathered from You to ensure Your continued good experience on Our website, including through email communication. We may also track certain aspects of the passive information received to improve Our marketing and analytics, and for this, We may work with third-party providers.

  The tone of this particular document is not at all academic. For example, there's no boilerplate about the risks of human experimentation (even polling studies often have that). There's this peculiar concern about improving "marketing and analytics". And the bald assertion that the user is responsible for keeping up with changes to a document they will mostly likely never, ever read in the first place is a classic business move.

  Starting Score:	1		point
  Moderation		+1
  Informative=1, Total=1
  Extra 'Informative' Modifier		0
  Total Score:		2

• Re:Sounds like a for profit scheme (Score: 2) by SpockLogic on Sunday November 03 2019, @02:09PM

  by SpockLogic (2762) on Sunday November 03 2019, @02:09PM (#915320)

  Well spotted.

  Thank you.

  --
  Overreacting is one thing, sticking your head up your ass hoping the problem goes away is another - edIII

  Parent

• Re:Sounds like a for profit scheme (Score: 2) by NotSanguine on Sunday November 03 2019, @05:27PM

  by NotSanguine (285) <{NotSanguine} {at} {SoylentNews.Org}> on Sunday November 03 2019, @05:27PM (#915369) Homepage Journal

  When I went to the site and was immediately prompted to either log in or create an account, I clicked on the proper button (close tab).

  --
  No, no, you're not thinking; you're just being logical. --Niels Bohr

  Parent