Stories
Slash Boxes
Comments

SoylentNews is people

posted by martyb on Friday April 19 2019, @06:34PM   Printer-friendly
from the significant-change dept.

In science, the success of an experiment is often determined by a measure called "statistical significance." A result is considered to be "significant" if the difference observed in the experiment between groups (of people, plants, animals and so on) would be very unlikely if no difference actually exists. The common cutoff for "very unlikely" is that you'd see a difference as big or bigger only 5 percent of the time if it wasn't really there — a cutoff that might seem, at first blush, very strict.

It sounds esoteric, but statistical significance has been used to draw a bright line between experimental success and failure. Achieving an experimental result with statistical significance often determines if a scientist's paper gets published or if further research gets funded. That makes the measure far too important in deciding research priorities, statisticians say, and so it's time to throw it in the trash.

More than 800 statisticians and scientists are calling for an end to judging studies by statistical significance in a March 20 comment published in Nature. An accompanying March 20 special issue of the American Statistician makes the manifesto crystal clear in its introduction: "'statistically significant' — don't say it and don't use it."

There is good reason to want to scrap statistical significance. But with so much research now built around the concept, it's unclear how — or with what other measures — the scientific community could replace it. The American Statistician offers a full 43 articles exploring what scientific life might look like without this measure in the mix.

Statistical Significance

Is is time for "P is less than or equal to 0.05" to be abandoned or changed ??


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 0) by Anonymous Coward on Friday April 19 2019, @07:27PM (11 children)

    by Anonymous Coward on Friday April 19 2019, @07:27PM (#832257)

    The letter is signed by a bunch of medical scientists. Their problems with statistical significance do not apply to science generally. Physics uses a .0000003 cutoff to determine statistical significance, and it's not a problem. Medical research has the problem because .05 is such a weak result.

  • (Score: 0) by Anonymous Coward on Friday April 19 2019, @07:36PM (10 children)

    by Anonymous Coward on Friday April 19 2019, @07:36PM (#832258)

    http://nautil.us/blog/the-present-phase-of-stagnation-in-the-foundations-of-physics-is-not-normal [nautil.us]

    Progress in physics stopped in the 1970s, right when they adopted NHST (testing something besides the predictions of their theory). Since then all theyve done is "verify" what they already "knew".

    It destroys every field that adopts it because it is pseudoscience. In science you predict something and hope to *not* observe a "significant" deviation.

    • (Score: 1, Funny) by Anonymous Coward on Friday April 19 2019, @08:21PM (2 children)

      by Anonymous Coward on Friday April 19 2019, @08:21PM (#832273)

      Wow! The Null Hypothesis AC! Didn't see that coming! What were the odds of him posting in this thread, I wonder? Statistically significant, I bet. P=1.

      • (Score: 0) by Anonymous Coward on Friday April 19 2019, @08:35PM (1 child)

        by Anonymous Coward on Friday April 19 2019, @08:35PM (#832279)

        Wow, the AC with nothing of value to say about the topic.

        • (Score: 0) by Anonymous Coward on Friday April 19 2019, @09:48PM

          by Anonymous Coward on Friday April 19 2019, @09:48PM (#832315)

          The Null Hypothesis is the AC! Thereby proving that the Parent AC is not statistically significant.

    • (Score: 1, Informative) by Anonymous Coward on Friday April 19 2019, @08:38PM (6 children)

      by Anonymous Coward on Friday April 19 2019, @08:38PM (#832281)

      Progress in physics slowed because the low-hanging fruits have been picked, and successful fields like electrical engineering get spun off. do you think there is an endless supply of potential discoveries like electromagnetism, lasers, and transistors?

      Before gravitational waves were measured, it was not known how many events would be visible. Some people thought none at all. Statistical significance was used to prove that the measurements were meaningful. Now the term is gauche? A Nobel Prize was recently given for work that required it.

      • (Score: 0) by Anonymous Coward on Friday April 19 2019, @10:05PM (5 children)

        by Anonymous Coward on Friday April 19 2019, @10:05PM (#832324)

        Physicists said all the low hanging fruit was picked in 1900 too. It might be true one day but usually its just an excuse.

        https://hsm.stackexchange.com/questions/2129/who-said-that-essentially-everything-in-theoretical-physics-had-already-been-dis [stackexchange.com]

        • (Score: 3, Insightful) by Coward, Anonymous on Friday April 19 2019, @10:28PM (2 children)

          by Coward, Anonymous (7017) on Friday April 19 2019, @10:28PM (#832337) Journal

          Do you have any evidence that physicists now are less intelligent or creative than a century ago? Yet there are many more of them but they are producing fewer fundamental discoveries. That is some pretty strong evidence for the low hanging fruit having been picked.

          Null experiments are something to do when you don't know what else to do. I'm not a big fan myself, and on the list of Nobel Prizes in physics [wikipedia.org] I didn't see any for null-hypothesis testing. Physics is not dominated by this approach.

          • (Score: 0) by Anonymous Coward on Friday April 19 2019, @11:48PM

            by Anonymous Coward on Friday April 19 2019, @11:48PM (#832374)

            Yes there are many more of them wasting their time with NHST-based experiments or confused about what evidence needs to be explained by theory or not due to BS generated by NHST.

          • (Score: 3, Interesting) by RamiK on Saturday April 20 2019, @04:34AM

            by RamiK (1813) on Saturday April 20 2019, @04:34AM (#832445)

            Do you have any evidence that physicists now are less intelligent or creative than a century ago?

            A century ago you had John von Neumann and far less scientists-per-capita and people alive. In the spirit of the topic, it averages out to smarter overall.

            --
            compiling...
        • (Score: 2) by hendrikboom on Saturday April 20 2019, @02:20AM (1 child)

          by hendrikboom (1125) Subscriber Badge on Saturday April 20 2019, @02:20AM (#832415) Homepage Journal

          Gravitational telescopes and particle colliders are sufficiently expensive that their fruit cannot be called low-hanging.

          • (Score: 0) by Anonymous Coward on Saturday April 20 2019, @11:38AM

            by Anonymous Coward on Saturday April 20 2019, @11:38AM (#832510)

            The need to rely on brute force could also indicate a lack of cleverness.