SoylentNews is people
SoylentNews
https://www.bmj.com/content/363/bmj.k5094
A study has been done, and the surprising result is that parachutes are no more effective than a backpack in preventing injuries when jumping out of an airplane.
It's "common sense" that parachutes work, so it has been a neglected field of science. This surprising and counter-intuitive result is an excellent example of the importance of doing science.
... or maybe it's a perfect example of how top-line study headlines can be mis-representative, especially when portrayed by the mass-media, and how understanding study scope and methodology is important.
This discussion has been archived.
No new comments can be posted.
Study "Proves" Parachutes Ineffective
|
Log In/Create an Account
| Top
| 68 comments
| Search Discussion
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Remember Darwin; building a better mousetrap merely results in smarter mice.
(Score: 0) by Anonymous Coward on Sunday December 23 2018, @10:10PM (18 children)
I already put forth the max effort interacting with bio-trained people. I did all the normal bs they usually do plus actual science with quantitative models that made precise predictions, and getting other peoples data to check the predictions, proving it could be done when they said biomedical research is "too complex" for that. Result: "so... was there a significant difference?"
They dont care dude. Worse, most dont want to hear it because it is 10x harder than what they are getting away with. It is actually effectively impossible for real science to compete along the number of publications metric because of that. Like some version of Gresham's Law, all the good science is getting hoarded in classified programs and trade secrets.
And no I dont only post this on soylent news, or think it is anything more than me treating people the way I wish to be treared.
(Score: 0) by Anonymous Coward on Sunday December 23 2018, @10:52PM (17 children)
From an outside perspective, your posts just appear dismissive. The kind of dismissive attitude that people use to make themselves feel superior to those they put-down or appear higher-status because of their cynicism.
If you aren't the same person as the "CRISPR isn't real because it just kills cells" or the "all science is false because of NHST", then I apologize for the mix-up.
(Score: 0) by Anonymous Coward on Sunday December 23 2018, @11:07PM (10 children)
Amazing how the NHST supporters can only ever seem to argue against strawmen. Crispr is real and does selectively kill cells, and NHST has nothing to do with science.
As I said, some people seem to be incapable of getting it. I usually assume its due to some bad training, but maybe there is something deeper going on there with an entire mindset. Those people aren't my audience.
(Score: 0) by Anonymous Coward on Sunday December 23 2018, @11:48PM (9 children)
It seems you are the person I thought you were.
I don't know what is so difficult for you to understand: NHST is treated as a floor. If the data can't even pass a NHST, then it is beyond unreliable. Passing a NHST is not evidence against a hypothesis, like you seem to treat it, it is weak evidence that the data isn't complete garbage.
(Score: 0) by Anonymous Coward on Monday December 24 2018, @12:06AM (5 children)
Passing NHST is not primarily used like you say, and anyway I "dont understand" that because that would be wrong.
Unless you actually believe the null model may be true, its completely controlled by sample size (you will always get significance if you spend enough). It doesnt matter if the data is garbage or not.
I mean look at this study where they "didnt pass NHST", do you see people concluding the data is beyond unreliable? No we dont, proving your claim is just wrong.
Trust me. There is no legimate use for NHST (when testing a strawman). I have searched everywhere, the only thing people use it for is to subsequently commit one or more logical fallacies like yours here. And yours is hilariously opposite of the the one comitted in the parachute paper. Contradictions like this can only happen because NHST is based on a logical fallacy.
(Score: 0) by Anonymous Coward on Monday December 24 2018, @12:58AM (4 children)
It is pretty funny that you are using an obviously satirical paper as evidence, but I will concede that I did not state things clearly as I meant for positive claims. Here's an example:
Paper A:
Main claim - Drug X can inhibit breast cancer progression in mice.
Evidence - Two groups (Mock-treated and Drug X-treated) with 20 mice each. Average cancer stage of Drug X-treated mice was lower (by one stage) than Mock-treated, p=0.3.
Paper B:
Main claim - Drug Y can inhibit breast cancer progression in mice.
Evidence - Two groups (Mock-treated and Drug Y-treated) with 20 mice each. Average cancer stage of Drug Y-treated mice was lower (by one stage) than Mock-treated, p0.001.
Journals would typically reject "Paper A" because their survival study didn't even meet the commonly used threshold of whichever statistical test they used. In other words, their evidence isn't even internally consistent and/or their methods of data collection were not precise enough to discriminate between their groups. Now, you might say, "if they used n=1000 per group, then they would see an effect", but that would also likely be dismissed as not "biologically significant" because if your power test says you need an n=1000 genetically identical mice with genetically identical parental tumor lines, then your effect size is so small that it does not have a biologically meaningful result and even less likely to translate into humans.
If you completely discount NHST then you would say that Drug X is equally likely, given what is provided and assume that everything is equal except the distribution of data, to inhibit breast cancer progression as Drug Y. Which of the claims is more likely to be true?
(Score: 1, Informative) by Anonymous Coward on Monday December 24 2018, @01:30AM (3 children)
I would never design a series of studies like this to begin with. The plan is optimized for producing the maximum number of papers rather than learning about cancer and curing it as quickly as possible.
I would study many untreated mice under various conditions and then come up with at least one quantitative model that could fit the observations. Eg, we coud fit to incidence of various stages by age, something we also have for humans. I think SEER has data by stage, but they definitely have overall incidence of many cancer types.
Just guessing, but the curve be affected by mutation rate of cells in that tissue, rate of clearance by the immune surveillance, apoptosis rate, number of mutations required for detectable cancer, division rate in that tissue, number of cells in the tissue, calories consumed, etc. Some of these parameters may be degenerate or even caused by one another, so it would have to all be worked out.
NB: that also requires measuring all those parameters carefully in the normal mice, I bet there isn't even good data on how many cells there typically are in each tissue at a given age... because everyone has been wasting time with NHST.
Only then would I think about how Drug X and Drug Y, etc are supposed to work and which parameter(s) of the model they should affect and how. Once I have made my predictions, I would run the study while giving the drugs and see if the parameters of the best model fit changed in line with the predictions. If they do, then I would think I had a handle on how the prospective treatment was working. Of course, any major side effects should be predicted and accounted for by the model as well.
I'd say, who knows why the variance was higher in the first case? It could be due to the drug, or they just messed up the experiment somehow, or just "random" stuff.
Also, not sure if you realize this, but if there really was no effect of the drug on cancer stage both p-values would be equally likely). In R:
https://i.ibb.co/THQ4QWW/null-True-Pdist.png [i.ibb.co]
(Score: 1, Informative) by Anonymous Coward on Monday December 24 2018, @01:35AM
To clarify my own post:
This isnt really true since the sampling distribution could be different for some other reason (as suggested by the previous sentence). It is more correct to say "no difference in the populations each sample came from". Also, further reading:
https://stats.stackexchange.com/questions/10613/why-are-p-values-uniformly-distributed-under-the-null-hypothesis [stackexchange.com]
(Score: 0) by Anonymous Coward on Monday December 24 2018, @01:44AM (1 child)
So you think that Paper A was more likely to have messed something up or didn't properly account for an experimental variable, but you still think it is equally likely as Paper B?
(Score: 0) by Anonymous Coward on Monday December 24 2018, @01:53AM
No, it could be B that messed up, or neither (could just be irreducible variation in the mouse system), or both. But sure, something was different between the two studies.
(Score: 0) by Anonymous Coward on Monday December 24 2018, @12:59AM (2 children)
This is exactly how NHST is commonly used:
https://www.bmj.com/content/363/bmj.k5094 [bmj.com]
So if they keep getting "no significance", then people should stop using parachutes when jumping from aircraft.
(Score: 0) by Anonymous Coward on Monday December 24 2018, @01:10AM (1 child)
*When that aircraft is resting on the ground and motionless.
(Score: 0) by Anonymous Coward on Monday December 24 2018, @01:40AM
Billions of dollars are spent annually on people jumping from landed motionless airplanes with parachutes?
I don't think they meant to avoid extrapolating to flying airplanes, but even if they had meant to limit the conclusion to stationary ones it would still be fallacious reasoning.
(Score: 1) by khallow on Tuesday December 25 2018, @06:18AM (5 children)
You have a point to that? Let's give you an inside perspective [economist.com]. For example:
Sorry, your "outside perspective" is ignorant. There are deep, decades-old problems in most fields of science. It's not going to get better because someone whines that the criticism is presented in a imaginary, dismissive manner, or as the earlier AC whines, because few who are part of the problem will listen to the criticism. All we can do at this point is spread awareness.
As to Null Hypothesis Significance Testing, the key thing to remember is that it is a tool for finding initial hypotheses and developing models almost from scratch. If you're still using it, as a number of fields are, decades later after you should have found those hypotheses and models, then you're doing something very wrong. If your research is considered normal despite being NHST on decades old fields, then the field itself is doing something wrong.
(Score: 0) by Anonymous Coward on Tuesday December 25 2018, @08:45AM (4 children)
You are coming around, but still think there is some validity to NHST. There isn't.
It is as scientific as praying to think of the right answer.
(Score: 1) by khallow on Tuesday December 25 2018, @03:33PM (3 children)
(Score: 0) by Anonymous Coward on Tuesday December 25 2018, @06:22PM (2 children)
This doesnt explain what you think the NHST step is supposed to contribute. You don't do NHST here. You describe/explore the data or clean it and throw it into some machine learning algo (depending on the goal) where you choose a model based on out-of-sample predictive skill.
(Score: 1) by khallow on Tuesday December 25 2018, @06:43PM (1 child)
That sentence wasn't supposed to explain, this sentence was:
NHST is a machine learning algorithm. Perhaps one could swap it out for a much better algorithm, but a key problem with any such approach from research purposes, is that it needs to generate a testable model that you understand in the end. For example, I can generate some rather opaque genetic algorithm for modeling phenomena, but it'd be work to figure out whether the model is modeling something real or a loophole that I haven't found yet. NHST spits out correlations that you can test right away.
(Score: 0) by Anonymous Coward on Tuesday December 25 2018, @08:13PM
NHST spits out binary conclusions and involves concluding something beyond "the null model doesnt fit". It very correctly can tell you when the model doesnt fit.
Also, you can calculate p-values and use them without NHST... NHST != "Hypothesis testing" != "Significance Testing": https://arxiv.org/pdf/1603.07408.pdf [arxiv.org]
An interesting thing is that I've seen Neyman, Pearson, and Gosset ("Student") lapse into NHST, but never Fisher. He is always very careful not to get confused between the "research hypothesis" and the "statistical hypothesis".