SoylentNews is people
SoylentNews
According to a study conducted through heartbeat measurement app Cardiogram and the University of California, San Francisco, the Apple Watch is 97 percent accurate in detecting the most common abnormal heart rhythm when paired with an AI-based algorithm.
The study involved 6,158 participants recruited through the Cardiogram app on Apple Watch. Most of the participants in the UCSF Health eHeart study had normal EKG readings. However, 200 of them had been diagnosed with paroxysmal atrial fibrillation (an abnormal heartbeat). Engineers then trained a deep neural network to identify these abnormal heart rhythms from Apple Watch heart rate data.
Cardiogram began the study with UCSF in 2016 to discover whether the Apple Watch could detect an oncoming stroke. About a quarter of strokes are caused by an abnormal heart rhythm, according to Cardiogram co-founder and data scientist for UCSF's eHeart study Brandon Ballinger.
Yes, but can the Apple Watch then pace you or shock you?
(Score: 0) by Anonymous Coward on Sunday May 14 2017, @01:45PM (9 children)
Screening is all nice and well, but let's get the numbers together:
- 6158 people
- 200 of which had been diagnosed with paroxysmal atrial fibrillation ('unhealthy')
-> 5958 of which had been diagnosed with no paroxysmal atrial fibrillation ('healthy'). (Note: As I don't know any verified numbers, I take these numbers as representative for the general populace.)
- Accuracy of 97 % (as they don't specify, I take that as each sensitivity and specificity)
So follows:
- Of 200 affected ('unhealthy') people, 97 % (194 people) are getting the correct positive diagnosis
- Of 5958 healthy people, 3 % (179 people) are getting a false positive diagnosis
-> If you get a positive diagnosis, there's only about a 50:50 chance (179:194) you actually suffer from paroxysmal atrial fibrillation
The diagnosis doesn't help much, especially if you consider that paroxysmal atrial fibrillation only increases the risk of strokes, it doesn't somehow make them. So, if you are really worried about strokes, don't use a smart watch, go to your cardiologist and let them take an EKG proper.
(Score: 0) by Anonymous Coward on Sunday May 14 2017, @02:12PM (7 children)
The techcrunch article is garbage. If you read here you discover they mean AUC instead of accuracy:
https://blog.cardiogr.am/applying-artificial-intelligence-in-medicine-our-early-results-78bfe7605d32 [cardiogr.am]
You can get 90% accuracy with this data by always guessing "not arrhythmia".
(Score: 0) by Anonymous Coward on Sunday May 14 2017, @02:33PM
Oops, that would be 97% rather than 90%.
(Score: 3, Insightful) by AthanasiusKircher on Sunday May 14 2017, @02:46PM (1 child)
Yep. Actual numbers, based on the actual research:
- 6158 people
- 98.04% sensitivity, which means out of 200 people with heart conditions, they found 196 of them. 4 people who had heart conditions apparently weren't identified correctly (false negatives). Well, that's based on the number of unhealthy participants reported by Techcrunch; I don't know where the ".04%" comes from if those numbers are accurate.
- 90.2% specificity, which means out of the 5958 "healthy people," the Apple Watch correctly identified 5374 as actually healthy, as well as 584 as having a heart problem. Total people flagged as having heart problems therefore seems to be 780 of the 6158.
Thus, if the Apple Watch identifies you as having an "abnormal heart rhythm," you have a 196/780 = ~25% of actually having one. If the Apple Watch identifies you as NOT having an "abnormal heart rhythm," you have a ~0.07% of having one (and not having it detected).
To those unfamiliar with medical stats, this may sound overly good or overly bad. But this is typical of a preliminary diagnostic test -- high sensitivity, but mediocre specificity. What you want there is a very low false negative rate, and a subsequent confirmation test can weed out the false positives. The problem with such tests is that they end up worrying a large number of people who don't have any problem at all.
----
Stats note: Techcrunch misleadingly uses AUC of 0.97 as "accuracy," but that's not what the stat usually is interpreted as. Generally speaking, it usually is equal to the probability [wikipedia.org] that the classification scheme will rank a randomly chosen positive higher than a randomly chosen negative. Whether AUC is actually a good proxy for "accuracy" or strength of a model is debatable and depends on the situation. Based on the definition, AUC is obviously high for tests with very low false negative rates, but it won't necessarily give you good info about false positives.
(Score: 0) by Anonymous Coward on Sunday May 14 2017, @06:07PM
Original AC here. Thanks for digging into the actual numbers from TFA, they don't make it any better. ;)
So beware of the tests! As parent said: 'The problem with such tests is that they end up worrying a large number of people who don't have any problem at all.'
This automatically happens whenever a rare condition is tested (e. g., when the condition is significantly rarer than 1 in 2 people, say, 10 % and lower) and with a specificity only in the nineties—99.9 % and each more significant digit raises the bar, but which test/cheap screening gets that good?
(Score: 1) by Rich26189 on Sunday May 14 2017, @03:36PM (3 children)
This inquiring mind wants to know, what does ACU stand for in this context? I’ve no experience with stats or medical research. I followed the link, saw the acronym but no explanation, did I miss it? A DDG search for the acronym returned nothing of value neither did searching on “allacronyms.com. You’re warning us it should not be confused with accuracy, what is it?
(Score: 0) by Anonymous Coward on Sunday May 14 2017, @04:04PM (1 child)
It isn't a medical term, it means area under the ROC curve: https://www.kaggle.com/wiki/ROC [kaggle.com]
(Score: 1) by Rich26189 on Monday May 15 2017, @05:44PM
Thanks
(Score: 0) by Anonymous Coward on Sunday May 14 2017, @04:36PM
The kaggle wiki also contains this helpful page: https://www.kaggle.com/wiki/AreaUnderCurve [kaggle.com]
(Score: 3, Informative) by Nerdfest on Sunday May 14 2017, @02:19PM
Basically more free Apple advertising from a tech blog.