SoylentNews

upstart writes in with an IRC submission for SoyCow1337:

Technique reveals whether models of patient risk are accurate:

After a patient has a heart attack or stroke, doctors often use risk models to help guide their treatment. These models can calculate a patient’s risk of dying based on factors such as the patient’s age, symptoms, and other characteristics.

[...] “Every risk model is evaluated on some dataset of patients, and even if it has high accuracy, it is never 100 percent accurate in practice,” says Collin Stultz, a professor of electrical engineering and computer science at MIT and a cardiologist at Massachusetts General Hospital. “There are going to be some patients for which the model will get the wrong answer, and that can be disastrous.”

Stultz and his colleagues from MIT, IBM Research, and the University of Massachusetts Medical School have now developed a method that allows them to determine whether a particular model’s results can be trusted for a given patient. This could help guide doctors to choose better treatments for those patients, the researchers say.

[...] Computer models that can predict a patient’s risk of harmful events, including death, are used widely in medicine. These models are often created by training machine-learning algorithms to analyze patient datasets that include a variety of information about the patients, including their health outcomes.

While these models have high overall accuracy, “very little thought has gone into identifying when a model is likely to fail,” Stultz says. “We are trying to create a shift in the way that people think about these machine-learning models. Thinking about when to apply a model is really important because the consequence of being wrong can be fatal.”

[...] The researchers’ new technique generates an “unreliability score” that ranges from 0 to 1. For a given risk-model prediction, the higher the score, the more unreliable that prediction. The unreliability score is based on a comparison of the risk prediction generated by a particular model, such as the GRACE risk-score, with the prediction produced by a different model that was trained on the same dataset. If the models produce different results, then it is likely that the risk-model prediction for that patient is not reliable, Stultz says.

“What we show in this paper is, if you look at patients who have the highest unreliability scores — in the top 1 percent — the risk prediction for that patient yields the same information as flipping a coin,” Stultz says. “For those patients, the GRACE score cannot discriminate between those who die and those who don’t. It’s completely useless for those patients.”

The researchers’ findings also suggested that the patients for whom the models don’t work well tend to be older and to have a higher incidence of cardiac risk factors.

Journal reference: Identifying unreliable predictions in clinical risk models, npj Digital Medicine (DOI: 10.1038/s41746-019-0209-7)

Original Submission