SoylentNews

hubie [soylentnews.org] writes:

New Bayesian model can combine conflicting rankings, measure uncertainty and reveal what drives them [rice.edu]:

Who, really, is the best player in baseball?

Ask five sports media outlets and you may get five different answers. Rankings can vary dramatically from one source to another, change from year to year and leave out different players altogether, making it difficult to determine where there is genuine consensus and where opinions diverge.

Now, Rice University statisticians, in collaboration with Cornell University, have developed a new method designed to find clearer answers in messy rankings data. The research was recently published in the Journal of Quantitative Analysis of Sports [degruyterbrill.com].

The approach, called Bayesian Multivariate Rank Regression, or BMRR, combines rankings from multiple sources and across multiple time points while accounting for disagreements among the rankers, incomplete lists and factors that might influence the rankings. And just as importantly, the method measures just how certain or uncertain the resulting consensus is.

"Rankings look simple on the surface, but statistically they're actually very complicated," said Rose Graves, a doctoral student studying statistics at Rice and the study's corresponding author. "Two experts may rank different numbers of items, disagree about the order or even change their opinions over time. We wanted to create a way to bring all of that information together while accounting for uncertainty."

For their real-world test case, the researchers turned to one of the most closely scrutinized ranking systems in sports: annual lists of Major League Baseball's best players. They analyzed preseason rankings published by ESPN, CBS, Bleacher Report, Yahoo Sports and MLB from 2021 through 2024. Each outlet ranked its top 100 players, but the lists varied in both who was included and where players were ranked. To create a consistent pool for comparison, the researchers focused on 55 players who appeared in at least one outlet's rankings in each of the four years.

Those differences illustrate exactly what makes ranking data difficult to analyze. Simply averaging a player's position across several lists can mask meaningful information, especially when a player is omitted from one list altogether or when assessments shift substantially from year to year.

BMRR instead treats the rankings as evidence about an underlying value and uses a hierarchical Bayesian framework to combine that evidence. The model can incorporate incomplete rankings and ties, borrow information across years and estimate how closely each source aligns with the overall consensus. Unlike a single definitive list, the result also includes a measure of uncertainty, showing when the data strongly favor one player and when two players are effectively too close to separate.

[...] "Rather than asking only who ranks first, this framework lets us ask how confident we are in that conclusion, why someone may be ranked highly and how much agreement actually exists among the people doing the ranking," said Marina Vannucci, the Noah Harding Professor of Statistics at Rice and one of the authors of the paper. "That kind of uncertainty quantification can be extremely important when rankings are being used to inform decisions."

[...] While MLB provided a compelling proving ground, the researchers designed BMRR for a much broader problem.

Rankings appear almost everywhere: Search engines order websites, voters rank candidates, experts prioritize intelligence information and clinicians may rank treatment preferences. Existing statistical approaches often struggle when different raters evaluate different items, when rankings are repeated across multiple criteria or points in time or when analysts want to incorporate additional information about what is being ranked.

"The biggest challenges when working with ranking data are remarkably similar across different fields and industries," said Dan Kowal, a previous Rice statistics faculty member now at Cornell University and second author on the paper. "We believe that BMRR can be impactful in many of these areas beyond Major League Baseball."

The authors encourage broader application of their method and have made the code for BMRR available on GitHub [github.com], so other statisticians can apply this method to their own data.

"Wherever multiple sources disagree about what belongs at the top, our framework offers a way to identify a consensus while measuring how confident that consensus should be," Graves added.

Journal Reference: https://doi.org/10.1515/jqas-2026-0012 [doi.org]

Original Submission