No prior judgments
To develop his algorithm, Dr. Turner used a branch of AI called “unsupervised learning.” It doesn’t make a prior judgment about what the data should look like, Dr. Turner explained.
“You’re just analyzing natural trends within the data.”
The algorithm tries to find and discover clusters or patterns within the data. “It’s like saying to the algorithm: ‘I want you to tell me what schools you think should be grouped together based on the data I feed you,’ which is the data that the user selects based on his or her personal preferences.”
U.S. News has been transparent about the metrics it uses, Dr. Turner notes. “When I started looking into how rankings are developed, I saw that there was transparency, and the reasoning for choosing the metrics used to develop the ranking was pretty sound,” he said.
“But I didn’t see any justification as to why they chose the particular metrics and weighted them in the way that they did.”
Dr. Turner extracted data from the 2023 U.S. News report, which ranked 109 allopathic medical schools, and applied several scenarios to the results to create his alternative ranking system.
In one scenario, he used the same research metrics used by U.S. News, such as a peer research assessment, median federal research activity per full-time faculty member, median GPA, median MCAT, acceptance rate, and faculty-student ratio.
In another scenario, he included four additional metrics: debt, in-state cost of attendance, USMLE Step 1 passing rate, and percentage of underrepresented students with minority race or ethnicity at the school.
For example, a user can rank the importance of the diversity of the class, amount of debt students expect to incur, and amount of research funding the medical school receives. After selecting those factors, the tool generates tiered results displayed in a circle, a shape chosen to avoid the appearance of the hierarchy associated with traditional rankings, Dr. Turner said.
“A prospective student might not care about acceptance rates and MCAT scores, and instead cares about diversity and debt,” Dr. Turner said. He looks forward to extending this approach to the ranking of colleges as well.
‘Imperfect measures’
“The model and interesting online tool that Dr. Turner created allows a premed [student] to generate custom rankings that are in line with their own priorities,” said Christopher Worsham, MD, MPH, a critical care physician in Mass General’s division of pulmonary and critical care medicine.
But Dr. Worsham, also a teaching associate at Harvard Medical School’s department of health care policy, expressed concern that factors figuring into the rankings by U.S. News and Dr. Turner’s alternative “are imperfect measures of medical school quality.”
For example, a student interested in research might favor federal research funding in their customized rankings with Dr. Turner’s model. “But higher research funding doesn’t necessarily translate into a better education for students, particularly when differentiating between two major research systems,” Dr. Worsham noted.
Dr. Worsham added that neither ranking system accurately predicts the quality of doctors graduating from the schools. Instead, he’d like to see ranking systems based on which schools’ graduates deliver the best patient outcomes, whether that’s through direct patient care, impactful research, or leadership within the health care system.
Michael Sauder, PhD, professor of sociology at the University of Iowa, Iowa City, said the model could offer a valuable alternative to the U.S. News ranking system. It might help users develop their own criteria for determining the ranking of medical schools, which is a big improvement over a “one-size-fits-all” approach, Dr. Sauder said.
And Hanna Stotland, an admission consultant based in Chicago, noted that most students rely on rankings because they “don’t have the luxury of advisers who know the ins and outs of different medical schools.” Given the role that rankings play, Ms. Stotland expects that every new ranking tool will have some influence on students.
This tool in particular “has the potential to be useful for students who have identified values they want their medical school to share.” For example, students who care about racial diversity “could use it to easily identify schools that are successful on that metric,” Ms. Stotland said.
Sujay Ratna, a 2nd-year med student at Icahn School of Medicine at Mount Sinai in New York, said he considered the U.S. News ranking his “go-to tool” when he was applying to med school.
But after reading Dr. Turner’s article, the AMSA membership vice president tried the algorithm. “I definitely would have used it had it existed when I was thinking of what schools to apply to and what [schools] to attend.”
The study had no specific funding. Dr. Turner, Dr. Worsham, Dr. Sauder, Ms. Stotland, Ms. Brett, and Mr. Ratna report no relevant financial relationships.
A version of this article first appeared on Medscape.com.