What’s Really Behind the UCSF’s Top Five Ranking

The US News and World Report published its annual rankings of the nation’s medical schools last month, ranking the UCSF School of Medicine fourth in both “Research” and “Primary Care.” That’s a big deal, right?

On the day the rankings were published online, UCSF prominently featured an article on its home page proclaiming: “UCSF Ranked Among Top Medical Schools in Nation by US News.”

Similar declarations were splashed across Facebook profiles, featured in popular news stories, and heard in passing conversations from the library bathrooms to the cafeteria salad lines. UCSF is a Top Five school!  UCSF is a Top Five school!

But the astute (or even mildly interested) observer might wonder what it actually means for a school to be highly ranked by the US News and World Report. UCSF is ranked No. 4 in what? By what measure? By whose account?

Methodology revealed

According to the US News and World Report, its methodology for ranking the nation’s “Best Medical Schools” consisted of a survey that was sent to all 149 of the accredited schools of allopathic and osteopathic medicine in the United States.

Using the data from the surveys that were returned, the magazine calculated a school’s rankings based on a set of indicators tailored to reflect its strength in either “Research” or “Primary Care.” The highest-scoring institution was given a rating of 100, and the scores of every other school were scaled accordingly.

Let’s take a look at the 2014 “Research” rankings, where UCSF received an adjusted score of 86. Without even knowing what this number means, what immediately jumps out is the small range of scores near the top.

UCSF is just one point behind Johns Hopkins and three points behind Stanford, the institution that took the No. 2 spot. Washington University in St. Louis, ranked No. 6, is just two points below UCSF. The small numerical difference between similarly ranked institutions begs the question: What really separates schools on the US News and World Report rankings?

Who’s Really No. 1?

“I don’t believe you can rank medical schools in numerical order,” said David Wofsy, Dean of Admissions at the UCSF School of Medicine. “I do believe generally that the fact that we rank high across very distinct areas in which schools are judged does say something meaningful about UCSF. But I don’t believe anyone knows which school in the country is No. 1, No. 2 or No. 3. I don’t think it works that way.”

Instead, Wofsy believes that there are many outstanding medical schools in this country, and he questions the ability of any ranking system to universally reflect what makes schools attractive to students.

However, the US News and World Report certainly would like to try, so let’s try to decipher what exactly went into UCSF’s “Research” score of 86.  

The “Research Activity” metric, which only takes into account grants from the National Institutes of Health (NIH), clearly favors universities with large health science programs such as UCSF. In 2012, the UCSF School of Medicine received almost $450 million in NIH funding, the most in the country.

But what does this figure tell us about UCSF’s medical school? Does this reflect the strength of its research programs? The fundability of its projects?  The grant-writing skills of its faculty and staff?

Speaking of the skills of the faculty, US News uses “Faculty Resources” as another indicator of a medical school’s strengths. This category is based on a simple ratio of the number of full-time science and clinical faculty to the number of full-time MD or Doctor of Osteopathy students.

The thinking perhaps goes that a school like Harvard, with a faculty-to-student ratio of 13:1, can offer more to its students than a smaller institution like Brown, with a ratio of 1.8:1. (UCSF, by the way, has 3.1 full-time faculty for every full-time student.)

The bulk of the “Student Selectivity” category is based on the mean MCAT scores and GPAs of the previous year’s matriculating class. This is perhaps a proxy for a school’s desirability in the eyes of its applicants — better schools attract more quantitatively competitive applicants.

However, using desirability as a proxy quickly raises the question of how students evaluate which schools to apply to and ultimately, matriculate at. One possibility: the US News and World Report rankings.

Perennial cycle of information

Most enigmatic, however, is the “Quality Assessment” category. This indicator, worth the largest piece of the rankings pie, is based on an institution’s reputation as evaluated by the deans and residency program directors of the nation’s 148 other medical schools.

When she was Dean of the College of Medicine at Ohio State University, Catherine Lucey, current Vice Dean of Education at UCSF, received a booklet every year listing every medical school in the country.

She then went down the list and ranked every school based on what she called “comprehensive excellence,” which encompassed not only their excellence in research, but also the quality of their medical education programs and their patient care.

But where did she find this information? As it turns out, in a lot of disparate places, including the American Association of Medical Colleges, the University Health Consortium, and her own knowledge of the educational programs that have been established in schools around the country.

“There are lots of ways you can get information that you can integrate on your own, but there is no integrative rating that incorporates things other than what the US News and World Report does at this point,” Lucey said.

The challenges in finding comprehensive information about each and every one of the 149 accredited medical schools sets up an awkward situation — the most readily available source of information about a school in order to rank it in the US News and World Report might be the US News and World Report. The subjective nature of this reputational score may help explain the consistency in which schools that have been highly ranked in the past appear in the top 10 or 20.

“Reputation is not a very dynamic number,” said Lucey. “You might imagine that once you’re on top, you tend to stay on top.”

And stay on the top UCSF has. For the past 10 years of rankings, UCSF has ranked in the top 10 spots for both “Research” and “Primary Care.” For the past four years, UCSF has ranked in the top five.

The consistency with which UCSF appears in the top of the list is not an insignificant matter, even if academic physicians like Lucey and Wofsy may look toward many other indicators for judging the strengths of individual schools.

But since all schools are interested in their public image, publications such as US News and World Report can play a big role in shaping how people who may never attend UCSF view the institution.

This, I think, gets to the crux of why these US News and World Report rankings matter. They matter because we talk about them on the elevator, to our grandparents, on Facebook.  We write about them, both matter-of-factly and with a critical perspective, in The New Yorker, Academic Medicine and yes, even UCSF’s Synapse. (I should also note that they matter because they make money for the US News and World Report, which released its last print monthly in 2010 and now charges up to $34.95 for exclusive access into its rankings data.)

So is it a big deal that the UCSF School of Medicine was ranked fourth this year in both “Research” and “Primary Care?”

You tell me.